close
Comments you submit will be routed for moderation. If you have an account, please log in first.
Modify

Opened 4 years ago

Closed 11 months ago

Last modified 7 months ago

#7 closed bug (fixed)

Handling of charsets (e.g. UTF-8) when using external editor.

Reported by: emptystate@… Owned by: damato
Priority: normal Milestone: YAM 2.9
Component: charset handling Version: 2.5
Severity: major Keywords:
Cc: OS Platform:
Blocked By: Blocking:
Release Notes:

improved charset/codeset handling support when using external editors aware of UTF-8

Description (last modified by damato)

When receiving an email with a charset that is not supported by the font system (I think, I don't know how YAM is detecting this charset), YAM shows a question mark for every character that is not standard ASCII. When YAM passes this mail to an external program or saves as a file it still replaces the characters with question marks. YAM should not replace these characters in these situations.

For example, my friend recently sent me an email in UTF-8 with an attachment. Their MUA decided to package the text as base64. In YAM when I read this mail I get "????? ?? ????" etc in the message window. If I save the displayed message I get the same "????? ?? ????" etc, if I save the raw message I get base64. If I try to reply and call an external editor that can handle the character coding, YAM passes it the "????? ?? ????" text instead of the original text.

Just FYI, this did not occur in YAM 2.4.


Moved from SF:
https://sourceforge.net/tracker/?func=detail&aid=1972552&group_id=13560&atid=113560

Attachments (0)

Change History (17)

comment:1 Changed 4 years ago by damato

I think I see what's happening: codesets.library sees that the text cannot
be converted into something compatible with my locale.

If that's the case then this is a big problem. I don't want to switch my
system locale every time I get an email in a different language. Also, I
can't see MorphOS or AmigaOS supporting all languages any time soon, the
only way I can access the text will be to save it as raw from YAM, base64
decode it and then load into an appropriate viewer.

Is it possible to ask codesets.library if it can convert the text or not;
if not, give the user the option of saving the mail in the original coding
(i.e. base64 decoded, headers removed etc) ?


Moved from SF. Original poster: hill28

comment:2 Changed 4 years ago by damato

  • Priority changed from major to undecided
  • Severity set to major

comment:3 Changed 4 years ago by damato

  • Component changed from stable build to codeset handling

comment:4 Changed 4 years ago by damato

  • Cc yamos-svn@… removed

comment:5 Changed 4 years ago by damato

  • Priority changed from undecided to normal
  • Status changed from new to pending

Can you please supply the mail in question that is base64 encoded and causes the issue to appear so that we can reproduce the issue on our own machine and think about a possible solution?

comment:6 Changed 4 years ago by emptystate@…

I was the reporter of this bug.

I just checked the behaviour of this problem in YAM 2.6 and I need to revise the description a little.

  1. I receive an email in UTF-8, encoded as Base64.
  2. YAM displays ????? for glyphs in the message (glyphs for this coding are not available so this is OK).

3a. Saving raw text results in Base64 plus headers, as expected.
3b. Saving the displayed text results in a "???? ????" type message (not UTF-8).
3c. If I hit reply and then call an external editor, the text passed is the "???? ????" type message.

When I receive a message coded in this style I have to save in raw format, edit to remove the email headers and decode the Base64. This is functionality already existing in YAM.

I think the correct behaviour should be:

3b. YAM decodes the message from Base64 and saves UTF-8.
3c. YAM passes the UTF-8 message to the external editor instead of the processed version.

I am sure you can easily manufacture your own test message, but let me know if you still need test data.

comment:7 Changed 4 years ago by damato

  • Milestone set to YAM 2.7
  • Status changed from pending to accepted
  • Summary changed from Handling of charsets to Handling of charsets (e.g. UTF-8) when using external editor.

Ah ok. now I get it. So the problem is that YAM does not pass plainly pass the UTF-8 mail to an external editor. However, we have to find a general solution for that because there might be users without a UTF-8 capable editor and thus they don't want to have 3-byte characters in their external editors. So we have to think about it once more. But I admit that the current solution is suboptimal.

comment:8 Changed 4 years ago by damato

  • Reporter changed from hill28@… to emptystate@…

comment:9 follow-up: Changed 4 years ago by emptystate@…

thus they don't want to have 3-byte characters in their external editors

UTF-8 encoded text may have 1, 2, 3 or 4 bytes per character.

there might be users without a UTF-8 capable editor

How can YAM know what software I have and what it is capable of ? Surely external software will have to decide for itself if it can handle the input ?

comment:10 in reply to: ↑ 9 Changed 4 years ago by damato

Replying to emptystate@…:

there might be users without a UTF-8 capable editor

How can YAM know what software I have and what it is capable of ? Surely external software will have to decide for itself if it can handle the input ?

YAM cannot know that. That's why we have to think carefully about that and eventually introduce an option or possibility to enable/disable the new functionality as soon as we have implemented that.

comment:11 Changed 3 years ago by damato

  • Description modified (diff)
  • Milestone changed from YAM 2.7 to YAM 2.8

comment:12 Changed 20 months ago by damato

  • Milestone changed from YAM 2.8 to YAM 2.9

comment:13 Changed 13 months ago by damato

  • Owner changed from somebody to damato
  • Status changed from accepted to assigned

comment:14 Changed 11 months ago by damato

(In [6886]) * YAM_COg.c, YAM_CO.c, misc: renamed "readCharset" variable to "localCodeset"

to make its purpose more clear. Also moved its GUI config items to the
"FirstSteps" config page and moved the external editor setup options to
"Misc". Added a possibility to specify a codeset to which text should
be automatically converted to when starting an external editor. If not
set the currently configured codeset of the write window is used or the
global one set in YAMs' configuration. This should address one item of
a long standing ticket and thus refs #7.

comment:15 Changed 11 months ago by damato

(In [6936]) * YAM_RE.c, mime/uucode.c, mime/base64.c, mime/qprintable.c: modified mail

file decoding routines to always decode to UTF8 instead to the local
charset. This results in the temporary mail files to be always encoded
in UTF8 and then recoded to the local charset only when loading the text
in a texteditor object. This refs #7.

comment:16 Changed 11 months ago by damato

  • Resolution set to fixed
  • Status changed from assigned to closed

(In [6956]) * YAM_RE.c, mui/ReadMailGroup.c, mui/AttachmentObject.c, mui/WriteWindow.c:

reworked RE_DisplayMIME() function and referenced code areas to actually
use the new Codeset information in each MIME type. Now YAM will automatically
convert the underlying MIME part data from UTF8 to the MIME-type related
codeset if the mail part in question is actually a part which contains
printable text and is thus encoded in UTF8 accordingly. This should finally
fulfill a long standing ticket and make it possible to set e.g. UTF8 as
the target codeset and YAM will not convert it at all. This closes #7.

comment:17 Changed 7 months ago by damato

  • Release Notes modified (diff)

Add Comment

Modify Ticket

Action
as closed .
The resolution will be deleted. Next status will be 'reopened'.
Author


E-mail address and user name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.

This list contains all users that will be notified about changes made to this ticket.

These roles will be notified: Reporter, Owner, Subscriber

  • emptystate@…(Reporter, Participant)
  • Jens Maus(Owner, Participant)