close
Comments you submit will be routed for moderation. If you have an account, please log in first.
Modify

Opened 10 months ago

Closed 10 months ago

Last modified 9 months ago

#466 closed bug (fixed)

NB 27/11 takes very long to open a msg with big attachment

Reported by: JosDuchIt Owned by: tboeckel
Priority: undecided Milestone: YAM 2.9
Component: MIME handling Version: 2.8p1
Severity: major Keywords:
Cc: OS Platform:
Blocked By: Blocking:
Release Notes:

significantly improved performance of parsing emails with large attachments. Now YAM will reuse the size parameter of each MIME part to speed up memory allocations during MIME parsing.

Description

Summary

When trying to open a msg with a big attachment 7.4 MB pdf file, nothing appears in the
Msg listview, and Yam seems to be frozen, other applications run very slowly, Snoopy as well. The folders dragbar can be used, but nothing moves.
The Yam window is not redrawn when you move other windows over it
The freeze reported in #451 was probably due to also clicking in a folder.

Steps to reproduce

  1. wait till the message is open
  2. chose an small one
  1. chose the same message again (or an other big one)

Expected results

Opening the msg text should not depend on attachment size

Actual results

as reported above

Regression

Notes

Attachments (0)

Change History (17)

comment:1 Changed 10 months ago by JosDuchIt

iThe problem is about the NB 17/11 not 2.8p1, forgot about the version selector

comment:2 Changed 10 months ago by damato

As usual, we need more information before we can identify the problem. So please answer:

  1. does it ALWAYS happen with that email or was that only happening once and now it works?
  2. if it always happens please send us (in private mail) a copy of that particular email (the raw msg file) so that we can investigate ourself.

comment:3 Changed 10 months ago by JosDuchIt

I am typing this while the same YAM session is going on:

  • The attached files (icons) .pdf etc can be seen immediately, the upper part of the message from/to and content can not be seen.
  • after the litigious msg is open i try to open an other big one

again an empty msg LV
you can have the dragbars moving the listviews, and the redrawing is OK ( in the same yam session that is)
I click the "read" button , it brings us back to the reported problems (redrawing included)
it does not seem related to just one file.
The bigger the fiile the longer the wait

comment:4 Changed 10 months ago by JosDuchIt

Please send me a big attachment and i'll comment on that

comment:5 Changed 10 months ago by damato

Sorry, but you are not answering my questions but talking about other things. So please, again, answer the questions I listed above and if the problem always occurs with the SAME mail, then please send US that mail in private mail.

comment:6 Changed 10 months ago by JosDuchIt

Sorry i fail to see what question i did not answer ( i got your question while typing the text below)

  1. after the litigious msg is open i try to open an OTHER big one
    • again an empty msg LV
  • you can have the dragbars moving the listviews, and the redrawing is OK ( in the same yam session that is)
  1. after the litigious msg is open (and that wait is over) ii try to open an OTHER BIG one
    • again an empty msg LV
    • you can have the dragbars moving the listviews, and the redrawing is OK ( in the same yam session that is)
    • click the "read" button , it brings us back to the reported problems (NOT redrawing icorrectly the YAM window ncluded)
  2. Conclusions
    • it does not seem related to JUST ONE FILE
      • The BIGGER the file LONGER THE WAIT

In the Ticket presentattion i said

The freeze reported in #451 was probably due to also clicking in a folder.

I do clarify in the above that during the wait reactions as seen by the user, namely

  • Yam dragbars not responding
  • redrawing of YAM window is not done immediately when an other one has moved over it, it will be done only when the message text appears

are not always effectively seen:dragbars may respond, moving windows over YAM may not interfere with YAM's redrawing
What was seen also was that other YAM actions , than "clicking in a folder" reported in the Ticket description , and here "hitting the read button" may worsen YAM's reactions

  • initiate the redrawing problem
  • generate a freeze (see the 451 ticket's last crashlog)

I am awaiting instructions that do take this feedback into account:
Sending you the initial big file is not an option as it is a private one, That's why i asked you to send me a big file. Even if i did not see from my analysis (above) that the raw msg file could give you the clue.

Considering i was wrong once in assuming my analysis was right in a previous ticket, i take indeed into acount that it might.be wrong again.

I am not receiving such big msgs every day. So if you still want a raw msg file, please send me a msg with big attachments.

I infer maybe wrongly that you don't observe the problem with big files on your system, as i do here
Here attachmenst of 2 MB allready show a slow down, 5 MB is serious wait, 10 MB will be seen as a freeze or probably generate a freeze with some gadget manipulations. Abnormal race conditions may explain that behaviour.
I can also send you a big msg,

Just tell me what to do
.

comment:7 Changed 10 months ago by tboeckel

I think the problem is quite obvious. Many mail applications don't include the size of an attachment in the headers. Since we switched YAM from immediate temporary files to dynamic buffers when parsing a mail YAM has to constantly enlarge this dynamic buffer as more data is read. This includes copying the old data to the new buffer. All in all this is much overhead which unfortunately cannot be avoided. This is why I added support for the attachment's possible size information in r7160 to preallocate a large enough buffer to keep the complete encoded attachment and to avoid any further reallocation. But this doesn't help for attachments without this size information.

To make a long story short: be patient when dealing with large attachments and please stop mixing unrelated tickets.

comment:8 Changed 10 months ago by tboeckel

  • Component changed from undefined to MIME handling
  • Milestone set to YAM 2.9
  • Resolution set to invalid
  • Status changed from new to closed
  • Version changed from 2.8p1 to nightly build

comment:9 Changed 10 months ago by tboeckel

  • Resolution invalid deleted
  • Status changed from closed to reopened

Having thought about this issue another time I came to the conclusing that there is a possible solution to speed up the parsing of large attachments without size information. However, this solution comes at a cost. YAM will do a rough estimation of the possible size of the attachment. This estimation will always be too large, depending on the number of attachments it may be far too large, but on the other hand it avoids the need to constantly reallocate memory for the growing buffer.

comment:10 Changed 10 months ago by tboeckel

  • Owner set to tboeckel
  • Status changed from reopened to assigned

comment:11 Changed 10 months ago by tboeckel

  • Resolution set to fixed
  • Status changed from assigned to closed

(In [7361]) * YAM_RE.c: for attachments without an explicit size information by the header lines YAM will now do a VERY rough size estimation to be able to preallocate a buffer. Although this estimation might be far too big if there are several attachments it nevertheless speeds up parsing big mails/attachments a lot as this avoids the need to dynamically resize a constantly growing buffer for reading. The former approach required almost twice the size of an attachment as dynamic buffer, so this estimation shouldn't be too bad. This closes #466 again.

comment:12 Changed 10 months ago by tboeckel

Please let me add again that ticket #451 is absolutely unrelated to this ticket. #451 is about overwritten filter settings while this ticket is about slow mail parsing. It should be very obvious, even for the untrained user's eye, that there is no correlation at all.

comment:13 follow-up: Changed 10 months ago by JosDuchIt

Status is changed from reopened to assigned

Glad to hear that.
I had just decided to leave the 2.9dev alone.

this closes #466 again.

and i utterly regret that. Did you have complaints for the 2.8p1 method ? With the habit of people throwing big attachments (more than 2 Mb) at you, the new parsing method make YAM2.9 unusable. I am not exagerating. Give the user at least a parsing choice in settings.

The new problem is not only on receivng a big msg once. if it is the first mesg in "incoming" and all msgs read, YAM will try to open it again. Every time you want to verify its content, make a quote, or whatever, its content again, same problem
If you try to forward it, problem seems even worse (msg window blanks out)

And i really don't get it When you have read the msg file till the beginning of the first attachment you have all the info needed to present the mesage in the main window's msg listview or in the reading or forwarding window.
The msg bodies rarely are more than su 10 to 100 kB
The parsing of the rest of the file is only needed when someone wants to open an attachment.

I would suggest to have a good nights sleep and come back to it then, ask for a second opinion, start an ithread on the respective merits in the YAM forum here, or elsewhere, or discuss the possible solutions in this thread.

comment:14 in reply to: ↑ 13 Changed 10 months ago by damato

Replying to JosDuchIt:

and i utterly regret that. Did you have complaints for the 2.8p1 method ? With the habit of people throwing big attachments (more than 2 Mb) at you, the new parsing method make YAM2.9 unusable. I am not exagerating. Give the user at least a parsing choice in settings.

You should have read closer. Thore alread committed a change to the MIME parsing routines which will potentially improve performance with such emails you are having problems with. So please retry with the next nightly build and report back if things improve.

And i really don't get it When you have read the msg file till the beginning of the first attachment you have all the info needed to present the mesage in the main window's msg listview or in the reading or forwarding window.
The msg bodies rarely are more than su 10 to 100 kB
The parsing of the rest of the file is only needed when someone wants to open an attachment.

Well, it's not that easy and you probably simply don't know how a mail is encoded. The problem is, that yam HAVE TO load the whole message file since it has to identify if there is more than one readable MIME part in the message than just the main body part. In addition, it has to load the whole message since it has to identify the number of attachments a mail contains. Unfortunately all this information is not encoded in the message header. That's why YAM have to load and parse the whole message no matter how large it is.

But be rest assured that we always try to improve the performance of critical sections in YAM. So if there is anything we can do about improving the parsing performance of YAM we will do so.

That said, please retry with the next nightly build coming this night since I believe this should already improve performance quiet significantly.

comment:15 Changed 10 months ago by tboeckel

(In [7362]) * YAM_RE.c: added a detection of real attachments to be able to perform the rough size estimation for these parts only. All other parts without size information will be read dynamically. This refs #466.

comment:16 Changed 9 months ago by damato

  • Release Notes modified (diff)

comment:17 Changed 9 months ago by damato

  • Version changed from nightly build to 2.8p1

Add Comment

Modify Ticket

Action
as closed .
The resolution will be deleted. Next status will be 'reopened'.
Author


E-mail address and user name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.

This list contains all users that will be notified about changes made to this ticket.

These roles will be notified: Reporter, Owner, Subscriber

  • Joseph Duchâtelet(Reporter, Participant)
  • Thore Böckelmann(Owner, Participant)