close
Comments you submit will be routed for moderation. If you have an account, please log in first.
Modify

Opened 4 years ago

Closed 3 years ago

#210 closed enhancement (fixed)

Improved UIDL support and import.

Reported by: opiopi Owned by:
Priority: normal Milestone: YAM 2.7
Component: TCP/IP interface Version: nightly build
Severity: major Keywords:
Cc: OS Platform:
Blocked By: Blocking:
Release Notes:

Description

I test the new UIDL support in YAM for some time now and
as result i have 9 .uidl_... files in the YAM directory.
It seems all files contain the uidl's for the latest
loaded mails. I have only one account configured which
should not delete the E-Mails on the Server.

So here my request:
YAM should only write the .uidl_ files if 'Delete mail on server'
is *not* enabled. I see no reason why YAM should write the .uidl_
file when this option is enabled.

Such .uidl_ file(s) produce a lot unnecessarily overhead.

If YAM is getting mails next time then it load the .uidl_ files
for every account and check the content which must fail because
the mail(s) are already deleted on the server.
So here YAM loads 9 files but really necessary is only one.

A check for such file before getting mails is ok because
the configuration may have been changed and the user now want
to Delete the mails on the server.

If the .uidl_ file is finally empty (contain only the header)
then YAM should delete the .uidl_ file too.

Summary: YAM sould only remember uidl's for mails which are
really still on any server.

Additional a improved import function for the old .uidl file
would be nice too. Before YAM 2.7 is going into the wild
we should think about a way to get rid of the old .uidl file.
Otherwise YAM 2.7 and all later versions would still check for
the old .uidl file. Now we have the chance to change that.

A possible way is maybe at the first start of YAM 2.7 to
check the configuration and copy the old .uidl file into
all new .uidl_... files which has 'Delete mail on server'
not enabled and delete the old uidl. file afterwards.

Next time YAM is getting mails from this account(s) it
should delete not valid entries from the new .uidl_
file anyway and if it's ends in an empty file delete
the file. (see above)

Attachments (0)

Change History (8)

comment:1 Changed 4 years ago by tboeckel

The reason why YAM now keeps the UIDL even of mail which have just been deleted is quite simple. With gmx.de I had the problem that, although YAM sent over the DELETE command to the server, the mails sometime were not deleted when the download was aborted because of a broken connection. The next time I had to download the same already downloaded and already deleted mails again, because the former implementation removed the UIDLs as soon as YAM successfully sent the DELETE command. Now the UIDLs are kept as long as it is absolutely clear that the corresponding mails no longer exist on the server and can be savely removed from the UIDL database.

You say YAM checks something which must fail. I say YAM must check this, because personal experience proofed the opposite. Fact is: successfully sending a DELETE command is no guarantee that the mail will not appear again the next time.

I don't understand why you bother with the per account .uidl_#? files at all. How much overhead do the files really produce? Of course we can implement something to delete really empty (except the header) .uidl files. This is quite easy to do. But I will not revert the implementation to the former state. You could also argue that YAM keeps an empty .findex file in each folder, even if there are no mails in that folder. But a missing .findex file is worse than a known to be empty .findex file. From my point of view the same applies for the .uidl files.

Some days ago you complained that YAM must make absolutely sure that the .uidl file(s) must contain valid data. Under no circumstance it must happen that already downloaded but not deleted mails may vanish from the database, because downloading tons of mails across slow internet connection makes you go crazy. I can tell you, even if I don't use a standard modem anymore my internet connections is about as slow as yours, because I live right in the center of one of those white spots on Germany's DSL map.

Now YAM does this kind of housekeeping by remembering all mails, even the deleted ones, just to be absolutely sure. Currently the server can ignore any delete request and YAM will still downloaded the mails only once. This may sound like paranoia and a very big overhead. But I can assure you, this paranoia is necessary, maybe not for your provider, but for mine. And the overhead is minimal. How large are your .uidl files? How many (deleted) entries do they contain? How many mails do you get with each download from each account? Numbers please!

comment:2 Changed 4 years ago by opiopi

Sorry but i have 8 different (active) providers and no one
makes such problems but i have always a fast non broken
(DSL) connection and therefore i don't think about such case.

But it seems the implementation in YAM is a little wrong:

If the YAM send a 'DELE ...' command *and* receive a
'+OK message ... marked deleted' *and* send (later) a
'quit' command *and* receive a '+OK ...' then the
server enter the update state and must delete the mails.

So if YAM not receive both '+OK's then it should of course
write a .uidl file to avoid to load that mail again.
In all other cases (dele *and* quit was successful) the
.uidl file is really not needed.

If the GMX Server does not work like that then the GMX
server is broken and GMX should be informed about that.
Otherwise YAM should check the received messages do the
necessary things depending on the messages. (write the
uidl file or not). It's IMHO a better way as always
write the file.

BTW:
I cancel my GMX account long time ago because they send
too much (own) spam.

I don't bother with the per account .uidl_#? files but i
i try to avoid unnecessary overhead.

Because you want numbers:
I have currently 9 new .uidl files with a summary size of
20295 bytes (one of them is 18339) and the old .uidl file
with 18189 Bytes. It's not so much and changed every time
i get mails.

comment:3 Changed 4 years ago by tboeckel

The problem remains. It is well known that some servers don't answer a "QUIT" command with "+OK", but simply drop the line and assume the connection to be closed. I am referring to this entry in the Changelog:

2007-05-03 Jens Langner <Jens.Langner@…>

  • YAM_TR.c: the workaround for the unexpected connection drop warning wasn't completly working as it seems that the broken SMTP server implementations in question did accept the 'QUIT' command but dropped the connection immediately without replying with a positive status code. Now YAM will not warn for such unresponded QUIT command sequences and therefore should work as expected for those ugly, damn broken SMTP servers like 'smtp.gmail.com' and/or 'smtp.googlemail.com'.

From my point of view it is more safe to save the UIDL database in any case, even when the server is known to behave correctly. Upon the next download the database will be cleaned up and no longer existing entries will not be written out to the database file.

The overhead of reading the .uidl file is really minimal. YAM spends far more time on receiving the UIDLs of mails currently residing on the server than on reading that .uidl file. The file is read long before YAM tries to connect to a specific server. The status message "Checking UIDLs to avoid duplicates..." will be displayed while YAM is receiving the current list of UIDLs from the server. Just in case you think that this might be related to the parsing of the .uidl file, it is not related in any way.

comment:4 follow-up: Changed 4 years ago by opiopi

Jens was talking about SMTP Servers but we are talking about POP3 Servers.

I took the trouble and looked in the rfc's:
For SMTP look at:
http://tools.ietf.org/html/rfc5321#section-4.1.1.10
the important part is:

...
The sender MUST NOT intentionally close the transmission channel until
it sends a QUIT command, and it SHOULD wait until it receives the reply
...

So the Servers are not broken because of the word 'SHOULD'.
You should know what 'SHOULD' does mean. ;-)
At least it means not 'MUST'.

For the POP3 QUIT command in TRANSACTION State see chapter 6 at:
http://tools.ietf.org/html/rfc1939#page-10

But here a some other interesting pages about the QUIT command and how a server have to respond to commands at all.
After reading all pages it is still valid what i said before any YAM should only write the UIDL file if it's not receive a response for the QUIT command or YAM doesn't send the QUIT command because of a broken connection or whatever.

To test the GMX Server i also create a gmx account and it's work like expected:

SERVER[0033]: +OK POP server ready H migmx003
CLIENT[0021]: USER xxxxxxx@gmx.de
SERVER[0049]: +OK password required for user "xxxxxxx@gmx.de"
...
SERVER[0003]: .
CLIENT[0008]: DELE 3
SERVER[0005]: +OK
CLIENT[0006]: QUIT
SERVER[0028]: +OK POP server signing off

About te overhead a can say in the worst case YAM has to check here
at least 11 (now 12) files and has to update the same amount of files.
I would say producing such unneeded overhead is not 'really minimal'.
Additional a backup program has to do the same things again.

BTW (off topic):
Another issue is the QUIT command in the AUTHORIZATION State:
In chapter 4 (Page 4) http://tools.ietf.org/html/rfc1939#page-4
you can read:

After returning a negative status indicator, the server may close the
connection.  If the server does not close the connection, the client
may either issue a new authentication command and start again, or the
client may issue the QUIT command.

Currently YAM sends the QUIT command also if the connection is closed
but should not do that. YAM should send the QUIT only if the Server does
*not* close the connection. (or send a new authentication command).
A possile solution is also the absence of a reply not consider as error.

SERVER[0065]: +OK hello from popgate 2.46.7 on pop102.plus.mail.re2.yahoo.com
CLIENT[0016]: USER xxxxxxxxx
SERVER[0024]: +OK password required.
CLIENT[0016]: PASS xxxxxxxxx
SERVER[0035]: -ERR [AUTH] invalid user/password
CLIENT[0006]: QUIT
SERVER[-001]: 

and show 2 errors:

Der Postserver 'xxx' von Konto 'xxx' hat auf den Befehl 'PASS' falsch geantwortet:
-ERR [AUTH] invalid user/password

and

Der Postserver 'xxx' von Konto 'xxx' hat auf den Befehl 'QUIT' falsch geantwortet:

A user may only see the second error in the error window and does't recognize
the first error which show the true cause for the failure.

That is not so important but should be fixed too IMHO.

For completness I test it on 7 different servers (using a wrong password) and
only one ('pop3.arcor.de') doesn't close the connection and respond with a:

CLIENT[0016]: PASS xxxxxxxxx
SERVER[0020]: -ERR Login failed.
CLIENT[0006]: QUIT
SERVER[0028]: +OK Better luck next time.

Nice answer IMHO :-)

comment:5 in reply to: ↑ 4 Changed 4 years ago by tboeckel

Replying to opiopi:

BTW (off topic):
Another issue is the QUIT command in the AUTHORIZATION State:
In chapter 4 (Page 4) http://tools.ietf.org/html/rfc1939#page-4
you can read:

Sorry, but you are contradicting yourself here and even provide the perfect counterexample why YAM should send the QUIT command in any case.

In your first example session with pop102.plus.mail.re2.yahoo.com the server responds "-ERR" and silently closes the connection itself, thus YAM's QUIT command is not replied.

In your second example session with pop3.arcor.de the server also responds with "-ERR", but does not close the connection itself, otherwise it should not be able to respond to YAM's QUIT command with "+OK Better luck next time.".

Looking at your examples there is no indication if the connection will be closed by the server or not. Thus the final QUIT by YAM is perfectly ok from my point of view. The only difference is that YAM should not throw an error message in case the QUIT command is not responded to by the server. The next nightly build will ensure this.

comment:6 Changed 4 years ago by opiopi

My examples show only what YAM does and it show not that YAM
should the 'QUIT' command in any case.

If YAM has no possibility to find out if the connection is closed or not
and it's easier to send a 'QUIT' command against a closed connection to
find out that the connection is closed then it should be OK to do so.

The main point is to not show an error for this case, which seems to be
done now. Thanks for that.

BTW: IIRC a way to find out if a connection is closed was if recv()
response with 0 or -1 (i dont know anymore). But the way YAM goes should
do the same without a possible timeout from the socket.

comment:7 Changed 3 years ago by tboeckel

  • Status changed from new to pending

Can we assume this issue to be fixed then? The false error message due to the possibly missing answer of a QUIT command is definitely fixed. And what about the .uidl files? Do they really cause such untolerable performance penalties as you think? Is the overhead of reading and writing the files noticable at all?

comment:8 Changed 3 years ago by tboeckel

  • Resolution set to fixed
  • Status changed from pending to closed

No further reply, closing this one.

Add Comment

Modify Ticket

Action
as closed .
The resolution will be deleted. Next status will be 'reopened'.
Author


E-mail address and user name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.

This list contains all users that will be notified about changes made to this ticket.

These roles will be notified: Reporter, Owner, Subscriber

  • Frank Weber(Reporter, Participant)