Ten Commandments of How to Write an IMAP Client (2006)
washington.edu
washington.edu
Basically, all messages in a mailbox are numbered from 1 to N, and you can reference a message by number, and when a message is deleted all message numbers above it shift down by one. But since this could confuse clients that are sending inflight requests, you can't start translating from the new shifted-down-by-one numbers until you've told your clients about it, which could take a theoretically unbounded amount of time.
One implementation of this I've seen is to get a list of every UID in a mailbox at time of selection, and then update that as clients are informed of new or deleted mail. Needless to say, that's a bit problematic with short-lived or multiple connections.
So if your mailbox gets too big, too bad. This ends up complicating the client: probably influencing rules 1, 3, part of 5, and 8 listed here. (1) You can't have concurrent mailbox access because it might mess up MSNs reified to disk, (3) you can't have connections live too long because they passively eat up RAM with queued updates and a stored MSN list, (5) there are perfectly good UIDs to reference messages with that you have to learn alongside message sequence numbers, and (8) because opening a new session can be heavy.
There is no reason technical why this shifting has to happen. If there are ten messages on session creation, and message UID 3 is deleted, you then have UIDs 1-2,4-10.
If the server wishes to shift messages over it can, but it must bump UIDVALIDITY, which tells clients that there is a 'new generation' of UIDs for the given folder. But as long as the server's index of UID-to-message is not lost, there's no reason why renumbering has to occur.
It's no different than having an AUTO_INCREMENT int column in MySQL: you just keep going up without bothering to fill in any 'holes'.
That said, folders and messages can now be assigned UUIDs:
It is pretty much impossible to write a modern mail client relying on message sequence numbers, since there are too many ways the bookkeeping go wrong; so everyone uses UIDs (or UUIDs) nowadays wherever possible.
The need for a better protocol is why JMAP exists: https://jmap.io/spec-core.html
Although I see their point about non-REST, I think there's still a case for having a REST flavour available even if it may not be the most efficient.
I wonder if anyone's tried mail delivery over git repo yet.
This phrase made me laugh, because UW imapd was largely written by Mark Crispin, who invented the IMAP protocol!
It seems that in about 2007, Crispin forked UW IMAP to create his own IMAP server, "Panda IMAP" [1], which at least one website claims to be "fully compliant" with the IMAP RFCs while UW IMAP is not. [2]
I was pretty disappointed when I found out how ugly are JMAP requests. I'd also like to see REST variant as messages and their parts should be easily mappable to resources. Too bad SMTP was not also included in JMAP. ActivityPub comes really close to SMTP over HTTP/JSON in my opinion.
Message ids are monotonically increasing, but not necessarily consecutive. You may need to renumber when you hit the 32-bit ceiling, but that is much less frequent than what you describe. Worst case scenario, you can disconnect unresponsive clients when renumbering, clients behave well on disconnects.
This invalidates your conclusions.
> (1) You can't have concurrent mailbox access because it might mess up MSNs reified to disk
You can. Renumbering is not frequent.
> (3) you can't have connections live too long because they passively eat up RAM with queued updates and a stored MSN list
My dovecot IMAP processes use an average of 5MB reserved memory, of which 4MB is shared memory, so a server with 100GB memory can hold close to 100k connections.
> (8) because opening a new session can be heavy.
Monitoring tells me that on loaded production servers, session open + select + list + logout occurs comfortably in less than 100ms. That's not heavy.
Dovecot starts to fail on mailboxes with about a million messages. It's a limit I can live with. Users have learned to archive messages by date.
I'm not familiar with Dovecot's code or very good at reading C, but I think part of how they handle it is here: https://github.com/dovecot/core/blob/81b5b188c478ec36bea8bda...
Basically, building a mail index for UID<->MSN translation that can serialize to disk, which would explain how it doesn't necessarily need absurd amounts of RAM and can select fast, probably at heavy cost when EXPUNGeing old messages. I'm sure there were worse implementations that did just maintain it all in RAM way back in the day, and still some that do (like the one I ran into).
That does explain why indexes can get corrupted and need to be cleared out, and why load balancing IMAP can get tricky.
They're somewhat obscure though.
#7 makes me sad. Evolution supports a 'virtual trash folder' perfectly, but since I also delete mails in other clients, my mailstore is polluted by a malingering selection of folders with names like Trash, Wastebasket, Deleted Items and so on. The same remark applies to a 'virtual junk folder' and malingering Spam, Junk folders.
The 'All About Calendars' file is a wealth of interesting tidbits about time: https://www.washington.edu/imap/documentation/calendar.txt.h...
The short answer is that you can access from multiple device. The long answer is, it depends on the server datastore implementation. Namely, any server using mbox will probably lock the mbox file to the first connection, and hence allow one connection per mailbox (folder).
Most large scale servers won't use mbox, though. The problem is described, in the original article, from writing an IMAP client perspective. You can't assume _all_ servers will allow multiple connections, even if most do.
[edited]
I believe that's the case. The ancient and venerable UW implementation of the ancient, venerable, and awful Unix "mbox" format from the UW server has a lot of these sorts of problems.
The UW server did not have this problem with some of the other formats. It suffered a lot from supporting a lot of different formats in borderline hostile conditions (i.e., users were assumed to have filesystem access to the server so no improvement to locking or caching was practical).
None of the current commercial implementations suffer from this, but I suppose I can say that the lowest common denominator is still pretty low.
I may be overstating this because what I'm describing would also outlaw message delivery, which I don't think is the case. I think one of the formats did have problems with trivial concurrent access but I was never a UW-imapd expert. I am sure that the mbox format has problems with EXPUNGE (removing messages flagged deleted) as well as many other issues.
I guess it's also worth reiterating that STATUSing a SELECTed mailbox (one that's open in another connection) is a pretty big waste of server resources, as someone else noted.
I think they are talking about one particular client instance. In other words, you should have some data structure where you can look up a connection by mailbox and use that existing connection rather than opening a new one.
"Hello IMAP server. I'd like to deal with the message with this UID."
"The UIDVALIDITY has changed since then."
"Can you tell me the new UID from my old UID perhaps?"
"No."
* https://tools.ietf.org/html/rfc8474
Though if your mail server loses the UID-to-message mapping index and has to rev the UIDVALIDITY, that's a more a problem with the implementation IMHO.
Oh no! Before the response came back, my connection to the server died. I don't know if my request arrived and was accepted.
Time passed and I've just now managed to reconnect. I need to know if the delete happened or not.
So I'm working on an app that uses IMAP. I need to know which emails I've already processed. Has anyone ever used the message ID header to identify an email as unique/seen? Or do you hash the email in its entirety? Or is there some other way to do it?
I guess what I'm still confused about is determining how to handle a UID validity change. So I'm afraid to use UIDs.
MessageID headers are often missing, even though that is "invalid". I've also got thousands of examples of messages with identical MessageID values. (Largely from automated systems that use some trivial/broken template to generate the emails.)
I needed this when i was working on my engineering project, and added 'processed' flags for already processed items. This way i could query all unprocessed items with "UNSEEN UNKEYWORD <flag>".
* https://tools.ietf.org/html/rfc3501#section-2.3.2
UIDS are more than simple 'message numbers'; they are part of a tuple (Folder, UIDVALIDITY, UID). It may be easy to think of UIDVALIDITY as a kind of 'generation number': for a given folder, as long as the UIDVALIDITY has not changed/incremented, the UID-to-messege has to be the same per the spec.
Though recently the OBJECTID extension has been published:
A UIDVALIDITY change should blow your whole cache. In practice they are rare.
Thank you for saving me from the heartbreak of finding this out later. Seriously you just saved me a ton of time.
>A UIDVALIDITY change should blow your whole cache. In practice they are rare.
So even in a huge pile of mail, everything would have to be reprocessed. Since it's so rare, I'm definitely going to set up metrics to monitor when it does happen. Do you have any insight on how people store the mail? Do they actually use the UID to reference it or is it used as more of a cursor?
At one point I had wrappers around Python's IMAP/SMTP implementations that I excuted from Go, since at least Python had 20 years to make theirs work properly.
Any other implementation I tried failed somewhere along the way, especially as soon as you start moving messages around in folders or anything else beyond simply fetching messages.
SMTP is simpler from my experience, but then so is the work it performs.
I even played around with the idea of cooking my own to get out of this loop, but reading the specifications soon killed that idea. It would take a life time, probably be deprecated by the time, and still wouldn't work properly.
The new JSON-stuff honestly doesn't look much better from what I've seen. It seems to still be doing IMAP, only using JSON instead, which wasn't the problem. I'm not holding my breath about it becoming standard enough to depend on.
It's a mess, so much that I'm thinking it would probably be better to start over entirely.
Essentially, the single biggest issue with IMAP is that most clients nowadays maintain an offline shadow copy of the database, and so what they need to do is use IMAP to synchronize these two databases. And IMAP is a lousy synchronization protocol--most of your IDs are all ephemeral. Extensions to IMAP make it better, but you can't rely on servers implementing them (or clients using them!). Incidentally, that's also why most library implementations of IMAP suck: you can't build a good interface that works around server deficiencies without baking in the integration into your database model, and that heavy opinionation makes it hard to use the IMAP code outside of the full email application.
regarding JMAP that should not be the case. From their FAQ the intent is to make it easier to use already common workflows.
It irks me to no end, and un-checking that accursed options box in the IMAP folders dialog is something no user should ever need to do to find their "hidden folders".
Its not even a good email client when used against Exchange but its 'good enough'