Gluon, a high-performance IMAP library
proton.me
proton.me
> IMAP servers (which include applications such as the Proton Mail Bridge) need to be able to handle this situation. When one client moves messages into or out of a mailbox, the server needs to notify all other clients of the changes so that they can update their own view of the mailbox. And until the clients have received the update, the server needs to remember what each client thinks the mailbox looks like to correctly interpret the client’s commands.
Holy F what a clusterfuck
Using UIDs is opt-in, by prefixing `UID ` in front of some commands. There are indeed extensions that add more `UID ` commands to the protocol. I've implemented IMAP clients and never used sequence numbers at all.
So you don't need to maintain message sequence numbers, except for the few places in the protocol where only the message sequence number is reported (e.g., the unsolicited message deleted alerts). In practice, I would actually expect that trying to use message sequence numbers would be buggier than not using them. But servers still have to support them!
And that's why when people say "JMAP is pointless", just go back and read some of the email specs!
Of course, from what I understand is there are JMAP/IMAP gateway options, that should ease availability... assuming these things actually gain any traction ever.
The amount of deadlocks in adopting things like JMAP, IMAP NOTIFY, OAuth2 (+ Dynamic Client Registration) and so on and on is immense. Won't because others don't, too few do, we see no benefit yet etc. it's really hindering.
> Holy F what a clusterfuck
Or you could use the unique identifier of the message (UID) instead of the sequence number:
> Messages in IMAP4rev1 are accessed by one of two numbers; the unique identifier or the message sequence number.
* https://www.rfc-editor.org/rfc/rfc3501#section-2.3.1
> A 32-bit value assigned to each message, which when used with the unique identifier validity value (see below) forms a 64-bit value that MUST NOT refer to any other message in the mailbox or any subsequent mailbox with the same name forever.
* https://www.rfc-editor.org/rfc/rfc3501#section-2.3.1.1
The designers of the IMAP protocol aren't as dumb as the above quotation would have you to believe and multi-client operations were thought of.
It's not that new, it's not a hail mary, and it is very well supported.
I last wrote IMAP server-side software in 2000. Even then, I was not aware of any IMAP servers that didn't support UID. It was essential, as most clients expected it to be there.
There might still have been some at that point, but I'd be shocked if anyone is still running any IMAP2 or IMAP3 servers.
But IMAP is pretty twisted in other ways too. IMAP is _not_ a request-response protocol like you might think from other protocols. It looks like it at first because you can say things like (from memory, not the real protocol):
> cmd1 LIST (here cmd1 is a request/response identifier chosen by the client; the server sends responses prefixed by this identifier)
< cmd1 LIST RESPONSE...
> cmd1 LIST folder1
> cmd2 LIST folder2
> cmd3 LIST folder3
< cmd3 LIST folder3 RESPONSE...
< cmd1 LIST folder1 RESPONSE...
< cmd2 LIST folder2 RESPONSE...
But it gets weirder because you can receive _unsolicited_ responses. Sometimes you want that like
> cmd1 WATCH folder1 (starts a watch which may have responses but not immediately)
> (do other stuff)
< * NEW MESSAGE folder1 DATA... (some time later)
< * FLAGS CHANGE folder1 DATA... (much more time later)
> cmd1 OPEN folder1
> (do stuff with folder1)
(much later)
< * CLOSING folder1 WAS DELETED
It's a cool way to do what it wants to do for GUI clients in particular because IMAP and GUIs have the same event-oriented model. But it makes implementing IMAP as a library a little more difficult because you need bidirectional hooks into the containing application. And you can't easily just have a `if POP3 then... else if IMAP4 then...` abstraction layer without some work to make it happen. Heck even just having a function that returns the list of messages requires a background thread and a little bit of lying about what's happening underneath. Leaving data on the socket unconsumed can get you disconnected by keepalive checks so the background task is required if you want to avoid that (which by observing mail.app it seems to just accept the disconnection and reconnect when it happens). Python has an imap library in the stdlib which "avoids" this by being so low-level that it sidesteps some of the issues by making you implement request/response on top of it, and its model also makes perfect IDLE implementation difficult (maybe impossible? but I didn't put enough time into it to prove this).
Because of the complication many GUI clients don't actually support the liveness built in to the protocol. They just ignore any reponses they didn't ask for, don't use WATCH/IDLE, and do the 15 minute polling cycle as if they were POP3. The protocol has all of the tools to have the liveness of Google Docs but I'm not aware of any GUIs that use it right.
The fact that the server may send you messages even they are not responses... of course, otherwise how do you think notifications can work? The only other options is the client to poll the server periodically (how it was done by older clients, and iPhone client...), that is inefficient, consumes resources and you don't get immediate notifications. With an open socket the software stays in the background and gets woken up by the OS when a byte arrives, how is it complex?
> The protocol has all of the tools to have the liveness of Google Docs but I'm not aware of any GUIs that use it right
Never had any problem with Thunderbird. If you have Thunderbird open in the background and the GMail web interface both open at the same time most of the time you see first the notification from Thunderbird than the email in the GMail web interface. It's really fast, to the point that if I send myself a mail from my phone it seems to arrive immediately, and it's quite impressive.
It's not independently difficult per se, it's just mismatched to the way that other protocols often work in a way that, in practise, few clients implement well. Most clients and libraries implement it as if it were request-response and as a result either produce wrong answers or are notably less efficient as you note.
> Every protocol works this way, nothing new.
No? POP3 and HTTP as easy examples are simple blocking request-response. Some do either pipelining or multiplexing with the command-tagging, but don't have unsolicited messages. I can think of only a few other comparable protocols with this paradigm but "every" is just not true.
> of course, otherwise how do you think notifications can work?
Yes? This is indeed _why_ it works this way. I don't know what you're correcting me about. It does this to implement notifications and liveness. We're agreeing.
> Never had any problem with Thunderbird
Thunderbird is one of the better implementers, yes. There are messages that it ignores like "folder deleted" but according to Dovecot https://www.dovecot.org/clients/ it implements IDLE correctly which is uncommon among clients. See https://wiki2.dovecot.org/Clients for more client quirks. The dovecot config file https://dovecot.org/doc/dovecot-example.conf has a imap_client_workarounds setting for many of these common bugs including "delay-newmail" which disables several of the unsolicited message types until the server is confident that the client is actually expecting to receive something.
In practise, in real life, clients get this wrong. That's all I'm trying to say.
Hacker News always has the well-actuallies but I'm pretty familiar with this and have implemented both an IMAP server and client. I'm not sure what you're trying to "correct" me about.
Well HTTP/1.1, HTTP/2 (and HTTP/3) surely not. You can make more request on a single channel and also the server can send you data you didn't requested (PUSH). Otherwise a modern site with a ton of resources would either take a minute to load or have to open 100 connections to the server.
MQTT is another example of protocol that works this way.
> "every" is just not true
Every protocol that was born after the 90s, let's rephrase it like that.
HTTP/1.1 does not allow data you didn't request in random places. You need to specifically request a feed of server sent events. In every other instance you know that the response that come back will be replies to a request (in the case of SSE's, the reply will just keep coming as long as the server sends events), and in the same order as the requests. That means that a "dumb" client can just synchronously send requests and read the reply in a string request/response manner without worrying about getting something else back. Neither of those constraints hold for IMAP.
I don't think it's quite as complex as suggested above, though. You just want to provide access to the socket in case the client wants to select() etc., and offer an API to send a request, a queue to read responses from that allows you to filter by request, and maybe a wrapper that sends a request and waits for a reply matching that request (while putting everything else onto the queue). Then the client can decide if it wants to pretend the protocol is mostly request/response or take proper advantage.
Another option is to open an idle connection dedicated to this and for a server to stop doing what wasn’t asked for. These legacy protocols always have been “all first thoughts slapped together and then some afterthoughts on top of that”.
Kind of like Siri lol
[0] https://en.m.wikipedia.org/wiki/JSON_Meta_Application_Protoc...
I think it is not a great idea to say that it is not a good idea to not implement things that no one uses, because that is a certain way to not make more people use it.
There are some servers that support it such as Stalwart [0], though I don't know about any older servers and/or clients that support it well.
(a) is IMAP good / sufficient / fit for purpose
(b) if not, is there a better alternative available
(c) if there is, is it worth the development effort to support it.
If we assume the answers to (a) & (b) are NO & JMAP, then the benefits are included in those answers & only question remaining is (c). The person I was replying to seemed to be only arguing that the answer to (c) is NO.
If this question made any sense we would never make anything new at all. How do you think new things get started?
I disagree on this, but that's ok that we differ in opinion. Also, i would disagree about the crumbling of matrix - either in the protocol sense or app/client or even implementation sense. Could some areas like clients, UX, etc. improve? Sure, of course! Evolution of software need not end if there's room to improve. I'm an admitted fanboy of matrix, but even i know that its not perfect and i hope for continued improvement of matrix. As great as email has been for decades - warts and all - i simply have been expecting (hoping?) for something else by now. It need not be matrix - though again, i'm a fan - but i just meant i hope for open, decentralized mechanism, protocols that serve the world, and become as solid and de facto as email has been (again warts and all). In 10 years, i also beleieve email will still be around...Now, whether it will be considered as standard and universally-accepted as a means for communications, we'll see i guess.
I presume you're suggesting a new alternative to email rather than Matrix being itself an alternative (which isn't its intent).
The thing about email is it's a set of related protocols - if you want a modern alternative you need to look at those protocols & address legacy problems with them, replacing the ones that need replacing. That's what JMAP is. What you're asking for is JMAP.
JMAP probably isn't perfect, but nothing is & there's no better alternatives I know of. And one major provider using it = more adoption than any other modern alternatives.
IMAP. IMAP is the better alternative.
It’s universally supported and battle-tested. We have great servers and clients for it.
It’s not perfect, but it’s also not fundamentally flawed in any way that might put the future of e-mail at risk.
Bolting HTTP and JSON onto e-mail via JMAP — and requiring that everyone implement an entirely new, larger protocol stack — is not an obvious value proposition.
So the definition of the word "alternative" in this instance is "something other than IMAP" - the argument for JMAP is not competing with IMAP, there's a pre-determined assumption that we're already unhappy with IMAP, hence the search for an "alternative".
It sounds like you disagree fundamentally with moving on from IMAP in the first place, which is fair enough. That's not a debate I'm getting into here: all I'm saying is IFF we are moving on from IMAP, JMAP looks like the best alternative.
I don't know about servers, but I'm still waiting to see a great email client. (For now I get by with Thunderbird, and it's actually going downhill.)
Yes, the intent of my statement was hoping for an alternative to email, and i did not mean to imply that matrix would be the thing that would replace email. Both things seek to achieve different things. and even in some rare overlappiny use-cases, i would say that at this time matrix does not universally replace email.
What i should have stated was that with such attention and (at least in some circles) excitement around newer areas like matrix...That maybe traditional email software devs might have - by now - looked into newer alternatives to email. With apologies to email software devs.
And, yep, I had heard of JMAP some time ago....and i figured by now that it would be far more popular than it is, and that it would have been implemented by many more entities by now. Oh well.
Well IMAP is 37 years old and Google still haven't gotten around to implementing it properly. Email as a technology moves pretty slowly.
For example the (in)famous label thing, why should a letter belong in a single folder. Pointless restriction.
I was under the impression the first public IMAP spec had support for "flags"... did it need later fixing?
Still hope at least both will add push notification to 3th party apps. And a BIG hope for https://stalw.art/
Worth mentioning that Fastmail use an open source backend: Cyrus[0] - which started adding JMAP support in 2015 (though I suspect that was Fastmail devs contributing the feature).
Their primary targets are Microsoft Outlook, Apple Mail and Thunderbird, none of which speak JMAP at present. So Proton has very little to gain from implementing JMAP, since only niche clients support it at this point.
Classic chicken and egg problem, unfortunately.
Why would dovecot want to add an entire HTTP server just to reproduce what they already have, with more layers of poorly fit abstraction, despite there being no significant mail clients that require it?
JMAP, on the other hand, is hardly used, and it doesn't seem to solve any issues that users face, which means demand is never likely to reach critical mass.
The abstractions and libraries available make this easier... if you look at IMAP it's byzantine by comparison.
On the other hand lots of devs know how to work with JSON-over-HTTP APIs and a lot of modern languages even come with JSON serialization/deserialization out of the box, with makes writing quality clients a lot more accessible to a lot more people.
The megacorps are trying real hard to kill off open protocols and switch everyone over to their in house "versions" of a public protocol for making proprietary protocols.
Recently I wanted to use the alpine terminal email client, which does support OAuth2, but I was thwarted by a notice saying "application alpine requesting access; write to your admin to authorise". Several months later, apparently whether to add alpine or not to the list of approved software is still "being discussed". Not exactly holding my breath for it ...
https://explained-from-first-principles.com/email/#internet-...
https://news.ycombinator.com/item?id=34846606
Proton Mail Bridge (for use with something like a desktop Thunderbird on Linux setup) does require the paid tier of Proton, which isn't immediately clear. Unlike Google they're not making money of harvesting and selling personal data, so I don't really mind.
addendum: The IMAP state diagram is a finite state machine, interestingly enough:
https://www.researchgate.net/figure/Finite-state-machine-of-...
As soon as your e-mail leaves your server, it will spawn pions and be screened.
I definitely want to play around with it.
- Nobody reads or interacts with emails anymore, they use threads. Threading is one of the most impossibly hard challenges to solve and requires a thread-first view of the inbox.
- IMAP represents now a small minority of servers. ProtonMail may be embracing it but for actually interacting with mailservers, don’t expect good interoperability or fast interactions when working with the two big dogs (Exchange and GSuite)
- Similarly most mailservers don’t act in good faith, their rate limits for non-internal clients are ridiculously low for modern email, and the failure rate is surprisingly high for commands
- There is no real spec for IMAP (the article quotes 3501, but 4549 and 6531 are also “standards”), it’s a suggestion at best and every mailserver goes its own way. For example deleting a message does different things in Exchange and GSuite. GSuite has “labels” which are represented as IMAP folders but don’t always act like them.
- It’s unclear what the mailserver will accept. Valid MIME isn’t always going to get through
- Most email that’s important is business email nowadays. These are 80% Exchange. Exchange/365 tenants for the most part block IMAP (found this out the hard way) and Microsoft really wants to push Graph (just like they tried to push EWS). This manifests in weird ways like big angry red warnings on the dashboard if you try to enable IMAP, and automatically disabling it on new tenants, so most IT admins will refuse to turn it on even if you explain it’s just as secure.
- Remaining 20% are mostly GSuite. Google, as usual, is trying its hardest to create an email client monopoly for its users, and so there’s a $75000 annual fee for “security audits” by “vetted firms” regardless of how many real compliance certifications you have. They say this is for privacy and security but it’s just a gatekeeping fee and this is obvious due to the large number of email data abusers out there (looking at Unroll). There are even companies (Nylas) that run their whole business by speeding up the certification process and providing an API that works better than Google’s poorly documented own SDK. If you get through all of this your users are shown a consent screen where everything is unchecked by default, which sounds cool but basically means many of your users are going to login without knowing how to give consent to your email client.
Overall, the space is characterized by Google trying its hardest to kill any startups in the space and prevent innovation that could threaten their consumer monopoly, and Microsoft trying to keep people in their ecosystem by just changing their external API frequently to force competitors to rewrite their codebases constantly.
Not remotely true. Perhaps you're thinking about Gmail webmail users?
> and requires a thread-first view of the inbox.
Seeing how Sent messages, which aren't in the inbox, are typically part of threads - even that would not be enough...
> IMAP represents now a small minority of servers
Not sure what you're counting. If you're counting email users on the servers - Google and Outlook365 both support IMAP. If you're counting kinds of server _software_, then maybe, I have no idea.
> their rate limits for non-internal clients are ridiculously low for modern email
You mean, bandwidth rates, or rates of commands/second etc.?
Also, if by "modern email" you mean "email where people send large files" - that IMHO is a problem, and people just shouldn't do that. Same for other social media BTW. The amount of smartphone space wasted by dumb video clips your friends shared with you by WhatsApp or whatever is stupefying :-(
Anyway, people who try to use email to send and receive large files are kind of abusing the service. IMHO.
> - Most email that’s important is business email nowadays.
I don't think so. Or rather - maybe in the country you live in, and even then I doubt it.
> Valid MIME isn’t always going to get through
... and it doesn't help that so few clients can properly present, let alone generate, MIME messages other than with the most simple of structure. :-(
> Exchange/365 tenants for the most part block IMAP
AFAICT, most people who use MS services can use their publicly-available IMAP servers.
> the space is characterized by Google trying its hardest to kill etc.
(sigh) not surprising unfortunately. Monopolistic corporations gonna monopolize-coroprately I suppose.
> Not remotely true. Perhaps you're thinking about Gmail webmail users?
Speaking for myself - I have threaded / conversation view in outlook on desktop and Android - and Gmail on Android.
>> and requires a thread-first view of the inbox.
> Seeing how Sent messages, which aren't in the inbox, are typically part of threads - even that would not be enough...
Both Gmail and Outlook show the whole thread / conversation regardless if only one mail (eg: latest reply) is "in" the inbox (and rest is "archived" for example).
This is via propiatary communication with the servers (Gmail / o365).
This is the work flow to replicate one way or another.
https://global.oup.com/academic/product/one-9780199688258?cc...
As long as the product is different it’s fine. IMAP library vs some Java thing.
Then again, I don’t use the “hugging face” emoji almost ever. But even if I did use it, I would probably think of it visually or just as the word “hug”. That’s how I think of and use other emojis — either visually or as a simple word.
Either way, I will intentionally continue to think of huggingface.co as being a reference to Alien facehuggers.