Hacker News Reader

Comment by Nextgrid | Hacker News Reader

The problem is that email has been taken over by a handful of providers, all of which have "growth & engagement" as their business model and would rather not implement any open protocol at all.

IMAP remains supported for backwards-compatibility, but I'm sure its shortcomings are seen as a feature to push users towards the official clients so there is no incentive to implement another open protocol to address them.

View on HN

This is a problem, but the other problem is the tools to implement the service. You can use Cyrus, Apache James, or Stalwart, and that's it. These tools do not integrate with anything; in the case of Stalwart (the only JMAP server not in "experimental" status) you can't just install a JMAP server attached to an existing email service; you have to hand it complete control of everything, even unto the on-disk storage of mail. This is a complete non-starter for any email service outside of hobby/personal hosting.

In other words, even if someone wanted to make JMAP available to their users, they'd have to tear the whole thing down and rebuild it in JMAP's image, and the tools available to do so are not fully-featured or integratable enough to do so.

> These tools do not integrate with anything; in the case of Stalwart (the only JMAP server not in "experimental" status) you can't just install a JMAP server attached to an existing email service; you have to hand it complete control of everything, even unto the on-disk storage of mail.

This point you mention is being improved. The next release of Stalwart JMAP (expected in one or two months) will delegate user management to either a SQL database or an LDAP directory. Messages will be stored in either Maildir or MinIO/S3. And all other information will be in either SQLite or FoundationDB. All these features were already implemented except MinIO/S3. Development progress can be tracked at https://github.com/stalwartlabs/mail-server/tree/main/crates...

Cyrus is what fastmail use for their own mail servers. And even the fastmail web ui uses jmap.

I don’t know where it says jmap support is experimental, but I think it’s pretty stable and well supported in practice. The main maintainers of the project depend on it utterly.

They don't use the word "experimental" any more but they call it a "work in progress": https://www.cyrusimap.org/imap/developer/jmap.html bottom of the page.

The file storage approach is unfortunately super common and makes a bunch of really basic things difficult and slow. Need to phase it out more heavily, especially from new implementations.

Telling people who have decades of perfectly serviceable email solutions that they need to 'phase out' what works is a great example of an unconvincing argument. What you say might be true from a developer's perspective but maildir is a battle-tested, accessible, well-understood and interoperable storage format. Bundling everything up behind some bespoke SQL schema might ease performance hurdles but it sucks for everything else.

Does it work though? Nobody runs such systems at scale, for very good reasons, because maildir just sucks in so many ways. Horses have also been battle-tested, they're well-understood, but they don't transport tonnes of cargo and getting kicked in your face because they don't like you isn't pleasant.

So if it takes a custom schema in a (No)SQL database, so be it, it's simply so much more performant, scalable, reliable and usable. It's simply idiotic how much maildir-based software starts "dying" at around 100000 emails in a folder, in 2023!

Case in point: MS making it more and more annoying to use IMAP.

Hell, now you need to create oauth2 app just to login from non-approved client, as they recently forced OAUTH2 auth on IMAP clients. Sure thunderbird works but most lesser known clients require a lot of fuckery to make work (and possibly admin permissions for domain, not sure).

https://github.com/simonrob/email-oauth2-proxy

"Just works" I run it with mbsync at the command line.

You do need a client id and some interaction with your O365 admin.

This looks promising, thanks.

I did the song and dance for claws-mail and one day it just... stopped working ;/

I would be migrating away (to anything) if my email provider pulled that on me.

It's entirely idiotic history.

We had our own e-mail, but we had some problems that were generally caused by management wanting @example1.com and @example2.com to be entirely equivalent, not just an alias, but still land on the same account.

which was fine for the email, as it can just have custom rules, but any related services like calendars bugged, as it saw different domains as different users. We tried to hack around it but it was still buggy in places.

Boss finally got pissed off and decided to migrate to O365, all while IT was screaming nooo. "Because our clients use it too, it will be easier to collaborate"

...and it turned out that what he wanted isn't possible in O365 *fucking anyway* so all e-mails are under @example1.com. And we replaced some weird quirks with many more quirks we can't even hope to fix. We waste more time dealing with this shit than we did on running our own mail servers. Which we didn't like, coz running mail servers sucks, but O365 sucks harder.

My university uses MS for their services, and they stubbornly refuse to let me use IMAP (not even with OAuth). I have to either use GNOME Evolution for Micro$oft exchange protocol or a proprietary app on my phone, which enforces bullshit control on a device I paid for.

> which enforces bullshit control on a device I paid for

That's basically the raison d'etre. You, presumably a student, are getting caught in the crossfire of regulatory compliance and they don't want to maintain two email systems.

Source: Used to be the one implementing those controls. We didn't like them either, I promise.

Effectively, MS has bulldozed companies that use Microsoft365 into using its own proprietary protocols and applications (Outlook). It scares companies into not enabling IMAP (even with OAuth2).

It really really hasn't been. Email is just an enormously massive ecosystem with tremendous amount of variance that anything new will take a long time to adopt and a lot of convincing and proving.

Just look at how far the deployment of something as essential as DMARC is, it's sad. That's not some big vendor's fault, it's all the small legacy cruft nobody wants to clean out as it might break things.

Yep. If you go and register a new gmail account today, there will likely be no option to enable IMAP access for that account, altogether.

That appears to be incorrect. https://support.google.com/mail/answer/7126229

Why would you post a baseless conspiracy theory that is easily disproven by anyone in less than a minute?

FYI, the above is true for "child accounts", as my son painfully discovered the other day, after his school shoved Gmail down his throat. No IMAP in the settings, or anywhere else, for the next generation of users, not until they are 16+ in my country.

What makes you think this is for youth accounts, and is not simply the preference of the workspace administrator for your school?

Why would you need to enable IMAP ? It works out of the box. Every non-android platform uses IMAP to access your account (k9 mail, outlook, thunderbird, the iOS/mac email client, etc).

You do have to go into the gmail settings to enable IMAP, FWIW.

As a former K9 Mail developer I will say that the Google extensions to IMAP for Labels and OAUTH (neither of which are standards) are a pain in the backside that no email platform would have dealt with were it not for the fact they are huge.

(I did a spike development for OAuth and a separate investigation into Labels when I had some free time).

So they are definitely not nice IMAP citizens.

Oh and then there's the issues with Labels that their own documentation apparently gets wrong...

To be fair here, labels are a great feature and the default limitation of a letter existing in only a single folder is a dumb remnant of ancient times.

If it's still not standardized then the pitchforks are aimed in the wrong direction.

Are you sure?

Because, IIRC, IMAP access must be explicitly enabled by going to Settings -> Forwarding and POP/IMAP -> IMAP access: (o) Enable IMAP. And I think it is not enabled by default.

Reply on news.ycombinator.com