JMAP: It’s Like IMAP but Not Really (2019)
unencumberedbyfacts.com
unencumberedbyfacts.com
I don’t see Outlook365 or Gmail (or even Apple for its iCloud email that has a relatively smaller user base) announcing any implementations of JMAP on their services. I don’t see smaller paid email service providers announce support for it. As it stands now, I don’t see a future for JMAP outside of Fastmail.
A popular client like Mozilla Thunderbird still hasn’t started implementing JMAP [1] and one of the reasons is because…the client and server documentation for JMAP haven’t been updated for a few years. [2]
So yeah, JMAP is like IMAP, but not really. We probably need a successor to JMAP that is liked enough to be implemented by email service providers and by email clients.
Certainly once calendars and contacts are done, and hopefully tasks as well, that will make a more complete ecosystem that is even more attractive than just the email component!
My first reaction as I have seen this standard popping up was not "my email will be faster/better", but "I will be able to setup a nice support@mycompany.com" email address and have both email/web integration with it.
If developers start to embrace JMAP for a derivative usage, I think it could drive the adoption.
Question: what do you think is the rough timing on Calendar/Sharing/Contacts/Tasks to be ratified? Within the next year, 2-years, etc?
eg say I wanted to use jmap to implement some of hey's functionality -- for instance the screener that isolates and allows ban of new emails on first contact. Rather than a client, can I have a server that listens to my fastmail inbox and (potentially) moves emails as they're delivered?
Any chance you're thinking about allowing UI space for things like that?
It's sad that superior standards need to be evangelized to even hope to gain traction.
Why would either of these build a bridge over their lock-in moat?
All the other major email providers are selling their own proprietary services or only offering webmail clients.
They have no interest in implementing modern open specifications. They only support things like SMTP and IMAP for legacy reasons.
Also, you can't overlook that a significant number of people use Gmail/Google Apps and Outlook.com/Office 365 which each use a proprietary communication protocol that is unlikely to change in the future.
So, it does make me wonder, who does JMAP benefit?
JMAP benefits FastMail (and others). The CEO of FastMail is also a heavy-lifter in standardizing JMAP, Bron Gondwana. It's probably appropriate to mention he's also a lead developer of the Cyrus IMAP free-software project, so it's hard to accuse him of making a private moat for his company.
Self-hosters, and small businesses, who can run software, and possibly contribute, but not write their own email software from scratch—-benefit.
JMAP is targeting email, and soon calendar, contacts, and maybe more general messaging, to run better suited to current network and user conditions.
https://www.ietf.org/blog/jmap/
Who does TCP benefit, JSON, .well-known?
who does vcf/vCard benefit, or .ics (even if it started as a proprietary format)?
If I made myself the spare time, or had a business model, I would love to implement chat bridges (for personal, self-hosted) small-scale use, using JMAP.
I like, I'm hopeful for the potential of, the promise of, REST-ful state transfer but with multiplexing of requests internal to the protocol.
Think of the protocol as sending a git-patch to modify state, instead of sending a diff per file, "resource".
Analogous to how QUIC (which is still adding standardization features) works around the problem of head of line blocking on TCP connections (and on http2's HTTP sequentially multiplex-able response streams), a core theme supporting in existing JMAP for email & attachments is multiple application requests per web-request, on-the-wire payload.
Said another way, I appreciate the multiplexing in: - per web request, - per application request (JMAP adds multiplexing here) - per resource, accessible to the user, on the server, - you can create, read, update, delete, move, etc.
JMAP core and email standards haven't changed in the past few years because they're done! In the JMAP working group at the IETF we're currently heavily focused on getting the calendaring and contacts bits finished. That unlocks significantly more benefit for a JMAP client because you don't need multiple different authentications.
As for email clients - yeah, it's been slow going, partly because when COVID hit we all shrank back into our shells a bit, and certainly my travel and networking was cut back! It's good to see how it's perceived from outside our little bubble where JMAP is doing great and we're moving more and more of Fastmail's internal APIs to be JMAP-style because it's so easy to multiplex requests together and route them separately through our middleware into different systems, then combine the results back to together in a single response to the calling system or browser.
IMAP development started in the 1980s (RFC 1064), and was run over POTS modems for many decades: I find it hard to believe that it can't handle the 'low-end' of things.
The fact that many clients assume high-end / high-speed links is not a fault of the protocol.
All of those support IMAP currently, an open standard. Exchange has MAPI as well.
Email is the best federated system we might ever have, and it has thousands of clients that can work with nearly any provider.
JMAP has not taken off, but there is a chicken and egg problem. Most clients don't support it so there is no pressure on services to support it. Though I imagine if Gmail supported it then suddenly all the clients would support it
Gmail supports IMAP, with custom extensions to support threading. As far as I understand it they use a custom protocol for their own gmail apps because of IMAP's shortcomings.
Its basically impossible to build a modern email app directly on top of gmail's public APIs. You can do it - but only by mirroring and re-indexing all the email.
Technology and people move quickly so fossilized systems such as email get left behind. There is too much friction to improve a system which has a large number of different implementations. Where any change is hotly debated and people still resist features added to email over a decade ago. Naturally the proprietary systems win when they can roll out a new feature and have it universally work on day one.
Do others believe/experience this?
I ask because this sounds like a very Bay Area comment.
Email is for significant communications.
Outside of work, I'm using daily for things like booking bands, talking to strata, etc.
While email in a lot of large companies might tend to be becoming a one-way distribution system; it is still vibrant and heavily used in smaller enterprises.
And that’s just internally. How do you handle stuff that also requires external contributors?
Across countries everyone uses a different messenger app. For some whatsapp, other LINE, etc. Email is faster to make sure you reach them all.
I don't think many companies operate this way. Any formal things, e.g. regulation, finance, HR, will likely use email much more. And talking to a third party company, which some people will talk to 200 of in a year, will all be email, at least to start with.
Side Note: My wife's work went all in on MS Teams. They held multiple trainings and kept showing how it could handle all the functionality of email. You could schedule appointments and even share file..... Six months later everyone still uses email and MS Teams is barely used at all.... email just works.
From my end it looks like the "no one uses email anymore" is just marketing hype.
The IMAP spec is poor; it's ambiguous and unclear. The only way to build a reliable client is to test interworking against as many IMAP servers as you can find. A better protocol would be a good thing.
The SMTP spec is NOT poor. There's no need to replace SMTP. It works fine.
It's some time since I read an explanation of JMAP (with the article doesn't provide). As far as I'm aware, it provides roughly the same service, but using HTTP and JSON. I'm not a supporter of the "replace everything with JSON" movement; I don't see the point.
I'm not that keen on the "replace everything with HTTP" movement either; if you piggy-back your new protocol on an existing protocol, there's a risk that your new application will start dragging the underlying standard along with it, to the detriment of the original application the underlying protocol was designed for. I.e., it would be regrettable if JMAP resulted in pressure on the HTTP standard to start including features motivated by the requirements of mail retrieval.
No. This doesn't even come close to representing my real-world experience.
What a weird way to call a technologically superior solution.
I think you may have become bubbled.
At the moment it is down because I did a migration from one host to another, there is some sort of login problem on the new box with the old database. RSN I'll have time to diagnose and fix.
Nevertheless, a valid current-day example for non-mobile devices requiring a tolerance to dropped connections: I regularly dock and undock my laptop when heading to meeting rooms, and simply switching from docked Ethernet to WiFi causes a network switch that breaks a lot of apps. Engineering tolerance for this kind of thing is only good.
Reality is people use multiple different and mobile devices, roam across connections, and intermittently go offline. Building software and protocols that are designed around this is beneficial to every use case - even those using whatever you consider a non-"terrible computer" to be.
When I worked in an office, I had the same issue as the OP - docking/undocking my laptop resulted in switching from wired to wireless, and connections (eg: SSH, RDP) would often drop. I ended up unplugging the ethernet at my desk and using wireless most of the time because it was just easier.
What would help is protocols actually support the concept of a "reconnection" -- ie: a TCP session that has the same identifier or token as a previous one that lets it it silently resume instead of starting fresh with new authentication. This isn't even that hard to implement if designed in from the start.
JSON protocols have completely taken over the market because they are massively easier to deal with and trivially parse in to structs which don't require a 200 page document to detail the protocol.
You can write a JSON API that's equally bad, doesn't support paging, doesn't let you select just the fields you need, is missing key information in the extracted fields forcing you to parse the entire RFC822 text as a fallback, etc. Dealing with the parser is just one part of the battle.
Complexity has exploded as of late. It used to be a joke that programs grew until they became able to read mail. These days programs grow until they contain a web engine. It's not enough anymore to just have a chat window, the chat window has to generate thumbnails of webpages for you.
We would be better off having a smaller selection of better software over 100 different email server implementations which can't agree on a basic set of features gmail had a decade ago.
You can't "give some grace period of compatibility" in an open / distributed ecosystem, every version of the protocol lives forever.
YAML's dumb fuzzy booleans still cause issues all the time even though Yaml 1.2 was released 13 years ago.
Edit: From a quick glance it seems like JMAP actually supports submission.
* Corporate email services, aka Outlook/Exchange
* Surveillance email services, aka Gmail, etc
* Real email services that expose the raw email database, such as Maildir, to the users.
For the first 2 kinds, I'd rather POP and keep my interaction with them to the minimum. For the last kind, IMAP/JMAP is too limiting in capability.
To sum up my thoughts about this thread:
Appreciate the great discussion! Wherever you land on the validity of JMAP, as a long (long) time IMAP client developer I stand by my position that JMAP is a huge improvement over legacy IMAP. Not perfect, but walks a reasonable line between what IMAP client devs understand and the modern world of APIs 30 years later. Is it REST compliant or a SOAP derivative? Most likely no and yes. Does it only benefit client developers? Probably, but that is a big benefit to the E-mail ecosystem. IMO the biggest barrier to new and innovative E-mail clients is the obtuseness of the protocol itself, and JMAP goes a long way to bridge that gap.
AFAIK search in IMAP is folder-based, so you have to iterate over the folders, do multiple searches, and merge the results. Not elegant, and probably not fast, but it doesn't seem like a protocol killer to me.
Fastmail are surely in a better position than you or I to figure out which kinds of searches are commonly performed!
Can the HTTP spec just add a CALL verb already?
If not, is anyone doing anything to improve this situation?
Sending packets over the internet across the farthest ends of the earth takes about 300ms roundtrip. What's preventing email from approaching this?
Latency is the biggest UX issue facing email IMHO, and a huge part of why it is continuing to lose ground vs alternative messaging systems, most of which are proprietary, but offer sub-second latency delivery. The world will become a worse place if the trend continues to its logical conclusion and proprietary silos eventually fully supplants email's remaining use cases.
The high average latency and even higher variance leads to people second guessing if an email they sent will ever arrive, and if an email they're expecting has ever been sent, resulting in crude workarounds like resend buttons becoming commonplace and seen as necessary, perpetuating the uncertainty around deliverability, and even further eroding trust in the system.
But most importantly, the latency problem makes email unsuitable for the most compelling use case for messaging platforms: real-time conversation.
Holding a spontaneous real-time conversation over email is painful, as anyone who has tried can probably attest. Both sides are constantly left wondering if the other side is simply taking their sweet time to reply, or if the email is just taking even longer than email usually takes to get delivered. Conversations that could have taken mere seconds end up taking minutes as a result.
This is why email has been consistently losing to messaging apps for fun conversations between friends where email's latency would suck much of the fun out of them, to Slack for mission critical work conversations where wasting time means literally wasting money, to live chat widgets for businesses looking to provide a customer support experience that delights, whereas conversations over email frequently tends to frustrate, the list goes on...
Solving the latency problem for email I believe is the key to slowing and possibly reversing the trend of email's slow but steady decline, and society will benefit massively from the continued prosperity of this rare miracle of an open system that managed to survive for so long despite all its shortcomings.
If that's what you need, you use a real-time messaging application. That's not what email is for. I disapprove of the incorporation of email functionality behind a messaging UI; I get fed up of sending a properly-composed email, and getting a one-liner reply, because it's hard to compose a thought-out, formatted email on a phone.
As for changed messages while synching, do your messages change after delivery? If not, a simple delivered after X would be sufficient.
Almost all requests in JMAP are to the same URL using an HTTP POST to submit a JSON body of “methods”.
More generally it’s how you layer over http when you can’t necessarily guarantee respecting its requirements like purity, idempotence, or cacheability (or don’t want to bother), or when you fear hitting it’s practical implementation limits.
JMAP was stupid when the Fastmail CEO or whoever was here pushing it, and it's still stupid.
This results in making it extraordinarily easy to break most parsers in use, today. For example, to kill Python:
n="$(python3 -c 'import math; import sys; sys.stdout.write(str(math.floor(sys.getrecursionlimit() - 4)))')"
left="$(yes [ | head -n "$n" | tr -d '\n')"
echo "$left" | python3 -c 'import json; print(json.loads(input()))'
So when you're handling something like email, on a mobile device which may drop a partial connection, you need a hell of a lot of extra work to show the partial download, which might even be everything except the closing braces, or you have to discard the whole thing - which isn't kind to your user.
Even so, parsing JSON is a well-understood problem, and there are tons of mature libraries to parse and generate it. I'd much rather work with a JSON-based protocol than one which made up its own wacky data format.
The same goes for the use of HTTP. It may not be the ideal protocol for accessing mail, but everyone knows how to work with HTTP, and tools to interact with it are plentiful.
The same could easily be said for mbox and maildir. These have been around forever, and if you have a library for JSON, you've probably got a library for those two, as well. But parsing them tends to be less error-intensive if you hit some kind of connection interruption.
The complaint for JSON in JMAP tends to come down to - does it improve the current status quo?
Never heard of the ">From" problem? :)
mbox has a well-documented escaping problem -- many tools which parse mbox files will mistake a line which begins with the word "from" for the start of a new message. As a result, many tools which generate mbox data will insert a ">" before the start of those lines, mangling the data slightly to prevent corruption.
But that's besides the point. mbox/maildir are data storage formats, not network protocols. The corresponding network protocols (POP and IMAP) have their respective problems as well, though.
Yes, they all have problems. That's... Our industry. You pick and choose your problems. Hence why it's easy to point out rough edges in JSON, and it's easy with mbox or maildir, too.
Do you have multiple high quality implementations for supporting mbox or maildir in nearly every language and framework in active use today?
If not, then they’re nothing like JSON.
Yes.
They're builtin to Python's stdlib [0]. There's dozens for JS. Part of the PHP stdlib. [1] A number for Rust. Several for Go. And so on, and so forth.
[0] https://docs.python.org/3/library/mailbox.html
[1] https://www.php.net/manual/en/function.imap-createmailbox.ph...
That's an IMAP library, not mbox/maildir, and it's part of an extension, not the standard library.
JSON is a data exchange format, it’s always been intended as a way to communicate between systems. You can use it for data storage just fine, but it’s not the primary role of the format.
Mbox was not designed as an exchange / interchange format, but as a storage format for MUA, with some possibility of interoperability.
> so special parsers are not needed: this is quite intentional.
Two lines from a song rjbs and I made (but never published) about JMAP a few years ago, back when I was a Fastmail employee, set to the tune of the Major-General’s Song.
It's a diseased mind that takes a shitty programming language, JavaScript, and turns it into a shitty structured data format.
But probably not because it adds an other layer of friction and complexity to deployment, which doesn’t do much good to your odds.
And it requires an entire additional ecosystem of inspection and protection capabilities to appear.
Not to mention the parsing and serialisation of a bespoke protocol, and the risk of all the usual framing and escaping errors in the protocol.
It also requires going to Iana for a port.
Except it isn't. As you say, IMAP could do with modernisation; JMAP isn't that, it's a replacement.
If you want to replace a protocol that's in very widespread use, then your replacement needs to offer decisive benefits to operators and users, otherwise they won't switch. JMAP seems to offer benefits only to developers.
In theory that's doable. In practice modernizing a protocol with extensions is though, since support ends up patchy. Especially when the big email providers have few reasons to make a documented, non-propriety extension. Why would they when they can instead push people to their propriety apps?
It's definitely possible that JMAP doesn't catch on. In that case, we'll probably just keep languishing with non-modernised IMAP.
The key is getting client support, especially in mobile (where there is a noticeable user benefit) and in open source software.
I find this argument shortsighted. If it benefits developers then it does benefit users. Making developers' lives easier means they produce code faster (users like things done sooner), have fewer bugs in their code (users don't like buggy code), and have more time to implement other features (users like featureful software). I don't see how that doesn't benefit users.
I've spent a lot of time on it, but the reasons that JMAP was developed still exist and are still valid :)
Replacing it with something more sane seems like a good use of time.
> There are libraries for every single language imaginable.
These statements are contradictory.
[no application protocol implementer] implements HTTP, JSON, and TLS [because there are well-worn implementations that can just be used for the purpose].
I'd challenge you to find a single production suitable language which _doesn't_ have a http library.
Similarly, HTTP is a big standard that takes a long time to implement well. But very few people will have to implement it: they will use existing implementations.