Progress on JMAP, better standard for synchronising mail, calendars and contacts | Hacker News Reader

Progress on JMAP, better standard for synchronising mail, calendars and contacts

blog.fastmail.com

181 pointscarey35 comments

This is great to hear. I'm glad the project is progressing.

For those not in the know, JMAP is a protocol for mail that, unlike IMAP, has threads and labels built-in at its core. In other words, Gmail without Gmail.

And it's about time somebody came along and built something sane for this.

There have been a lot of valiant attempts at this. Thunderbird has a few buggy extensions that attempt it. Notmuch is a very well written library and command line MUA with a very fast on-disk storage format (with ncurses, emacs, and vim interfaces, as well as neat FUSE file system toys), but it is not a protocol, and therefore does not scale well beyond the limited set of MUAs for it that mostly need a shell or a flimsy web app. Mailpile is an attempt at making a Web 2.0 style Gmail clone, but it's only an MUA and the on-disk format is flimsy at best.

JMAP defines the protocol first. Then, multiple people can take a stab at the most sane implementation. I like this approach quite a bit.

Why not extend IMAP? It is by definition highly extensible, and unlike JMAP, it's not wedged into complex poorly suited protocols like HTTP.

Clients leave IMAP ports open indefinately; IMAP IDLE provides push e-mail over this mechanism.

Reading about JMAP, this just seems like adding unnecessary web overhead and complexity to an already complex problem space.

FastMail's "why" is better explained in a post from last year, "JMAP -- A better way to email" [0].

[0]: https://blog.fastmail.com/2014/12/23/jmap-a-better-way-to-em...

Well, HTTP/2 with server push should handle the "keep a connection open" and "let the server push things to the client" issues.

Because programmers graduating today literally can't imagine a world full of non-HTTP protocols.

> It is by definition highly extensible, and unlike JMAP, it's not wedged into complex poorly suited protocols like HTTP.

Why building on mature concepts that have proven themselves in practice over several decades? It's much more in line with the current web culture if smart rockstar developers half-assedly cobble together a RESTful shit fest.

It's just a smart move to have the API your webmail uses be a "standard"...

I don't understand why i should implement Fastmails web API, when there are several major protocols that are supported by a huge ecosystem already: IMAP, CalDAV, CardDAV.

While I think extending IMAP is the better solution, it is already too complex, as many clients and servers treat things just slightly different enough to be really annoying.

So we do that too - indeed I've done most of the work getting the Cyrus IMAPd server support of IMAP to the level of compliance that it has today, and am still involved in the standardisation battles. But I see from the responses to this comment that it's easier to make snarky comments on the internet than to understand where IMAP just doesn't do what we needed from a protocol, and why we designed JMAP.

The thing to replace is not so much IMAP (which has its own awfulness of implementation) but RFC822-format mail. It looks easy to parse but isn't, giving rise to bugs. Especially if you step outside the ASCII character set. How many of you have an email with a spurious `> From` in the middle because that has been escaped wrongly by something?

Besides, JMAP could be consumed directly by web apps.

This might be a naive question, but I don't know any better: Why can't your mailbox be represented on a server as a directory, perhaps like Maildir format, and then synchronised to your device using a directory sync tool, such as rsync?

Changes, or online access, could be done using any standard remote file access method, such as sftp.

Are there downsides to this approach?

This will be way too slow, and is not at all feasible on a device like a mobile phone where you want only the recent email cached locally, but want to be able to stream on demand the rest with minimal latency (so it appears to the user as though it's all loaded locally under ideal network conditions). The server needs to be able to sort the messages, calculate threading, unread counts etc.

You still need an index of your mail for searches and extra metadata for tags, threading, unread status etc. So you need a protocol for requesting this search and a format what how the client can use the result of that search to access the matching emails.

You need to be able to push email status changes out to connected devices, I don't want an alert for an email on phone if I've already read it on my laptop.

You need decisions around how attachments are stored. Most of the time you'll want to store the attachments separately from the text content of the email so clients don't have to download the attachments to access the text, so you'll need to decide on how this location would be communicated to a client.

Plain files are rather bad for multi-user access, if you have multiple users/devices all accessing the same email they're going to need some protocol for synchronization so you don't get data corruption or lost changes.

It would be interesting to think about email and sync using git on the backend.

Perhaps something like having each blob refer to an email message. Each tree would refer to a folder. Each commit would be made whenever an email is added, deleted, or moved to another folder (though it would not have a reference to a parent). For indexing, you could refer to each blob with a tag that had a name that corresponds to a UID. If an email is deleted, then the corresponding tag would be deleted as well.

Git's garbage collection would eventually remove references to orphaned objects and the transfer protocols would only push (or fetch) objects the other repository doesn't have.

I don't know how it would scale in terms of performance though.

> You need to be able to push email status changes out to connected devices, I don't want an alert for an email on phone if I've already read it on my laptop.

This. This sucks so much. Even gmail can't automatically dismiss a notification when I read a mail on the desktop!

What mobile client are you using? I've found that Gmail on Android is pretty good about dismissing notifications for messages that have been marked as read elsewhere.

Yes. Combine your approach with a new device, a mailbox size of several gigabytes and EDGE/2G network connection.

IMAP/JMAP will work fine in those conditions, as a client can simply fetch the list of folders and a few email headers, and then lazily fetch email bodies on demand. Updating the read status is pretty cheap and does not require a full sync between the device and server, and so on.

rsync is apparently extremely inefficient in a number of common cases, such as simply renaming a folder (completing synchronizing all the files in the folder again).

Also, some IMAP servers store message flags in the filename (when using Maildir). This means that a message that goes from unread to read, it would appear to rsync that one file was deleted and another created (as the filename on disk has changed); the "newly created" file (which could be many MBs in size, depending on its contents/attachments) would then have to be completely synchronized again.

I didn't realize just how inefficient it was until a few days ago when I read an article [0] comparing rsync against ZFS replication (even if you're not familiar with ZFS, you'll understand).

[0]: http://arstechnica.com/information-technology/2015/12/rsync-...

> rsync is apparently extremely inefficient in a number of common cases, such as simply renaming a folder (completing synchronizing all the files in the folder again).

If you run it the normal way this is (sadly) true.

However you can use the `--fuzzy` flag to make the behaviour a little more clever in handling this specific case.

How would you make this work on iOS/android?

That's what I'm doing. My maildir is just synced from the server with Syncthing.

I'm pretty excited about JMAP. We've been planning an overhaul of Usermin, our web-based mail client, and now that we have a UI developer, we'll be embarking on more ambitious features, like threading, off-line abilities, etc. Having threading handled automatically is already reason enough to use this, as it's a much harder problem than it seems at first glance to solve in a consistent way across many clients and servers.

The fact that it also handles calendars and contacts is icing on the cake.

Reading about JMAP last year during their holiday series of blog posts persuaded me to jump on-board for their mail service and to support the evolution of JMAP. No regrets!

JMAP homepage: http://jmap.io/

What clients/apps currently speak JMAP?

It's great to see a company building technology and open protocols without walled gardens and lockin.

Any news on whether the big mail providers have an interest (Microsoft, Gmail, Yahoo, etc.)?

There seems to be a lot of interest in sync related protocols in this thread and dismissal of JMAP as a result. This is bad, because the current state if synchronization protocols still kinda sucks - full disclosure, I work on http://GitHub.com/amark/gun which is a JavaScript based syncing database. Getting synch right in p2p or federated systems (email) entails more detail than centralized ones. So any work on synchronization should be seen as favorable compared to how bad other protocols are for sync.

What somehow bothers me is that the new protocol is all-in-one. Instead of building on / integrating with CalDAV / CardDAV and being modular, they are rolling everything in the same protocol. This is possibly convenient for a protocol controlled by one commercial entity, but why not just use Outlook's then?

Piggybacking it over HTTP also looks a bit strange, but probably helps corporate firewall penetration.

My favorite part about this is that one can implement and start using it at their discretion (i.e. it doesn't require the other server to also speak JMAP). That way I can have a mobile app that speaks JMAP and an IMAP-to-JMAP bridge that syncs my Gmail to it.

Does anyone know of such a bridge? I would love to run one for myself.

https://proxy.jmap.io/ - it contains links to the source code if you'd like to run it yourself. It's just a few thousand lines of perl, and I wrote it in two groups of about 2 weeks each time (about 6 months apart) - and each time very much part time amongst other things. It's actually not that hard if you already have IMAP, CalDAV and CardDAV client libraries.

If you want to make this a standard I recommend making a RFC for JAMP. (Then again they have docs, this would just make it seem more official.)

Interesting. I use Radicalie for Cal/CardDav. Maybe if I get some bandwidth, I could look at potentially contribution a JMAP extension.

Sadly, I don't see a lot of uptake on a new protocol unless Microsoft supports it in Outlook or someone develops a plug-in. Also, the mobile situation is going to be a tough haul.

Reply on news.ycombinator.com