An open source JMAP proxy, JavaScript library and webmail demo | Hacker News Reader

An open source JMAP proxy, JavaScript library and webmail demo

blog.fastmail.com

95 pointscdvonstinkpot18 comments

Can the JMAP protocol be made easier with HTTP/2? The protocol has its own sort of multiplexing (multiple calls in one HTTP call, worrying about executing things in a certain sequence, etc)

Wouldn't an HTTP/2 transport mechanism remove the need to do all this and greatly simplify it to feel more RESTy, but get all the benefits of multiple calls with near zero overhead?

JMAP itself is transport-agnostic, though of course we've only implemented it over regular HTTP so far.

Some methods have to be batched together and executed in order, as it's possible to arrange for the output of one method to be used for an input to another method within the same batch. It is however entirely feasible to have multiple batches containing a single method in-flight at the same time and indeed, the demo client and the FastMail client do just this.

I'd certainly be interested to see if HTTP/2 could help anywhere but I don't think there's many obvious wins - JMAP itself has very little overhead.

I really hope that this or something similar sees some uptake... Although I'm not familiar with JMAP I'm relatively familiar with SMTP, POP3 and IMAP and to be honest, they kind of suck and support things not needed, and things that are seem hacky...

A new interface for mail, contacts, calendar has been sorely needed for a while.

I, for one, am somewhat leery of people reinventing the wheel - especially for a browser-based environment, where everything has a history of being brittle and short-lived.

IMAP works amazingly well if you're using dovecot as a server (including search and push e-mail), and CalDAV/CardDAV also work well if your server actually implements things properly, and projects like Horde have shown you can run and deploy web-based services for them at scale (even though I still regret they're using PHP).

Sure, anything related to DAV has more than a few warts, but my main point here is that things like JMAP won't cater to an installed base or deliver benefits across the board until they finish filing all the corners on their shiny new... cube? Dodecahedron?

> I, for one, am somewhat leery of people reinventing the wheel - especially for a browser-based environment, where everything has a history of being brittle and short-lived.

Yeah, so are we (and we even introduce JMAP with the "now you have X standards" XKCD comic). This is our last resort. We've been pushing the limits of what's possible with IMAP for years, and we've come to the conclusion that there's just some things you cannot do efficiently with it.

JMAP is not not just for browsers, though that has been where our focus is so far. I have partial implementations of JMAP for fetchmail and mutt already done (I'm just trying to find time to finish them and get them out there). Yes, JMAP uses JSON for its data encoding and (currently) HTTP for its transport, but it's not inherently browser-based.

> IMAP works amazingly well if you're using dovecot as a server (including search and push e-mail), and CalDAV/CardDAV also work well if your server actually implements things properly, and projects like Horde have shown you can run and deploy web-based services for them at scale (even though I still regret they're using PHP).

The problem is, if you're not using a server like Dovecot or Cyrus that does implement all the extensions to IMAP that you need to get a modern email experience, then you're screwed. Ditto if your client doesn't support it. So we end up in the situation where we have all this stuff that is technically IMAP-compliant (ie implements RFC 3501) but gives a half-assed experience to the user.

Worse, if you wake up one morning and decide you want to build some amazing new experience on top of standards-compliant email, you'll spend the next few months trying to get your mail client bits working correctly before probably giving up. Even if you do get it working, you'll still have to work out what to do against servers that don't support CONDSTORE and QRESYNC. This is why all these pretty new mail-based apps generally target Gmail exclusively - it's a known environment.

So that's what JMAP is about. Use it and you get a straightforward protocol based on tech you probably already know that supports things you expect in 2015, like push, assistance on flaky networks, battery friendly, etc.

> Sure, anything related to DAV has more than a few warts, but my main point here is that things like JMAP won't cater to an installed base or deliver benefits across the board until they finish filing all the corners.

Do you see anything missing from the spec?

We recognise that this doesn't work without everyone supporting it, which is part of why we're releasing the spec and all this code and talking to every company with an email product that will listen. This won't work unless everyone can play and its easy for them to do so. We don't want to own this; that doesn't help anyone.

I don't know if JMAP will win. Something has to change though, that I'm convinced of. The web had its revolution (a couple of times). We think its email's turn.

When I read the spec the push support was ungood.

What it should do: Tell the server an XMPP or HTTP endpoint to which the server should push a specificed opaque blob when new mail arrives, and possibly other opaque blobs corresponding to other changes, and a (non-binding) rate limit.

With that, it would be easy to write a low-overhead server to support a particular mobile client. Accepts opaque blobs, decodes a little, sends blob to phone via giantmobilevendor.com (which also uses XMPP/HTTP), needs to store no state.

That's pretty much exactly what it does. Client supplies a HTTP endpoint, that gets called with new state strings. That endpoint arranges for that information to be shipped to the client and then the client issues calls to update methods to get its view of the world up to date.

Its easy to extend to a custom (non-URL) method type if you like, and works well with mobile push channels. This is what we do with our mobile apps at FastMail to push through GCM, APNS, ADM and Pushy.

Glad to hear you've fixed that, then.

Got sidetracked reading through the Overture.js documentation and source, which I hadn't seen before. It seems like a really impressive Javascript framework. Live queries ie. data views, lots of great animation support, nice set of stock components... It's a real shame they put all this work into a monolithic framework rather than contributing packages / ideas / insight to something like React or the broader ecosystem. I feel like these kitchen-sink frameworks are more or less on the way out -- this is probably a fantastic body of work that few people (myself included!) will learn well enough to appreciate.

Well, Overture was created several years before React came along (it actually predates Angular and Ember as well), so it would have been hard to make it as plugins to this! It's not actually as monolithic as it first appears: a small change to the build tool could easily let you just build the parts you want (e.g. just the view system, or just the datastore). Each folder inside the source/Overture/ directory is really a separate module. The dependency information between the modules is already in the files so that the build tool orders them correctly, so it would be a small extension to make it only include the modules you want + dependencies.

Basically, the library is perfect for us (and we think it's pretty neat; our web app is one of the largest, most complex and fastest out there, and the codebase is very clean). But we totally understand that the lack of more documentation, examples (or even books, tutorials and seminars!) compared to the popular libraries makes it harder for people to get into. We still wanted to put the code out there in case someone found it useful, even if only as inspiration rather than using it directly.

If you want to see an example of idiomatic Overture usage, take a look at the source of the JMAP demo webmail. It's fewer than 2000 lines of non-library code, and yet packs a lot in!

I remember brong and friends from FM talking about this on blogs and maybe the Email Forum. And it is finally starting to land!

Very cool. Gotta love FM. Thanks from a paying customer!

What's the authentication story, is it done at the HTTP level?

Yes, via the Authorization: header with a negotiated access token.

The spec includes a discovery and authentication mechanism that a client can use to get an access token and can work fine with passwords, OAuth, etc. We don't actually have a lot of experience with this part of the protocol though. We expect that vendors may want to provide an alternate method to obtain access tokens (maybe using OAuth tokens directly). We're expecting to learn more about these use cases as more implementations appear.

I hope it works with 2 factor auth (like TOTP) for applications too. Perhaps it's a bit outside of the protocol scope, but a "best practice" suggestion would be great.

I don't believe there's anything in it that would prevent that. If you squint, the auth exchange looks a lot like SASL, and that supports pretty much anything.

This is something we'll gain more experience with at FastMail over the coming year. We'll be sure to share that experience.

Are there any other implementations of JMAP, especially for calendar/contacts? Also, android sync service for the same?

There's not much around yet. The only things that are in any kind of usable/complete state are the proxy, the demo client and the JS client library, all mentioned in the original post.

Cyrus has partial support (the basic data model and contacts) and is running in production at FastMail. The aim is for full support this year.

On the client side, Roundcube has committed to adding support in the next version (after their recent successful Indiegogo campaign) and Thunderbird is serious investigating it (there's a GSoC student looking at it).

I'm aware of two companies that will be shipping products (client and server) based on JMAP this year. I'm not going to mention names because that's not cool.

Finally, I'm personally building a JMAP data model library in Rust, a standalone (non-IMAP) server on top of it, and I'm part way through adding support to fetchmail and mutt.

I'm not aware of a sync service but it would be pretty straightforward to build. I'll have a look at it sometime, probably taking code from the FastMail app for Android, which has a partial JMAP client in it already.

I need to pull this info together into a "software" page on jmap.io - I'll try to do that soon. In the meantime, if you're working on something that uses JMAP, let me know :)

Done: http://jmap.io/software.html

Reply on news.ycombinator.com