JMAP – a better way to email
blog.fastmail.com
blog.fastmail.com
I just tried the following experiments with my Gmail and FastMail account. There's a reason why FastMail just feels faster!
Try this:
1) While having the Gmail iOS app open, mark an email as read on the web
2) See how long it takes for Gmail to reflect the change
3) Now try with FastMail iOS app, see that it's faster
Try this too:
1) While having the Gmail iOS app open, delete an email on the web
2) See that Gmail app never updates until you manually refresh
3) Now try with FastMail iOS app, see that it happens instantly
How about the other direction?
1) Read an email on your phone
2) See that Gmail on the web never updates until manual refresh
3) Now try with FastMail... Instant update!I'm really considering moving from Google Apps to Fastmail... who else has made this jump? Totally worth it?
Edit: I have a secondary email account through Fastmail where I don't need the tagging ability, and have been very pleased with the service. If you don't need tagging, I think it's worth the switch. They're clearly active in improving the service, while Gmail feels like it's moribund.
I just haven't quite done it yet. From people I've spoken to, everyone praises Fastmail... seems like we all need that extra push.
Slow?
Thanks FastMail. Sincerely, thanks.
Modern, innovative, service-oriented products and a business model based on customers giving the company money in return for services.
I hope that this is a reminder to startups out there that you don't have to trick customers into handing over their personal information all the while pretending to offer your services "for free", in order to gain market share and be profitable.
The "traditional" business model can be quite successful.
So yes, if your goal is to attract massive amounts of venture capital and then get bought out, don't copy us. If you want to have some fun doing something you love without having to worry about anything else then this is definitely the way to do it :)
All the moving parts probably exist, but so far no one has pulled it together into something that a random person off the street can use.
Its not the problem JMAP is trying to solve.
The keyserver protocol includes commands to include keys in the keyserver network. How you send keys depends on the UI of the tool you use, geeks will do this:
gpg --keyserver <keyserver> --send-keys <fingerprint>
https://mail.gnome.org/archives/gnome-announce-list/2014-Nov...
The average user doesn't use GNOME or Linux though :(
There are a ton of interesting problems in the email world, including all the server to server communications. JMAP is lightning focused on solving ONE of those problems - the communication between end-user devices and their mail server - and doing it well.
My point is that using SSL on its own as the only line of defense should not be an excuse not to have anything else.
It's like saying my apartment doesn't have a spec for a security system nor a safe because the lock on the door does enough of a job to secure it.
Also, a widely studied protocol spec means nothing. The bugs are from the humans coding the implementations, it doesn't matter what level it is, they will have some bugs. Nobody can code a perfect secure implementation but we can have some kind of redundancies in the system, where if one security level fails, the rest can still have some reasonable security left.
Relying on SSL alone is not enough. But I don't think JMAP is the right place to do this, we may need something else in addition to JMAP.
At least JMAP is extendable, so that's one good thing it has.
If you were insane, you could do JMAP over XMPP, or JMAP over email. That would be neatly recursive...
I rather have a secure modular system than a single protocol that does everything. We simply just can't leave it as JMAP+SSL because once SSL cannot be trusted, it should be easier to replace it with a better solution or an updated SSL protocol. I don't think it is reasonable to include this within JMAP and force all services to update jMAP overnight because the internal crypto library had a hole.
https://www.youtube.com/watch?v=pDmj_xe7EIQ http://www.thoughtcrime.org/blog/ssl-and-the-future-of-authe...
If we have a solution, everybody would be using it by now. PGP or SIME was going to be it back in 90s but it didn't take off well.
The content of the message itself should be encrypted at the very least but how do you deal with the keys then?
If you were designing a replacement for server-to-server SMTP, it'd be nice to require encryption, possibly end-to-end encryption.... although encryption can also be layered on top, at the RFC 822 level instead of the RFC 821 one (as S/MIME, PGP, etc. do). But this isn't an SMTP replacement, it's a replacement for IMAP (and ActiveSync, POP3, etc.).
Inbox is a separate layer on top of existing email servers. My impression is that the pain point they're addressing is, you want to build some awesome email integration app and sell it to companies, but you're looking at how complicated it would be to speak IMAP and IMAP-with-quirks and ActiveSync and the Gmail API and everything else (in order to build a high-quality integration) and that's scaring you. You can outsource that problem to Inbox, they'll make sure they get good-quality data from any email server reliably, and you just speak an HTTP API. This involves deploying a separate server which caches email. You talk to the "sync engine" over a reasonable protocol, and the sync engine takes care of talking an unreasonable protocol to wherever the data actually lives.
JMAP is an API to directly access an email store. It's something that an email provider (Fastmail, most obviously) will directly support, but it also only helps you for those providers that actually speak JMAP.
If you're an email server developer, an ISP, etc., then JMAP is cool and Inbox is not really useful -- there's not much of a point in having an email server that speaks IMAP and then also sticking a frontend to convert IMAP to something else. You might as well just make your email server speak a reasonable protocol directly.
If you're an app author, Inbox is useful today, and JMAP is only useful, maybe, if all your users are Fastmail users.
If you're an in-house developer at a large company that's running Exchange, and realistically you have no hope of getting the email admins to switch email servers (and you wouldn't want to run the email service yourself, even if they offered), JMAP is useless to you, and Inbox is super compelling.
(There is probably an argument that the Inbox "sync engine" should speak JMAP on the front end / those APIs should converge. Certainly the Inbox sync engine should speak JMAP on the back end, so Inbox can advertise support for Fastmail.)
First of all— I want to say that we’re huge fans of Fastmail and really admire their team and focus. Rob and Neil actually came by for lunch this summer and hacked in our office for the day. We’ve discussed JMAP back and forth with them for a while, and specifically had a long thread when we added calendar endpoints to the Inbox platform. [1]
The fundamental differences between the Inbox APIs and JMAP are the kinds of applications built with them, the open source code, and the email providers supported.
The Inbox APIs are really designed to let developers access email data in non-traditional ways. Our site reads “Say goodbye to IMAP” but we’re not so much replacing IMAP, as creating a new protocol which liberates email data to a higher form. We think about email as “the database of your life” which means unconventional “clients” and access/manipulation patterns.
That’s one of the reasons we have a more REST-inspired design, whereas JMAP is closer to RPC. The Inbox API endpoints strive to represent canonical objects that developers simply treat as resources. JMAP is tailored specifically for traditional email clients, so it excels at fast operations (like archive -> reload), but compromises in other areas of simplicity and clarity IMHO.
The other main difference is code. The Inbox sync engine is open source with about 15k lines of code on GitHub.[2] You can run it yourself right now, and it works on all providers, including Fastmail! That means your users don’t need to be on any specific provider or backend. If you don’t want to run it yourself, you can use our hosted version which also includes support for enterprise environments that use Microsoft Exchange. [3]
I’ve been lurking on the IETF mailing lists for several years now, and one thing I learned is that spec documents hardly move the world forward. Developers need code that is battle tested and they can depend on. What use is a API spec if you can’t build an app with it?
The JMAP system is a great spec for FastMail’s internal mail service, and it works to power their own clients, but unless they make some drastic changes to the code and business, it’s not a platform for building 3rd party mail apps.
As an aside, I totally wish I had more time to write. I envy how prolific the Fastmail blog is these days. :) Keep up the great work Bron, Rob, Neil, and the rest!
[1] https://www.inboxapp.com/docs/api#calendar
Thanks for commenting! I agree that our APIs are orthagonal in some ways.
We definitely plan to ship open-source code for everything JMAP. I'm working on an IMAP<=>JMAP proxy, which I'm hoping to have in time for FOSDEM in February. I'll be giving a lightning talk about JMAP there.
I tell you what - that writing it taking a lot of my time! Evenings, on the train to and from work. It's not a pace we can keep up forever.
You guys are doing awesome work. Hope to catch up with you again - and hey, it will be great if you add a JMAP backend when JMAP is a bit more mature.
Worst case for FastMail, we still have an awesome API for our own backend. Best case, we help everyone else too.
It is :)
REST doesn't preclude doing multiple things in one round trip. Well it does seem JMAP is a bit more flexible when it comes to multiplexing very different actions. But the error handling doesn't seem to tell you which command has failed. You still can't get away from certain ordering issues. And to be honest, I don't see how it's going to be that easy to cache.
This really is RPC – using it as a straight replacement for IMAP might work well, but I wouldn't want to use it as a platform to build something very different than an email client on it.
Caching is pretty easy: you just keep a cache of each object type. The delta update mechanism means you can very accurately invalidate what you need to keep it up to date. The system is actually really flexible and a great fit for any CRUD based app; JMAP is really just a combination of a very powerful and efficient database access protocol, and the definition of some objects to represent email, contacts and calendars.
What I mean is, if tomorrow I add a file with the mediatype application/vnd.icebraining to my server, an intermediate HTTP proxy can still cache it even though it knows nothing about my format.
But if I add a new object (say, Note) to my JMAP server, along with the appropriate methods (getNotes, getNoteUpdates, etc), an intermediate JMAP server will be unable to cache them, since it needs to know the semantics of the new methods.
So frankly, I don't see how their protocol follows an uniform interface at all.
I still think it's pretty neat, though, don't take me wrong. I just want people to stop abusing REST. There's nothing wrong with RPC!
Like Fielding writes in his dissertation, "The REST interface is designed to be efficient for large-grain hypermedia data transfer, optimizing for the common case of the Web, but resulting in an interface that is not optimal for other forms of architectural interaction."
Also, caching JMAP may be easy on the client side, but not for any intermediate cache - unless it understands JMAP.
I'm not saying RPC over HTTP is bad, mind, but you do seem to lose some of the benefits of HTTP.
Approximately even fewer of the people here at HN like the idea of their email messages behing sent over non-secure channels.
At least we're not embedding HTTP status codes in XML...
I have the feeling the next big social network disruption is going to be leveraging e-mail in a big way. E-mail is the established quasi-p2p (it's a network of centralized services) platform that everyone has an account and a very complete social network on. I'm not saying it would be easy to launch a Facebook competitor from e-mail, but I believe it's still got the potential to launch one last social network and someone just has to do it.
I'm hoping the key lies somewhere in building the right sort of mail client, that's got the reliability and extent of e-mail, adding on top some hook feature and layering in perhaps a dash of privacy, cryptography and independence. There's got to be something viral in there.
http://techcrunch.com/2014/07/07/mit-and-dropbox-alums-launc...
Better and more robust email protocols, based on unencumbered open tech? Yes please. What else is there to say? :)
1. You recommend using a digest of the message to generate the message ID. Did you consider mandating that scheme or going farther and using a Merkle tree for the mailbox representation? It seems like this would allow for generating single requests that can fetch all new items.
2. Why SHA-1?
Edit: Also, thanks for keeping Fastmail running. As a customer of about ten years it's much appreciated.
SHA1 was a good balance between speed and unforgability at the time. There's still no known collision, and I'm working on putting a piece of randomness into the LMTP headers that are added to every message on delivery into Cyrus to make it even safer.
Totally a stub right now, but yes - we would like to have it directly in Cyrus...
"Finally, we know IMAP, SMTP and the DAVs aren’t going away any time soon. No protocol will succeed unless it provides an upgrade path from where we are now, and a compelling reason to switch. We will provide a proxy which can talk to existing servers and present them over JMAP."
Each server and user would have their own identity management concerns.
Basically someone can send a "friend request" and if approved by a human (the recipient) they may communicate freely from that point.
There's still friend-request spam on facebook, but not nearly as much as email because you can just ignore the people you don't know and block people you do know if they begin to spam you.
My initial thoughts are that you'd have a blacklist for servers as well as individuals. So if people mistakenly approve senders at around the same rate they mistakenly open spam emails then you'd only get through one time in ten thousand. If your local messaging provider realizes that 9999 out of ten thousand friend requests from a particular domain are denied, then they'd block the entire domain.
Since a friend request conveys little to no information beyond the fact that someone wants to communicate, they'd be unable to get their message out effectively. If one in ten thousand spam friend requests gets through by mistake and after the initial spam message one in ten thousand isn't immediately blocked, then the spammers would need to set up ten thousand domains, each sending out ten thousand messages to get through to one schmuck who might fall for it, and then they'd need 9999 new domains since all but one have been blacklisted. It would likely be enough of a deterrent to prevent most spammers from continuing, although there will always be some amount of spam.
At least I don't think most people would be likely to accept a friend request from IncreaseYourDickSize6969@nigeriamail.com -- but there may be more effective strategies for handling burner identities. How would you do it?
Can anyone explain what they mean by "platform push channels"? I don't understand how you can have realtime updates without leaving a connection open or periodically polling. Do they mean that they use the same mode of transport as Push Notifications?
We use this right now in our mobile apps. The apps respond by calling back to FastMail to check for updates.
More info:
http://blog.fastmail.com/2014/12/02/dec-3-push-it-real-good/
https://developer.android.com/google/gcm/index.html
https://developer.apple.com/library/ios/documentation/Networ...
There is a a protocol named Sieve ( http://www.ietf.org/rfc/rfc5228.txt ) which provides for delivery stage filtering rules. It is similar in capability and usage to procmail, but formalized and somewhat more modern (and not capable of running arbitrary system commands), and supported by some modern IMAP/POP servers, like Dovecot. Presumably if JMAP gets integrated into those servers, Sieve would also be available.
And, I would hope there wouldn't be an IMAP/JMAP translation layer, but instead servers would implement JMAP directly. Though I guess in the short term there might be some sort of proxy.
That makes me extremely queasy. Yes, HTTPS theoretically provides transport layer security, but a single breach of transport layer security results in the attackers permanent access to your mail.
I.e. run a MITM attack in a coffee shop, snoop up JMAP passwords, make use of them at a latter point in time.
I really hope I'm missing something important, or misread the spec.
Yes, we're assuming a secure transport. Most of the internet currently does. Most of the passwords you send over encrypted channels right now are plaintext. This is not something we're trying to solve with JMAP (if it even needs solving, which is debatable).
Authentication is an incredibly hard problem. Including it in the spec encumbers it. And allowing for arbitrary challenge/response auth mechanisms gains you exactly one thing - guaranteed incompatibility as the protocol gains popularity :)
Plain text passwords.
Mandating specific security mechanisms isn't future proof. Authentication is almost a side issue to JMAP itself. Getting a securely authenticated and protected channel is phase 1, sending and receiving the JMAP protocol items is phase 2.
The reality is that almost everyone is sending plaintext passwords over SSL these days, and the reason isn't that they hate you - the reason is that it means they can bcrypt the password on the server side.
An interesting factor about most of the challenge response protocols out there - the server side needs to know either the plaintext password or something pretty reversible about the plaintext password. That's how the server can have a little chat over the wire with your client about the shared secret that they both know.
Since my security protocol design credentials aren't that much greater than the average internet commentator, I don't trust myself to design, or the average client writer to implement, a fancy security protocol that nobody has used before. Tried and true please.
And your final paragraph is the reason I posted my original comment in the first place. Don't design a auth/security protocol when you design a mail protocol. Delegate to OAuth, move on. (Then again... debate rages there, too[1]. Auth is not fun(tm))
[1] http://hueniverse.com/2012/07/26/oauth-2-0-and-the-road-to-h...
It's a good comic. Good point.
We didn't take this decision lightly. There are significant reasons why all the existing standards don't do what's needed, and we made sure we built a technology preview and used it in the real world first - check the video or try out FastMail yourself.
The time is right for a better standard.
All you need is an IMAP server, accessed from all your devices. The one that comes with Android isn't bad. Thunderbird works fine on the desktop.
You don't have to use GMail with Android; when you first power up your Android device, click "Later" when it asks for a Google login. Then delete the Google One-Time Startup app. Google won't bother you again.
By moving to JSON and HTTP it also makes the barrier to entry a lot lower for client developers and opens up the possibility to improve speed yet further with GZIP and websockets.
All that while being able to stay compatible with legacy IMAP servers via a reference proxy provided for free by Fastmail.
Edit: Oops I've just noticed your handle so apologies for assuming you didn't read the article. I guess you've read it and still don't see the benefit. In which case would love to know why you think it's a wasted effort (personally, I think it looks like a welcome refresh).
IMAP servers that don't use a real database are sure to be a mess, but that's not the fault of the protocol.
I'd rather not have some startup in the middle of the mail chain. They'll start thinking they own mail.
"J" is misleading because it's not JSON that distinguishes JMAP from IMAP, but rather the use of HTTP, along with recently added HTTP features such as push. JSON is just the message packing scheme. HTTP is what does all the heavy lifting.
"M" is misleading because this protocol is designed for a lot more than mail. It also handles contacts and calendars, and I wouldn't be surprised if Fastmail made their file storage service accessible through JMAP as well.
What's even worse is that once you come up with a protocol that covers everything that Fastmail does, everyone and their dog will try to add more services to the protocol. Instant messaging? Got it. Collaborative editing a la Google Drive? No problem. And before you know it, the protocol is bloated as hell and you've basically reinvented HTTP.
Please don't try to do everything. Do one thing, do it well, and put a strict limit to the scope of the project for the time being. A few more minutes of battery life on a phone is not worth polluting the world with yet another example of massive scope creep. IMAP+SMTP using a stateless protocol would be cool, but anything more and I'm not sure.
Not actually true; the protocol does not require HTTP. Any bi-directional stream transport will work just as well.
> "M" is misleading because this protocol is designed for a lot more than mail.
Maybe "Messaging" would have been better. I don't know.
Part of the JMAP name is to make it look like IMAP so its easy to see the relationship. J is after I. And so on.
But lets not quibble about names. Perhaps its not a great name. It doesn't really matter because products get different, better names.
> reinvented HTTP > yet another bloated protocol
This is kind of "citation needed" stuff. I'm not saying JMAP is perfect, in concept or in implementation, but we've have a shipping product running on it for a couple of years. Its got runs on the board already, and we're inviting everyone to pick it apart and make it better.
> IMAP+SMTP using a stateless protocol would be cool
Both IMAP and SMTP are inherently stateful, and that's not the worst of their problems. Why not fix it all properly?
By the way, I've spent years on the protocol mailing lists with people who claimed that making it possible to submit email via IMAP was creating a bloated protocol. Support considerations, pah - real world problems don't matter. Meanwhile everyone was moving to ActiveSync so that they could actually get shit done rather than congratulating themselves on the purity of their model.
We definitely do want a model that extends easily (the record update and state-based changes model) to everything that a client wants to do, so that a client can get push updates to the things it cares about in a single way. If this model extends past email, then yay for that. Interestingly, caldav with its sync-token is further along this path than most protocols - it just doesn't have an open push mechanism.
I strongly believe that edge-triggered updates are the right way to do push. Send a small packet with no confidential data through the push channel - it doesn't even need to be encrypted, and it doesn't matter if it comes through multiple times - the client can protect itself by only checking for updates every N seconds if there were any pushes during that time period. The only attack is to block the pushes, but that one can't be avoided by any system - if you can't communicate, you can't communicate.
The actual data updates are pulled through the same secure channel as regular communication. Again, if you can get a secure link, and communicate down it, JMAP works.
In our case we send multiple state strings at FastMail right now (one each for email,contacts,calendar) so the client knows which things to ask for updates on. In theory, just a single token would be enough.
HTTP push features are only used at FastMail in the desktop client. On Android it's using Google's push. On iPhone it's using Apple's push. It's trivial to add new push channels, because they're just an edge trigger.
... But to be honest, I don't really care. A lot of people don't know what IMAP and POP and SMTP actually stand for -- they don't care, and neither do I.
I don't think this assertion makes sense. If it's JSON over HTTP (as you've said above), why would they need to reinvent HTTP to achieve any of the stuff they're trying to achieve? They already have all of HTTP to choose from, no need to reinvent it. And, you've made a lot of claims based on some kind of slippery slope argument (which is a fallacy).
I think it would be educational for everyone talking on this subject to spend some time learning a bit about the existing protocols in the space. SMTP is messy, and shows its age in a lot of regards (character encoding complexity, transfer of large data, authentication of sending servers, etc.). IMAP is somewhat less messy and slightly more modern, but still quite old.
Neither protocol is well-tuned for modern email usage. I don't know that JMAP will solve it, but given how much effort is going into web tools, both on the server and client side, and how many more people have experience developing with HTTP, I don't see any reason to not leverage that for email, too.