The JSON Meta Application Protocol (JMAP)
tools.ietf.org
tools.ietf.org
The JMAP mail spec, which is what most people will be more interested in, is awaiting final sign-off before publication as RFC 8621.
We’re using JMAP in production at Fastmail on our Fastmail and Topicbox products, right through the stack. Both of their web interfaces speak JMAP to the server. Unfortunately at this time it’s not ready for public use: definition of authentication technique is one thing that is deliberately left out of JMAP, and we’re currently using our own system which has no public documentation (and we don’t expose /.well-known/jmap yet either). The intrepid user may be able to figure it out, but we won’t be providing any support for it yet. We use the JMAP mail spec, the draft calendar and contacts specs, and a handful of our own specs for our own data types, also not publicly documented at this time.
JMAP is much more complex than the standard sort of simple REST API, but there is cause for all aspects of its complexity; the most notable complexity in it is its record state management, part of it being an object synchronisation protocol; an advantage of that is that robust offline support is feasible, even fairly straightforward. I’m looking forward to working on offline support in the Fastmail web app.
I personally think the increased complexity is well worth-while, and I have several products I’ve been sketching out and working on personally (completely unrelated to the world of email or to my employer Fastmail), and all the ones with any kind of client-server API I intend to build atop JMAP, because they’ll benefit heavily from it with respect to features like real-time interface updates and offline support (for web, desktop and command line programs).
(Also, a little bit off-topic, but as a Pobox customer, can I use JMAP / am I already using it from the webmail interface?)
I personally have no basis upon which to make any recommendations for JMAP clients, servers or libraries. The JavaScript one we use in our webmail is not particularly suitable for others to use, despite being open source, as it’s quite tied to our Overture web framework which pretty much only we use.
There is, however, a list at https://jmap.io/software.html of software that supports JMAP.
On your Pobox question: when reading your email at fastmail.com, you’re using the Fastmail webmail, which speaks JMAP.
Section 1.5 of the RFC nails the colours to I-JSON with a "MUST". Do you envisage any amendment to allow equivalent encodings? I know this is indirectly discussed elsewhere, I'm just going through an Avro infatuation.
There have been attempts to solve this with even-odd UIDs etc. but those broke down on further analysis. There have also been papers claiming to implement multi-master IMAP servers over CRDTs but these never addressed the strict UID consistency requirement and never tested with actual client implementations for sync correctness.
How does JMAP address the distributed multi-master server sync problem, assuming eventual consistency?
+1, and it's been a while since I've dealt with this, so my memory's fuzzy, but I think it's even worse than that in a couple ways:
* The failure mode here is not just that the client misses UIDs but that a given mailbox/uidvalidity/uid means something different to the client than to this replica. Then when the client says to delete one message, they delete a different one. This would be a lot less likely if the server could use a uuid for a message (so collisions are unlikely) instead of following a sequence and/or if it the identifier didn't need to change whenever it moves to another folder. (My memory is even fuzzier here, but iirc in gmail's case, its emulation of the folder model also means a message can have multiple uids at a time, one per label. That mismatch between the folder setup clients expect and the label model is another problem with IMAP. Strictly speaking, I think you can do it all with one folder and an IMAP flag for each label; it's just that the clients don't expect it to work this way.)
* Even with a synchronous replication design, there's the possibility that due to a bug or bad machine the server replicas won't match entirely, or just the client state doesn't match. And if you discover and fix such a problem on the server side, your only option for addressing the bad client state is to bump uidvalidity, which is super annoying to the client. It'd be nicer if you could do something that kicks off some softer sync that doesn't use as much bandwidth, lets the client be more usable while it's happening, and/or lets the server know what mismatches were actually found for diagnostic purposes.
I haven't looked into JMAP before, but I hope it materially improves this situation...
https://datatracker.ietf.org/doc/rfc8474/
They were built with reference to how Google do theirs so it will be relatively easy for them to support the standard names, similar to how special-use was built to give similar behaviour to XLIST.
That may be a problem for folks running things at the proverbial web scale, but if it's just a Dovecot server handling a couple of dozen or hundred accounts, is it that big of an issue?
No, unless you're designing a protocol to replace IMAP and you want all the proverbial web scale systems to adopt your new protocol. And then again, interoperability and universal adoption is probably something the smaller Dovecot installations would want.
Master-master conflict-free eventual consistency systems are easier and safer to operate, there's no nightmare damage from split-brain and things just work, even in the worst scenario. Master-slave failover is notoriously prone to error, even for small deployments.
The conversation seems to focus on JMAP as a replacement for IMAP and primarily as an email protocol. The RFC identifies itself as - > a protocol for clients to efficiently query, > fetch, and modify JSON-based data objects, with support for push > notification of changes and fast resynchronisation and for out-of- > band binary data upload/download.
It reads like it applies to APIs in general, or is this only for mail/messaging services?
edit: I see now, this is the core spec, not specifically the mail spec as referenced in another comment. Thanks.
I've wondered for a while how hard it would be to wrap dbmails[db] (postgres) schema/db in graphql and call it a day, for a jmap work-a-like, for example.
[db] http://dbmail.org/
It does describe APIs in general, and that is one of the reasons I like it so much. All of our internal APIs are JMAP: when you need to do something with a user, you call User/set; with a rule, call Rule/set, and so on.
JMAP makes it super easy to build new APIs because you can reuse most of the code; the only things you need to define are the data types (and any custom implementation for access controls, etc.).
Although Fastmail itself has much more legacy code (we’ve been around a while!), new things can start with a framework called Ix, which is available at https://github.com/fastmail/Ix. Ix is the basis for Topicbox (www.topicbox.com), which is built ground-up on top of JMAP.
The JSON Meta Application Protocol (JMAP) is used for synchronising
data, such as mail, calendars, or contacts, between a client and a
server. It is optimised for mobile and web environments and aims to
provide a consistent interface to different data types.
This specification is for the generic mechanism of data
synchronisation. Further specifications define the data models for
different data types that may be synchronised via JMAP.
JMAP is designed to make efficient use of limited network resources.
Multiple API calls may be batched in a single request to the server,
reducing round trips and improving battery life on mobile devices.
Push connections remove the need for polling, and an efficient delta
update mechanism ensures a minimum amount of data is transferred.
JMAP is designed to be horizontally scalable to a very large number
of users. This is facilitated by separate endpoints for users after
login, the separation of binary and structured data, and a data model
for sharing that does not allow data dependencies between accounts.I'd prefer s-expressions.
Personally, I prefer binary formats where feasible, because I like pursuing efficiency to extremes, and write all my own code in Rust.
Professionally, I happen to write mostly JavaScript, and no Rust at all, but we do still focus on performance at Fastmail. (It’s in the name!)
If you’re comparing JSON to s-expressions, it’s a wash in efficiency. They’re both “inefficient”, both representing generic data structures in regular text. They use slightly different punctuation, but that’s about it. The differences are negligible.
But when it comes to support, JSON is well-supported in all mainstream languages, while s-expressions have no central specification on precise syntax, and you’ll need to do your own thing. One of the deliberate design advantages of JMAP over IMAP is doing away with custom parsers, and using the standard tools people are familiar with, HTTPS and JSON.
When talking efficiency, you might then ask why not to use Cap’n Proto, protobufs or one of those sorts of things. Due to the need for extensibility (e.g. in Fastmail we add a few custom fields to JMAP core and mail data structures), rigid structures are unsuitable; you do need arbitrary key-value arrangement, despite it being larger over the wire.
All in all, JSON is a robust choice for just about any API.
Fun fact: at Fastmail we compress both request and response, even using a custom dictionary primed on real JMAP stuff, to improve the compression ratio. I understand standard DEFLATE was about equivalent in size to CBOR (a binary representation of JSON), and the 2KB dictionary brought it down even further, can’t remember how much it was off the top of my head. Neil Jenkins gave a talk about this and related matters last month, https://www.webdirections.org/code/speakers/neil-jenkins.php, no public video sadly. I’ve thought about writing a blog post about it all. An unfortunate side-effect of this compression is that you can’t inspect traffic in the network panel of the dev tools; you can add ?compress=0 to disable it, if you’re an interested Fastmail user.
What do you mean? Both Protobuf and Cap'n Proto are designed to be extensible. You can add new fields without breaking compatibility.
(Disclosure: I wrote Protobuf v2 and Cap'n Proto.)
Forgive me; I am not speaking from a strong position of grokking Cap’n Proto or Protobuf (I’m casually familiar with them but have really only once used them, maybe three years ago), and I believe I erred in what I said.
For Cap’n Proto, I don’t believe I’m far off the mark: to pick a simple example, extensions may add fields anywhere, and I believe you’re not allowed to skip field numbers, so you can’t just establish a registry of used field numbers and pick and choose the extensions.
proto3 looks like it probably can’t handle this either.
On proto2, though, it looks like I was approximately wrong, and you could have finite extensibility, and maintain a registry of assigned IDs so that extensions can be compatible. I’m not sure how suitable that would be in practice—the examples I see only allot a hundred numbers for extensions, which seems rather tiny; I’d picture allowing at least a million.
Anything like Map<str, str or any or whatever> I’m going to disqualify from consideration because either you use structs in some places and thereby limit the places that extension can apply, or you use maps right through and then get no benefit over JSON.
So I think they’d still be mostly technically unsuitable, but maybe Protobuf v2 would work. And maybe I’m completely wrong still—you’re the expert! But really, unless you get a major advantage out of such a thing, an important consideration is social suitability. Protobufs and the likes are not widely-used, require more effort to work with, and require more involved library support than JSON, which is much more universally understood and handled. Just look at how I’ve been fumbling around with them!
That is indeed tricky. With protobuf you could claim some high-numbered fields and hope that no one else does the same thing -- with some randomization you can probably be pretty confident. With Cap'n Proto you can't really do it at all since fields are identified by offset, and making structs arbitrarily large is probably not what you want. With JSON, because fields are identified by name, it's possible to choose a name that almost certainly will not collide with anyone else.
I'd argue that the right answer here would be to ask the spec owners to add extension points, which hopefully would not be controversial. Conceptually this could take the form of something like map<string, any> (there are cleaner idiomatic ways to do it in each of Protobuf and Cap'n Proto but the details are beside the point of this conversation).
That lets you store the the common things in highly compressed protobuf, and extensions can either be done via extension protos, or with stringly-typed map stuff for testing/weird stuff.
I don't think that's a deal-breaker. protobuf has an extension mechanism. [1] You could have a registry for an org to claim an extension number to a given message (and then use it to hold their own message with as much stuff in it as they like). I'm not as familiar with cap'n proto but assume it has something similar.
That said, I think JSON was a fine choice for this purpose. The efficiency is good enough with standard implementations; and there are much faster-than-standard implementations available if you have a special reason to want one. [2]
[1] https://developers.google.com/protocol-buffers/docs/proto#ex...
MessagePack has an ugly history with text data that's still reflected in some of the libraries interfaces. E.g., the Python library doesn't round-trip strings by default. (They'll turn into bytes objects.) There's a flag to do the right thing, but not everyone is aware of it.
Add to that that CBOR has an RFC, and I'll pick it over MessagePack every time.
Binary formats are mostly nicer from a parsing perspective: it's much easier to unpack. You're right, I think, that simple compression will mostly close any size gap.
There is: https://people.csail.mit.edu/rivest/Sexp.txt
It's imperfect, but it does get the job done. Among its advantages is that it's both pure text and binary: there's a canonical representation which is binary, and an advanced representation which is textual, and they map 1:1 to one another.
Examples:
(session
(capabilities
(urn:ietf:params:jmap:core
(max-size-upload 50000000)
(max-concurrent-upload 8)
(max-size-request 10000000)
(max-calls-in-request 32)
(max-objects-in-get 256)
(max-objects-in-set 128)
(collation-algorithms ("i;ascii-numeric"
"i;ascii-casemap"
"i;unicode-casemap")))
(urn:ietf:params:jmap:mail)
(urn:ietf:params:jmap:contacts)
(https://example.com/apis/foobar
(max-foos-finagled 42)))
(accounts
(A13824
(name "john@example.com")
(is-personal true)
(is-read-only false)
(account-capabilities
(urn:ietf:params:jmap:mail
(max-mailboxes-per-email nil)
(max-mailbox-depth 10))
(urn:ietf:params:jmap:contacts
…)))
(A97813
(name "jane@example.com")
(is-personal false)
(is-read-only true)
(account-capabilities
(urn:ietf:params:jmap:mail
(max-mailboxes-per-email 1)
(max-mailbox-depth 10)))))
(primary-accounts
(urn:ietf:params:jmap:mail A13824)
(urn:ietf:params:jmap:contacts A13824))
(username "john@example.com")
(api-url "https://jmap.example.com/download/{accountId}/{blobId}/{name}?accept={type}")
(upload-url "https://jmap.example.com/upload/{accountId}/")
(event-source-url "https://jmap.example.com/eventsource/?types={types}&closeafter={closeafter}&ping={ping}")
(state "75128aab4b1b"))
This is the equivalent of the JSON:
{
"capabilities": {
"urn:ietf:params:jmap:core": {
"maxSizeUpload": 50000000,
"maxConcurrentUpload": 8,
"maxSizeRequest": 10000000,
"maxConcurrentRequest": 8,
"maxCallsInRequest": 32,
"maxObjectsInGet": 256,
"maxObjectsInSet": 128,
"collationAlgorithms": [
"i;ascii-numeric",
"i;ascii-casemap",
"i;unicode-casemap"
]
},
"urn:ietf:params:jmap:mail": {}
"urn:ietf:params:jmap:contacts": {},
"https://example.com/apis/foobar": {
"maxFoosFinangled": 42
}
},
"accounts": {
"A13824": {
"name": "john@example.com",
"isPersonal": true,
"isReadOnly": false,
"accountCapabilities": {
"urn:ietf:params:jmap:mail": {
"maxMailboxesPerEmail": null,
"maxMailboxDepth": 10,
...
},
"urn:ietf:params:jmap:contacts": {
...
}
}
},
"A97813": {
"name": "jane@example.com",
"isPersonal": false,
"isReadOnly": true,
"accountCapabilities": {
"urn:ietf:params:jmap:mail": {
"maxMailboxesPerEmail": 1,
"maxMailboxDepth": 10,
...
}
}
}
},
"primaryAccounts": {
"urn:ietf:params:jmap:mail": "A13824",
"urn:ietf:params:jmap:contacts": "A13824"
},
"username": "john@example.com",
"apiUrl": "https://jmap.example.com/api/",
"downloadUrl": "https://jmap.example.com
/download/{accountId}/{blobId}/{name}?accept={type}",
"uploadUrl": "https://jmap.example.com/upload/{accountId}/",
"eventSourceUrl": "https://jmap.example.com
/eventsource/?types={types}&closeafter={closeafter}&ping={ping}",
"state": "75128aab4b1b"
}
Why not? I'm honestly curious, because I cannot comprehend preferring that mess of JSON to the s-expressions. The JSON just looks like a dog's breakfast, while the s-expressions look clean & easy to write, read, parse and use.
Also, I don't think they are functions: they're just lists with a convention. Those lists can be used to expression function calls, as in Lisp, but in a data format I think that the lists more often express structure.
One thing I love about s-expressions is that one indicates the type as the first item in the list, e.g.:
(foo …)
{
"@type": "foo"
}
Yes, clearly it's the JSON. How do you tell at a glance in the s-expr formatted data what's an object and what's an array? What's a keyword and what's a literal? I can barely even tell the heirarchy where it's very visually obvious with the JSON.
The JSON and s-expr have a different indent level, which is why the hierarchy is less obvious to you.
Debatable. To me there is a clear difference between a list of things of the same type, and a mapping of names to values of potentially varying types. I don't think the specific programming language really matters, they are two totally different concepts with much different applications.
> There's also no way in JSON to distinguish between an int and a float and a decimal type
True, but they are just small literals which are easy to differentiate at a glance, not big multi-line structures like objects or lists. Plus s-expressions have the same issue.
There are no objects — it's all lists (or arrays, if you prefer). This is a good thing, because ordering is always the same.
> What's a keyword and what's a literal?
Everything's a byte string: any interpretation above that is a matter for the protocol. This is a good thing, because it means you neatly sidestep issues like JSON's numbers actually being floats rather than integers, as well as issues like numeric precision.
'But I want types!' Of course, they are good. But the type of a number isn't just integer or float: it's integer-which-is-a-known-version or float-which-is-between-zero-and-one-exclusive. You need to check those types at ingestion anyway; it's not a big leap from integer-which-is-a-known-version to string-which-parses-as-an-integer-which-is-a-known-version.
> I can barely even tell the heirarchy where it's very visually obvious with the JSON.
They are both indicated with indentation. Probably I should have used two spaces for the JSON indentation, but I was lazy.
Why are the quotes needed? It would be a simply change in the spec if names without spaces in them could be expressed without quotes.
Having to quote the symbols is terrible though.
I guess that's part of the point from my point of view: it's not about hash tables or objects or even lists, but about what they represent. In my view focusing on the implementation (a hash table mapping strings to hash tables mapping strings to numbers or strings) is looking at the trees and missing the forest.
> it even comes with a lot of overhead (colons, quotes, brackets and the like).
according to Sqlite devs the that overhead is not much worse that for a binary encoding.
quoting: https://www.sqlite.org/json1.html
> The json1 extension does not (currently) support a binary encoding of JSON. Experiments have been unable to find a binary encoding that is significantly smaller or faster than a plain text encoding. (The present implementation parses JSON text at over 1 GB/s.)
Edit: I see, JMAP moves attachments out of band
Base64 mixes the byte, but Hexadecimal does not.
Try to write a parser for an arbitrarily complex and nested JSON object without running into runtime errors even after the fourth-or-so revision. There are too many "yes buts" in the living standard.
> that overhead is not much worse
It _is_ much worse than the two-byte "overhead" of one s-exp node.
I don't believe you. Atoms are roughly the same, and compounds, well:
[1,2,3] vs
(1 2 3)
{"a":1,"b":2} vs
(("a"1)("b"2))
(You might suggest plists, ("a"1"b"2), for dictionaries, in which case you save a whopping two bytes per entry. But I'd choose the nested-list format over plists, if I were encoding a dictionary into an S-expression.)
Which pretty much disappears when compressed anyway.
Anyway, for being a Fasmail user, JMAP rips IMAP to shreds in terms of performance. I've been following JMAP from afar and waiting for Fastmail to publish this for a long time.
People have ben actively doing this for XML with streaming parsers that emit events, where you don't just validate XML itself but the semantics of the payload you're receiving, and bail out early instead of parsing the whole XML into a DOM then validating the DOM structure and values.
Do you mean parsing the JSON bytes itself into maps and lists and atoms, or turning that into an application-useful object?
If the former, are you claiming that Python's standard json library, libjansson, JSON::XS, etc. have bugs? (I don't know why you would roll your own JSON parser in production code.)
If the latter, how are S-expressions any different?
(And there are non-buggy libraries that do both, e.g., serde.)
Could you give an example? The only deviation from the standard I've run into is JSONL being called JSON, which isn't actually meaningfully harder to parse.
I feel like it should be possible to translate from valid JSON to valid sexpr with a regexp parser (just match parenthesis and stuff)
> There are too many "yes buts" in the living standard.
I am not an expert and do not want to sound dismissive, but I feel like most of those "yes buts" are a solution to otherwise incompatible implementations. I fail to see where S-expressions have an advantage here. Is there a widely recognized standard that most implementations respect?
1.5. JSON as the Data Encoding Format
JSON is a text-based data interchange format as specified in
[RFC8259]. The Internet JSON (I-JSON) format defined in [RFC7493] is
a strict subset of this, adding restrictions to avoid potentially
confusing scenarios (for example, it mandates that an object MUST NOT
have two members with the same name).
All data sent from the client to the server or from the server to the
client (except binary file upload/download) MUST be valid I-JSON
according to the RFC and is therefore case sensitive and encoded in
UTF-8 [RFC3629].
A quick scan of it says to me it doesn't get all the issues but it does get the biggest one, which IMHO are the ambiguities in what constitutes a "number" in JSON. I-JSON explicitly limits it to 64-bit floats.
Not sure there is a decent standard spec and a good Java (say) parser around. Is there?
Here's a Java parser: https://github.com/bpsm/edn-java
There are no RFCs for s-expressions, but that's fixable, I guess. And yes - there are a few Java parsers for them.
And, perhaps, all those people know better than you.
Whether it's the “wrong thing” is another issue, though I've yet to see a standard of “wrong” proposed in the thread for which that is accurate.
Wide adoption and popularity trumps "being wrong" everyday.
And not just for web stuff: the way you describe a custom architecture to rustc is via JSON.
[0]: http://tutorials.jenkov.com/ion/ion-performance-benchmarks.h...
s-expressions seem like an out-of-left-field choice.
That's subjective and it seems plenty of people have the opposite opinion.
> unsafe to parse
Is this based on theoretical concerns or practical experience?
> comes with a lot of overhead
Keep in mind the alternatives it is largely replacing. XML, text-based email protocols each with their own unique grammars, etc.
> I'd prefer s-expressions.
Ah yes, the plaintive cry of the lisper in the desert. ;-)
I'm afraid in the 60+ years that s-exps have had to catch on in the mainstream, they haven't.
I can't explain this result but nonetheless it seems irrational to expect it to suddenly change.
I actually have a (I think) rather novel approach and there's some potential for little optimizations here and there, but it disgusts me that I'm doing this at all. The optimizations popular, shitty formats like JSON receives are purely wasteful.
It's suitable, though. The WWW wasn't made right from the start and there's always an idiot somewhere looking to add more shit to the pile. Not only does JavaScript exist, but it morphed into this ugly, inefficient, very poor ''format''. We don't have Lisp machines, but new processors are getting instructions to cope with JavaScript idiocy.
A numerical (binary) format is clearly best. It's small, random-access when properly designed, and efficient, but hey, everything's being pushed into port 80 so why not push all data into JSON?
Now, the rest of this message is in response to @chrismorgan from an organization pushing this filth.
>Personally, I prefer binary formats where feasible, because I like pursuing efficiency to extremes, and write all my own code in Rust.
We agree on that, then, but I prefer to write that Rust is just the effeminate man's Ada.
>If you’re comparing JSON to s-expressions, it’s a wash in efficiency. They’re both “inefficient”, both representing generic data structures in regular text. They use slightly different punctuation, but that’s about it. The differences are negligible.
This is wrong. Firstly, we should clarify what is meant by S-expressions. Let's assume the S-expressions have lists, strings, symbols, and numbers. That's already simpler than JSON. You're not hampered by JavaScript's floating point nonsense, either. There's fewer edge cases and less escaping. JSON is, at best, a worse-is-better form of S-expressions.
>But when it comes to support, JSON is well-supported in all mainstream languages, while s-expressions have no central specification on precise syntax, and you’ll need to do your own thing. One of the deliberate design advantages of JMAP over IMAP is doing away with custom parsers, and using the standard tools people are familiar with, HTTPS and JSON.
So, this ties into what I wrote before I even read your post, exactly. Cretins continue to push everything into the WWW protocols and act like their garbage is standard, simple, acceptable, or wanted; it's only technically standard, but it's a bad standard. I'd bet my foot you also like ''DNS over HTTPS''. A numerical (binary) format should be used, because it's the best option and people can be expected to bother writing an acceptor for such important data. There are plenty of languages that can easily accept a numerical format, but not JSON; think of, say, Forth. Oh, but everyone writes in Rust and JavaScript nowadays, right.
>When talking efficiency, you might then ask why not to use Cap’n Proto, protobufs or one of those sorts of things. Due to the need for extensibility (e.g. in Fastmail we add a few custom fields to JMAP core and mail data structures), rigid structures are unsuitable; you do need arbitrary key-value arrangement, despite it being larger over the wire.
All of this is unnecessary. Email doesn't need this drivel.
>All in all, JSON is a robust choice for just about any API.
Of course; forty years ago, UNIX was the solution for every problem. The WWW and JavaScript are nothing more than a continuation of this worse-is-better filth.
>Fun fact: at Fastmail we compress both request and response, even using a custom dictionary primed on real JMAP stuff, to improve the compression ratio. I understand standard DEFLATE was about equivalent in size to CBOR (a binary representation of JSON), and the 2KB dictionary brought it down even further, can’t remember how much it was off the top of my head.
I hate how everyone's response to inefficiency is ''Compress it!'' and ''Cache it!''. There's no reason it should be so inefficient to start with.
The fact that Fastmail, a well known business that is entirely based on successful deployment of email to paying customers has defined a protocol for object exchange etc that is based on Internet RFCs and standards is a Good Thing.
"Cretins" don't decide to use JSON, JSON is an object encoding that is supported by all languages and environments. It compresses to the same level as binary based encodings. It allows for arbitrary extensions that can be migrated in production without having to update all clients/servers simultaneously.
Binary formats tend to be brittle and require simultaneous end-to-end upgrades to handle change. Protobufs and other TLV based encodings help with this, but they still require recompilation and synchronization of IDL specifications.
Compression and caching are easy to insert into a network pipeline (and/or a process-to-process pipeline) easily and transparently. If you read Fielding's thesis on REST, one of the advantages of the HTTP "stack" (not HTML) is that caching and other "dependency injection" can occur transparently between the endpoints.
Efficiency is not by putting all your eggs in one basket. Efficiency comes from a) designing the protocol well, b) keeping the encoding decisions well away from the protocol's content so that decisions like caching and compression etc can all be added or removed without changing the underlying implementation.
All languages "accept" a "numerical format" (not sure what you mean here, but I assume you mean binary encoding. Unless you mean arbitrary length binary numbers. Do you think binary should continue to be on octet boundaries? Or are you talking about encoding an arbitrary bit length? How will you deal with word endianess?
JSON has some warts, the quoting of keys is a pain, the fact that comments cannot be included (but ignored) in the same way as whitespace makes it harder to document configurations or other static documents. The use of floating point is another wart, but if you are sending any numeric values that rely on floating point accuracy, your protocol is wrong. Define a fixed point integer definition instead (eg send monetary values in cents, not dollars).
Email obviously does need this "drivel".
https://news.ycombinator.com/item?id=19839104
IMAP was originally conceived essentially to allow a very thin client that proxies every user command back to the user store, and it also embeds some reliance on old implementation artifacts (UIDVALIDITY, anyone?). There are a lot of "fun" corner cases--I once found a message that was interpreted 5 different ways by 4 different implementations. (It doesn't help that MIME structure is actually pretty loopy and maps very poorly to how all modern clients view mail messages, resulting in many interesting interpretations).
Modern email clients are all structured very differently from the "thin client" perspective--everyone keeps their own copy of the email (metadata) database locally, and use IMAP essentially as a database synchronization protocol, but IMAP is pretty bad at synchronization, at least until the LEMONADE additions came in ~a decade ago. IMAP is especially bad at handling multiple folders correctly, since it assumes that every folder is independent of the other ones, and you can only sit and wait for messages on one folder per connection at a time.
JMAP is a generic protocol, but its primary motivating example and the thing that has influenced the design most is email, as a replacement for SMTP (between the user and server, not between servers) and IMAP.
The name JMAP was chosen quite early on. I + 1 = J. The words the acronym stands for have changed a couple of times along the way!