JSON is an incredibly inefficient format for shareable data: it is annoying to write, unsafe to parse and it even comes with a lot of overhead (colons, quotes, brackets and the like).
I'd prefer s-expressions.
I'd prefer s-expressions.
Personally, I prefer binary formats where feasible, because I like pursuing efficiency to extremes, and write all my own code in Rust.
Professionally, I happen to write mostly JavaScript, and no Rust at all, but we do still focus on performance at Fastmail. (It’s in the name!)
If you’re comparing JSON to s-expressions, it’s a wash in efficiency. They’re both “inefficient”, both representing generic data structures in regular text. They use slightly different punctuation, but that’s about it. The differences are negligible.
But when it comes to support, JSON is well-supported in all mainstream languages, while s-expressions have no central specification on precise syntax, and you’ll need to do your own thing. One of the deliberate design advantages of JMAP over IMAP is doing away with custom parsers, and using the standard tools people are familiar with, HTTPS and JSON.
When talking efficiency, you might then ask why not to use Cap’n Proto, protobufs or one of those sorts of things. Due to the need for extensibility (e.g. in Fastmail we add a few custom fields to JMAP core and mail data structures), rigid structures are unsuitable; you do need arbitrary key-value arrangement, despite it being larger over the wire.
All in all, JSON is a robust choice for just about any API.
Fun fact: at Fastmail we compress both request and response, even using a custom dictionary primed on real JMAP stuff, to improve the compression ratio. I understand standard DEFLATE was about equivalent in size to CBOR (a binary representation of JSON), and the 2KB dictionary brought it down even further, can’t remember how much it was off the top of my head. Neil Jenkins gave a talk about this and related matters last month, https://www.webdirections.org/code/speakers/neil-jenkins.php, no public video sadly. I’ve thought about writing a blog post about it all. An unfortunate side-effect of this compression is that you can’t inspect traffic in the network panel of the dev tools; you can add ?compress=0 to disable it, if you’re an interested Fastmail user.
MessagePack has an ugly history with text data that's still reflected in some of the libraries interfaces. E.g., the Python library doesn't round-trip strings by default. (They'll turn into bytes objects.) There's a flag to do the right thing, but not everyone is aware of it.
Add to that that CBOR has an RFC, and I'll pick it over MessagePack every time.
Binary formats are mostly nicer from a parsing perspective: it's much easier to unpack. You're right, I think, that simple compression will mostly close any size gap.
I don't think that's a deal-breaker. protobuf has an extension mechanism. [1] You could have a registry for an org to claim an extension number to a given message (and then use it to hold their own message with as much stuff in it as they like). I'm not as familiar with cap'n proto but assume it has something similar.
That said, I think JSON was a fine choice for this purpose. The efficiency is good enough with standard implementations; and there are much faster-than-standard implementations available if you have a special reason to want one. [2]
[1] https://developers.google.com/protocol-buffers/docs/proto#ex...
What do you mean? Both Protobuf and Cap'n Proto are designed to be extensible. You can add new fields without breaking compatibility.
(Disclosure: I wrote Protobuf v2 and Cap'n Proto.)
That lets you store the the common things in highly compressed protobuf, and extensions can either be done via extension protos, or with stringly-typed map stuff for testing/weird stuff.
Forgive me; I am not speaking from a strong position of grokking Cap’n Proto or Protobuf (I’m casually familiar with them but have really only once used them, maybe three years ago), and I believe I erred in what I said.
For Cap’n Proto, I don’t believe I’m far off the mark: to pick a simple example, extensions may add fields anywhere, and I believe you’re not allowed to skip field numbers, so you can’t just establish a registry of used field numbers and pick and choose the extensions.
proto3 looks like it probably can’t handle this either.
On proto2, though, it looks like I was approximately wrong, and you could have finite extensibility, and maintain a registry of assigned IDs so that extensions can be compatible. I’m not sure how suitable that would be in practice—the examples I see only allot a hundred numbers for extensions, which seems rather tiny; I’d picture allowing at least a million.
Anything like Map<str, str or any or whatever> I’m going to disqualify from consideration because either you use structs in some places and thereby limit the places that extension can apply, or you use maps right through and then get no benefit over JSON.
So I think they’d still be mostly technically unsuitable, but maybe Protobuf v2 would work. And maybe I’m completely wrong still—you’re the expert! But really, unless you get a major advantage out of such a thing, an important consideration is social suitability. Protobufs and the likes are not widely-used, require more effort to work with, and require more involved library support than JSON, which is much more universally understood and handled. Just look at how I’ve been fumbling around with them!
That is indeed tricky. With protobuf you could claim some high-numbered fields and hope that no one else does the same thing -- with some randomization you can probably be pretty confident. With Cap'n Proto you can't really do it at all since fields are identified by offset, and making structs arbitrarily large is probably not what you want. With JSON, because fields are identified by name, it's possible to choose a name that almost certainly will not collide with anyone else.
I'd argue that the right answer here would be to ask the spec owners to add extension points, which hopefully would not be controversial. Conceptually this could take the form of something like map<string, any> (there are cleaner idiomatic ways to do it in each of Protobuf and Cap'n Proto but the details are beside the point of this conversation).
There is: https://people.csail.mit.edu/rivest/Sexp.txt
It's imperfect, but it does get the job done. Among its advantages is that it's both pure text and binary: there's a canonical representation which is binary, and an advanced representation which is textual, and they map 1:1 to one another.
Examples:
(session
(capabilities
(urn:ietf:params:jmap:core
(max-size-upload 50000000)
(max-concurrent-upload 8)
(max-size-request 10000000)
(max-calls-in-request 32)
(max-objects-in-get 256)
(max-objects-in-set 128)
(collation-algorithms ("i;ascii-numeric"
"i;ascii-casemap"
"i;unicode-casemap")))
(urn:ietf:params:jmap:mail)
(urn:ietf:params:jmap:contacts)
(https://example.com/apis/foobar
(max-foos-finagled 42)))
(accounts
(A13824
(name "john@example.com")
(is-personal true)
(is-read-only false)
(account-capabilities
(urn:ietf:params:jmap:mail
(max-mailboxes-per-email nil)
(max-mailbox-depth 10))
(urn:ietf:params:jmap:contacts
…)))
(A97813
(name "jane@example.com")
(is-personal false)
(is-read-only true)
(account-capabilities
(urn:ietf:params:jmap:mail
(max-mailboxes-per-email 1)
(max-mailbox-depth 10)))))
(primary-accounts
(urn:ietf:params:jmap:mail A13824)
(urn:ietf:params:jmap:contacts A13824))
(username "john@example.com")
(api-url "https://jmap.example.com/download/{accountId}/{blobId}/{name}?accept={type}")
(upload-url "https://jmap.example.com/upload/{accountId}/")
(event-source-url "https://jmap.example.com/eventsource/?types={types}&closeafter={closeafter}&ping={ping}")
(state "75128aab4b1b"))
This is the equivalent of the JSON:
{
"capabilities": {
"urn:ietf:params:jmap:core": {
"maxSizeUpload": 50000000,
"maxConcurrentUpload": 8,
"maxSizeRequest": 10000000,
"maxConcurrentRequest": 8,
"maxCallsInRequest": 32,
"maxObjectsInGet": 256,
"maxObjectsInSet": 128,
"collationAlgorithms": [
"i;ascii-numeric",
"i;ascii-casemap",
"i;unicode-casemap"
]
},
"urn:ietf:params:jmap:mail": {}
"urn:ietf:params:jmap:contacts": {},
"https://example.com/apis/foobar": {
"maxFoosFinangled": 42
}
},
"accounts": {
"A13824": {
"name": "john@example.com",
"isPersonal": true,
"isReadOnly": false,
"accountCapabilities": {
"urn:ietf:params:jmap:mail": {
"maxMailboxesPerEmail": null,
"maxMailboxDepth": 10,
...
},
"urn:ietf:params:jmap:contacts": {
...
}
}
},
"A97813": {
"name": "jane@example.com",
"isPersonal": false,
"isReadOnly": true,
"accountCapabilities": {
"urn:ietf:params:jmap:mail": {
"maxMailboxesPerEmail": 1,
"maxMailboxDepth": 10,
...
}
}
}
},
"primaryAccounts": {
"urn:ietf:params:jmap:mail": "A13824",
"urn:ietf:params:jmap:contacts": "A13824"
},
"username": "john@example.com",
"apiUrl": "https://jmap.example.com/api/",
"downloadUrl": "https://jmap.example.com
/download/{accountId}/{blobId}/{name}?accept={type}",
"uploadUrl": "https://jmap.example.com/upload/{accountId}/",
"eventSourceUrl": "https://jmap.example.com
/eventsource/?types={types}&closeafter={closeafter}&ping={ping}",
"state": "75128aab4b1b"
}
Yes, clearly it's the JSON. How do you tell at a glance in the s-expr formatted data what's an object and what's an array? What's a keyword and what's a literal? I can barely even tell the heirarchy where it's very visually obvious with the JSON.
The JSON and s-expr have a different indent level, which is why the hierarchy is less obvious to you.
Debatable. To me there is a clear difference between a list of things of the same type, and a mapping of names to values of potentially varying types. I don't think the specific programming language really matters, they are two totally different concepts with much different applications.
> There's also no way in JSON to distinguish between an int and a float and a decimal type
True, but they are just small literals which are easy to differentiate at a glance, not big multi-line structures like objects or lists. Plus s-expressions have the same issue.
There are no objects — it's all lists (or arrays, if you prefer). This is a good thing, because ordering is always the same.
> What's a keyword and what's a literal?
Everything's a byte string: any interpretation above that is a matter for the protocol. This is a good thing, because it means you neatly sidestep issues like JSON's numbers actually being floats rather than integers, as well as issues like numeric precision.
'But I want types!' Of course, they are good. But the type of a number isn't just integer or float: it's integer-which-is-a-known-version or float-which-is-between-zero-and-one-exclusive. You need to check those types at ingestion anyway; it's not a big leap from integer-which-is-a-known-version to string-which-parses-as-an-integer-which-is-a-known-version.
> I can barely even tell the heirarchy where it's very visually obvious with the JSON.
They are both indicated with indentation. Probably I should have used two spaces for the JSON indentation, but I was lazy.
Why not? I'm honestly curious, because I cannot comprehend preferring that mess of JSON to the s-expressions. The JSON just looks like a dog's breakfast, while the s-expressions look clean & easy to write, read, parse and use.
Also, I don't think they are functions: they're just lists with a convention. Those lists can be used to expression function calls, as in Lisp, but in a data format I think that the lists more often express structure.
One thing I love about s-expressions is that one indicates the type as the first item in the list, e.g.:
(foo …)
{
"@type": "foo"
}
Why are the quotes needed? It would be a simply change in the spec if names without spaces in them could be expressed without quotes.
Having to quote the symbols is terrible though.
I guess that's part of the point from my point of view: it's not about hash tables or objects or even lists, but about what they represent. In my view focusing on the implementation (a hash table mapping strings to hash tables mapping strings to numbers or strings) is looking at the trees and missing the forest.
> it even comes with a lot of overhead (colons, quotes, brackets and the like).
according to Sqlite devs the that overhead is not much worse that for a binary encoding.
quoting: https://www.sqlite.org/json1.html
> The json1 extension does not (currently) support a binary encoding of JSON. Experiments have been unable to find a binary encoding that is significantly smaller or faster than a plain text encoding. (The present implementation parses JSON text at over 1 GB/s.)
Try to write a parser for an arbitrarily complex and nested JSON object without running into runtime errors even after the fourth-or-so revision. There are too many "yes buts" in the living standard.
> that overhead is not much worse
It _is_ much worse than the two-byte "overhead" of one s-exp node.
I don't believe you. Atoms are roughly the same, and compounds, well:
[1,2,3] vs
(1 2 3)
{"a":1,"b":2} vs
(("a"1)("b"2))
(You might suggest plists, ("a"1"b"2), for dictionaries, in which case you save a whopping two bytes per entry. But I'd choose the nested-list format over plists, if I were encoding a dictionary into an S-expression.)
Which pretty much disappears when compressed anyway.
Anyway, for being a Fasmail user, JMAP rips IMAP to shreds in terms of performance. I've been following JMAP from afar and waiting for Fastmail to publish this for a long time.
People have ben actively doing this for XML with streaming parsers that emit events, where you don't just validate XML itself but the semantics of the payload you're receiving, and bail out early instead of parsing the whole XML into a DOM then validating the DOM structure and values.
Could you give an example? The only deviation from the standard I've run into is JSONL being called JSON, which isn't actually meaningfully harder to parse.
I feel like it should be possible to translate from valid JSON to valid sexpr with a regexp parser (just match parenthesis and stuff)
> There are too many "yes buts" in the living standard.
I am not an expert and do not want to sound dismissive, but I feel like most of those "yes buts" are a solution to otherwise incompatible implementations. I fail to see where S-expressions have an advantage here. Is there a widely recognized standard that most implementations respect?
Do you mean parsing the JSON bytes itself into maps and lists and atoms, or turning that into an application-useful object?
If the former, are you claiming that Python's standard json library, libjansson, JSON::XS, etc. have bugs? (I don't know why you would roll your own JSON parser in production code.)
If the latter, how are S-expressions any different?
(And there are non-buggy libraries that do both, e.g., serde.)
Edit: I see, JMAP moves attachments out of band
Base64 mixes the byte, but Hexadecimal does not.
1.5. JSON as the Data Encoding Format
JSON is a text-based data interchange format as specified in
[RFC8259]. The Internet JSON (I-JSON) format defined in [RFC7493] is
a strict subset of this, adding restrictions to avoid potentially
confusing scenarios (for example, it mandates that an object MUST NOT
have two members with the same name).
All data sent from the client to the server or from the server to the
client (except binary file upload/download) MUST be valid I-JSON
according to the RFC and is therefore case sensitive and encoded in
UTF-8 [RFC3629].
A quick scan of it says to me it doesn't get all the issues but it does get the biggest one, which IMHO are the ambiguities in what constitutes a "number" in JSON. I-JSON explicitly limits it to 64-bit floats.
Not sure there is a decent standard spec and a good Java (say) parser around. Is there?
There are no RFCs for s-expressions, but that's fixable, I guess. And yes - there are a few Java parsers for them.
Whether it's the “wrong thing” is another issue, though I've yet to see a standard of “wrong” proposed in the thread for which that is accurate.
And, perhaps, all those people know better than you.
Wide adoption and popularity trumps "being wrong" everyday.
And not just for web stuff: the way you describe a custom architecture to rustc is via JSON.
Here's a Java parser: https://github.com/bpsm/edn-java
[0]: http://tutorials.jenkov.com/ion/ion-performance-benchmarks.h...
s-expressions seem like an out-of-left-field choice.
That's subjective and it seems plenty of people have the opposite opinion.
> unsafe to parse
Is this based on theoretical concerns or practical experience?
> comes with a lot of overhead
Keep in mind the alternatives it is largely replacing. XML, text-based email protocols each with their own unique grammars, etc.
> I'd prefer s-expressions.
Ah yes, the plaintive cry of the lisper in the desert. ;-)
I'm afraid in the 60+ years that s-exps have had to catch on in the mainstream, they haven't.
I can't explain this result but nonetheless it seems irrational to expect it to suddenly change.
I actually have a (I think) rather novel approach and there's some potential for little optimizations here and there, but it disgusts me that I'm doing this at all. The optimizations popular, shitty formats like JSON receives are purely wasteful.
It's suitable, though. The WWW wasn't made right from the start and there's always an idiot somewhere looking to add more shit to the pile. Not only does JavaScript exist, but it morphed into this ugly, inefficient, very poor ''format''. We don't have Lisp machines, but new processors are getting instructions to cope with JavaScript idiocy.
A numerical (binary) format is clearly best. It's small, random-access when properly designed, and efficient, but hey, everything's being pushed into port 80 so why not push all data into JSON?
Now, the rest of this message is in response to @chrismorgan from an organization pushing this filth.
>Personally, I prefer binary formats where feasible, because I like pursuing efficiency to extremes, and write all my own code in Rust.
We agree on that, then, but I prefer to write that Rust is just the effeminate man's Ada.
>If you’re comparing JSON to s-expressions, it’s a wash in efficiency. They’re both “inefficient”, both representing generic data structures in regular text. They use slightly different punctuation, but that’s about it. The differences are negligible.
This is wrong. Firstly, we should clarify what is meant by S-expressions. Let's assume the S-expressions have lists, strings, symbols, and numbers. That's already simpler than JSON. You're not hampered by JavaScript's floating point nonsense, either. There's fewer edge cases and less escaping. JSON is, at best, a worse-is-better form of S-expressions.
>But when it comes to support, JSON is well-supported in all mainstream languages, while s-expressions have no central specification on precise syntax, and you’ll need to do your own thing. One of the deliberate design advantages of JMAP over IMAP is doing away with custom parsers, and using the standard tools people are familiar with, HTTPS and JSON.
So, this ties into what I wrote before I even read your post, exactly. Cretins continue to push everything into the WWW protocols and act like their garbage is standard, simple, acceptable, or wanted; it's only technically standard, but it's a bad standard. I'd bet my foot you also like ''DNS over HTTPS''. A numerical (binary) format should be used, because it's the best option and people can be expected to bother writing an acceptor for such important data. There are plenty of languages that can easily accept a numerical format, but not JSON; think of, say, Forth. Oh, but everyone writes in Rust and JavaScript nowadays, right.
>When talking efficiency, you might then ask why not to use Cap’n Proto, protobufs or one of those sorts of things. Due to the need for extensibility (e.g. in Fastmail we add a few custom fields to JMAP core and mail data structures), rigid structures are unsuitable; you do need arbitrary key-value arrangement, despite it being larger over the wire.
All of this is unnecessary. Email doesn't need this drivel.
>All in all, JSON is a robust choice for just about any API.
Of course; forty years ago, UNIX was the solution for every problem. The WWW and JavaScript are nothing more than a continuation of this worse-is-better filth.
>Fun fact: at Fastmail we compress both request and response, even using a custom dictionary primed on real JMAP stuff, to improve the compression ratio. I understand standard DEFLATE was about equivalent in size to CBOR (a binary representation of JSON), and the 2KB dictionary brought it down even further, can’t remember how much it was off the top of my head.
I hate how everyone's response to inefficiency is ''Compress it!'' and ''Cache it!''. There's no reason it should be so inefficient to start with.
The fact that Fastmail, a well known business that is entirely based on successful deployment of email to paying customers has defined a protocol for object exchange etc that is based on Internet RFCs and standards is a Good Thing.
"Cretins" don't decide to use JSON, JSON is an object encoding that is supported by all languages and environments. It compresses to the same level as binary based encodings. It allows for arbitrary extensions that can be migrated in production without having to update all clients/servers simultaneously.
Binary formats tend to be brittle and require simultaneous end-to-end upgrades to handle change. Protobufs and other TLV based encodings help with this, but they still require recompilation and synchronization of IDL specifications.
Compression and caching are easy to insert into a network pipeline (and/or a process-to-process pipeline) easily and transparently. If you read Fielding's thesis on REST, one of the advantages of the HTTP "stack" (not HTML) is that caching and other "dependency injection" can occur transparently between the endpoints.
Efficiency is not by putting all your eggs in one basket. Efficiency comes from a) designing the protocol well, b) keeping the encoding decisions well away from the protocol's content so that decisions like caching and compression etc can all be added or removed without changing the underlying implementation.
All languages "accept" a "numerical format" (not sure what you mean here, but I assume you mean binary encoding. Unless you mean arbitrary length binary numbers. Do you think binary should continue to be on octet boundaries? Or are you talking about encoding an arbitrary bit length? How will you deal with word endianess?
JSON has some warts, the quoting of keys is a pain, the fact that comments cannot be included (but ignored) in the same way as whitespace makes it harder to document configurations or other static documents. The use of floating point is another wart, but if you are sending any numeric values that rely on floating point accuracy, your protocol is wrong. Define a fixed point integer definition instead (eg send monetary values in cents, not dollars).
Email obviously does need this "drivel".