No JSON/MAPS interoperability in 17.0? | Hacker News Reader

No JSON/MAPS interoperability in 17.0?

erlang.org

44 pointsrahij82 comments

Erlangers are particularly grumpy about JSON because unlike many other languages, Erlang lacks a decent default serialization and deserialization of JSON. This is primarily due to the fact that a string is a list of integers, thus making it hard to distinguish between the two, and somewhat less importantly, but still problematic, the klunkiness of the dictionaries. (This message appears to be in the context of the soon-to-come native addition of dictionaries, but as I imply, unfortunately that is not the biggest problem with JSON deserialization, the nature of strings in Erlang is. Binaries are not a precise conceptual match to a JSON string either, unfortunately; in my environment I actually use binaries that way because I can guarantee the differences won't affect me since I control both ends of the protocol, but that is not safe in general.)

This is not to say that the arguments are wrong, but I think you may not understand why Erlangers feel the way they do until you realize that Erlangers also don't generally get to experience the advantages of JSON, either; when you get none of the advantages and only the disadvantages, anything looks bad, no matter how good it may be in the abstract.

This is a particularly rich irony in light of the fact that Erlang's native data model is conceptually the closest language I know to natively using JSON as the only data type. Erlang has no user-defined types, and everything is an Erlang term... imagine if JS worked by only allowing JSON, so, no objects, no prototypes, etc. The entire language basically works by sending "ErlON" around as the messages, including between nodes. It's just that there's no good mapping whatsoever between Erlang's "ErlON" and JSON.

Erlang has amazing default serialization and deserialization. term_to_binary(T) and binary_to_term(B)

(I edited the last paragraph in before seeing your post here.)

The last paragraph is absolutely right. It's incredibly ironic...

Right, but what about JSON?

Parsing the binary blobs for interop is annoying, see for exmaple the Wings file format, made almost entirely of Erlang serialized objects:

http://en.wikibooks.org/wiki/Wings_3D/User_Manual/Wings_File...

https://www.youtube.com/watch?v=wKjxFJfcrcA

The first requirement that I look for in a data interchange format is that it should be readable, because I don't want to look at binary data responses. The second thing I look at is support for encoding and decoding it.

The only three formats I've found so far that are readable and well supported are XML, JSON, and YAML. XML is too hefty and wasteful. YAML has had a bad history of insecure encoders and decoders but overall is my favorite data format. However, it still has the downside of needing a special decoder since browsers don't support it, and it requires specific indentation for its hierarchical data format which is wasteful in its own way.

That just leaves JSON in my opinion. It's easily understood and read, and native browser and Node.js encoding and decoding is more than fast enough.

...I don't want to look at binary data responses

I've always found this pretty semantic. Obviously tool support for reading ascii or UTF-8 encoded text is very strong, but it's all binary, so it's a question of tools. I do find JSON the most palatable of the text-encoded formats, but I'd adore it if a solid binary replacement gained favor. Unfortunately I think a lot of the protocols that have had pretty good energy behind them (pbuffers, thrift, hessian, etc.) end up either going the compiled-stubs route ,or bundle RPC, or both.

I'd really like to see a binary protocol with solid type support (please God, let it define a built-in datetime) that can be written and read dynamically (without a header file/stub).

What about S-Expressions? My electronics CAD program recently switched to using it for footprint and layout files and I am very happy about it. It looks like this in text: https://github.com/KiCad/Footprint_Symbols.pretty/blob/maste...

Are S-Expressions a standardized data format? It looks very similar to Json, but harder to read.

I find it easier to read. It's easier to parse too so it doesn't rely on standardization as much. There was some effort towards standardization but nothing became of it.

https://en.wikipedia.org/wiki/S-expression#Standardization

s-expressions (http://c2.com/cgi/wiki?XmlIsaPoorCopyOfEssExpressions) are another option. Even people who prefer JSON recognize the advantages: http://irreal.org/blog/?p=713

Sexprs are (usually) node-labeled trees (xml is too), while json describes edge-labeled trees. Object-oriented data structures are edge-labeled graphs. Json is a slightly better fit for the most common implementation languages.

The page you linked to doesn't support your argument, BTW. It tries to assert that sexprs support hashes - by extending the syntax with a reader macro!

This line shows the author's confusion: "S-expressions are more powerful (because of the duality of code and data)". The guy is confusing a format that strictly should not have behaviour beyond constructing a data structure, with Lisp more generally. If you allow the data structure to contain code that further interprets the data structure, the complexity of sanitizing input greatly increases.

What on earth is the problem with (dict (key1 value1) (key2 value2))? I would render {people: ["joe", "bob"]} as (dict (people (list "joe" "bob"))).

> I don't want to look at binary data responses.

I keep seeing this but having looked at hundred megabyte large JSON objects or XML files, some might as well be binary to my eyes.

Agreed. And further to this, unless the JSON is indented for humans (not common) you still need a tool to format it. So if you're using a tool anyway, you could just as easily have used a binary format.

But that's a special case. Most JSON messages are probably short enough to read without a special tool.

The largest JSON file I ever dealt with was a few megabytes. 99% or more of JSON I deal with is well under a kilobyte. The vast majority is no more than a few hundred bytes.

Can I easily copy and paste a binary format between text editors and emails and IM? Can I easily write that binary format by hand?

If your answer is "just transform it to/from JSON", I ask this: Why bother, then, since JSON itself works just fine, and avoids unnecessary layers?

With JSON, I get something that can be both read by a human and processed a computer with a minimum of translation and abstraction. I know exactly what is happening every step of the way, and can trivially verify what data is being passed around by analyzing logs, sniffing network packets, and even delving into memory dumps.

I don't think you've dealt with JSON enough. I live and breathe JSON. It is the lingua franca binding together a couple dozen developers working on seven different services written in six different languages (including, incidentally, erlang, which certainly wasn't my idea), and myriad client implementations in at least as many languages and even more runtimes on a dozen different platforms. (And these numbers are actively growing.)

We could not work with a binary format. This is a matter of practicality. There are developers of every skill set and experience level involved, there are third-parties involved (including those with strict control over some platforms that severely limit what kinds of code we can run), there are even customer-service reps involved who see this stuff.

JSON is easy to parse even in languages that make binary data difficult to deal with. It's easy for humans, wizard and muggle alike, to both read and create. As a text format it interacts well with the existing tools the entire world uses on their computers every day.

To say we should just use a binary format because we could conceivably spend time writing new, less-integrated, less-convenient tools to work around the problems it gives us is to miss the forest for the trees. We don't need better performance, and we've got enough problems to deal with without creating more.

> The only three formats I've found so far that are readable and well supported

It probably doesn't fit your "well supported" criterion I suppose, but how about Clojure's EDN? I've been meaning to try it out more myself.

I always use "python -m json.tool" to pretty-print JSON data before looking at it. I could just as easily do the same with a MessagePack or BERT pretty-printer.

Let me also add Rebol into the mix. To quote Carl Sassenrath...

Every time I run across JSON examples, I see REBOL without the elegance. The two languages are related of course. REBOL strongly influenced the design of JSON.

ref: On JSON and REBOL - http://www.rebol.com/cgi-bin/blog.r?view=0522 (HN - https://news.ycombinator.com/item?id=5654895)

This essentially amounts to "JSON is slower and more CPU intensive than it absolutely needs to be."

I can live with that. Any solution that people come up with, there will be folks who can point out some sort of flaw in it. Sometimes the flaws are worth paying some attention to, when fixing them might lead to some tangible benefits. It's difficult to imagine a slightly less CPU intensive replacement for JSON bringing any tangible benefits with it, though.

The note on numbers can be specifically problematic since the format does not specifically restrict numbers to be represented by doubles but many clients will assume so, creating some hard to find bugs (it's bitten me many times). The others seem more minor compared to its lack of things like maps that support non-string keys.

Hmm.... isn't JSON required to be valid Javascript?

Javascript assumes all numbers are double-precision FP, yes?

> isn't JSON required to be valid Javascript?

Not necessarily. See JSON: The JavaScript subset that isn't - http://timelessrepo.com/json-isnt-a-javascript-subset

The lack of hypertext is a valid complaint, in my opinion, for a language designed in the web age. Sure, you can build a format on top of JSON that treats certain values as URIs, but that kind of thing should be built-in.

Just curious, but how do you envision hyperlinks working in JSON? And what's a use case for this? Generally I imagine putting "next" sorts of links in a JSON document is the way I resolve this, but I could imagine wanting to put a link directly on the text that would take you to more detail about the text. Is that the use-case? I guess I haven't needed that or I haven't known I've needed it because it isn't there :p

I'd copy it from the RDF world:

  {
    'author': {
      'name': 'Jack',
      'homepage': <http://jackshome.com/>,
    }
  }

As for the use case, the easiest to conceive is JSON-reading search engines. Think about what we could do with all that structured data ;)

I don't quite follow the whole hypertext problem. I mean URIs can be encoded as strings, there's no problem there.

HTML doesn't handle URIs any differently. They're just string attributes on elements(generally).

Is the problem that there aren't clients that can generally consume the JSON and find the hyperlinks? Is the idea that the JSON should be able to be rendered as a site kind of like the whole xml/xslt idea?

I guess I just don't get what the problem is here? What would built-in uri's look like?

HTML doesn't handle URIs any differently. They're just string attributes on elements(generally).

Sure it does, when it defines the semantics of the elements. HTML says "the 'href' attribute of the 'a' element is the URI of a linked resource". There's no special syntax, because it doesn't need to have; they're identified by their elements.

But on a more general format that doesn't define particular semantics to nodes, you have syntax that identifies URLs. For example, in Turtle - which is a format that I quite like - you encode strings using double quotes around the value, but URIs are encoded using angle brackets.

Example:

  dc:title "RDF/XML Syntax Specification (Revised)" ;
  ex:editor [
    ex:fullname "Dave Beckett";
    ex:homePage <http://purl.org/net/dajobe/>
  ] .

The advantage is that smart clients can parse that information and crawl the documents, generating useful data even if they don't necessarily understand this particular format. Think search engines, for example - they might not understand that <span id="author"> identifies the author's name, but they can still derive useful information for the web of pages that they crawl.

Using something other than JSON is a great idea if you don't want anyone to ever bother using your API.

Content negotiation is a great way to expose multiple serialization formats. No reason it has to be just one of the well known formats. The main tricky part is equivalence of more complex structures in JSON, for example msgpack supports more than just string keys in maps.

So now I get to support more than one format, with corresponding increases in complexity...

..But only the JSON format will have any users, and I still have to deal with anything I find objectionable about JSON.

In most systems I've deployed multi-format support it was a matter of a few extra lines of code rather than a doubling of effort. Pretty much all JSON values have obvious (aka. automatic) equivalents in other formats. JSON is truly the lowest common denominator, which is its strength.

The adage "The simplest things that could possibly work" comes to mind. For many things JSON is good enough.

Going with JSON first makes sense, if you need to create an alternate more efficient protocol in the future you'll at least have a few more data points to use when selecting something new like msgpack/protobuffers/thrift or whatever.

Only an Erlang mailing list would prompt someone to complain that a document format requires valid UTF-8.

This is a glimpse of the last place on earth where relying on US ASCII is considered a positive good.

I was thinking to myself while reading the link "Why on earth would someone think UTF8 is a problem?" and then I saw it was a erlang thread. To complain about it being slow to validate while talking in an erlang thread... christ. It must be frustrating trying to deal with json, which is full of strings, in a language like erlang.

JSON decoders decode string to binaries.

https://github.com/talentdeficit/jsx

Erlang has been receiving and sending data. It just like to deal well defined binary messages and it likes to encode/decode them into its own representations (records, terms) at the boundary where they come in and leave the system.

The thing is that JSON doesn't actually require or otherwise specify UTF-8. A JSON encoding is defined as a sequence of unicode codepoints: it doesn't actually tell you how to send those over the wire. You have to do that yourself, and specify the encoding out of band somehow.

If JSON did actually specify UTF-8 and provided a sensible escape mechanism for the whole of unicode, this would be an improvement, because it would turn JSON into an actual standard data-exchange format, rather than a textual encoding of its infoset.

They're not talking about strings in JSON; they're saying that a JSON-serialized blob must itself be valid UTF-8. So, to decode JSON, you first have to parse the whole thing to convert it to codepoints (and possibly bail at this step)—and then (lex-and-)parse those codepoints again to turn them into a data structure, making sure to pass through things that turn out to be strings as the same sequence of codepoints you already parsed out rather than passing them to a UTF-8 decoder there and then, like you would in any other format.

Erlang's internal serialization format, meanwhile—and all the language's native pattern-matching constructs—are built on what are basically "generators" that consume a (possibly-infinite) binary stream, and lex tokens directly out of it. JSON doesn't really work with this approach; what you end up doing is having one generator that consumes the binary stream and emits codepoints, and then another generator that consumes codepoints and emits tokens. This introduces a lot of intermediate allocations and message-passing—whereas, with most Erlang protocol handlers, your TCP handler passes you a slice of a VM-managed shared binary, and then you just pass around and re-slice that slice.

Prefixed length formats that don't make assumptions on encoding are a very good pick. The most important thing is that once you read the length, you can just bulk-read the payload without actually parsing it, that's up to the higher layer (if parsing is needed at all).

A general purpose serialization format that requires per-char processing is a terrible pick.

Agreed. I had to implement base64 encoding to pass data through JSON, yeah it works but seems terribly wasteful.

> Its numbers representation is double-precision floating-point, meaning it's incredibly imprecise and limited.

Except that JSON actually just specifies the number type as an arbitrary precision decimal number.

Many implementations use floating point numbers when decoding JSON, but that is not inherent in JSON.

My biggest complaint about numbers in json is that often times floating point numbers get turned into integers when encoding/decoding using some implementations. (e.g. (float)2 gets encoded as just 2 and when decoding, it is an integer rather than a floating point.

Well, regardless of the specification, most JSON deserializers approximate numeric values as floats, at least in the default configuration. Enough so, that the Bitcoin exchange JSON-APIs I know of all put numbers in strings. So `{"balance": "12.65334221", "currency": "BTC"}` instead of `{"balance": 12.65334221, "currency": "BTC"}`. It would easily be pretty catastrophic if there are any rounding errors in your currency calculations, and if the bug arises already in your JSON deserializer it might be hard to detect even if your program internally employs BigNums (arbitrary precision) or such.

JSON and Erlang just don't get along in the way you'd like them to, unfortunately. Some of that has to do with how Erlang handles (or doesn't handle) strings. Some of that was that there was not a data structure that closely mirrored JSON's structure in Erlang up until Maps were introduced.

The problem isn't really JSON - JSON is an exceptional format for what it is supposed to do - the problem is that Erlang was created for a specific purpose and that purpose wasn't to vend out strings/JSON over HTTP.

> Erlang handles (or doesn't handle) strings.

JSON string are translated to binaries. There is no reason to keep repeating oh inters=strings. Maps will help.

> The problem isn't really JSON

I think the author disagrees. He talks about the problems of "JSON". There are a few he notes:

* No binaries.

* No way to have hyperlinks

* Limited floating point number implementations

You have to translate your strings to binaries before you try to encode to JSON. A JSON encoder has no way of determining whether it should be encoding an array or a string without this. Instead of one string type, you have "strings" and <<"strings">>, and due to the lack of typing, it is actually pretty tough to keep track of which one you have. It also is an exercise in finger acrobatics to <<>> all your strings.

JSON doesn't solve every problem. It does solve the problem, quite elegantly, of creating a 'endianless' interchange format that is easily consumed, constructed, and debugged.

That reasons exists since a decade with emails. And did we change the system ? No, because it is simple and it work.

Since the author seems to have a problem with text-based formats in general, here's a counterpoint by Mike Pall on the LuaJIT mailing list [1]:

"On a tangent: IMHO most binary serialization formats like BSON, MessagePack or Protocol Buffers are misdesigns. They add lots of complications to save a couple of bytes. But that does not translate into a substantial improvement of the parsing speed compared to a heavily tuned parser for a much simpler text-based format."

[1] http://www.freelists.org/post/luajit/Adding-assembler-code-t...

So don't use JSON when you need super high performance and be careful about parsing JSON numbers?

It may not be perfect, but compared to the complexity of the XML world and the opaqueness of binary formats JSON is a very pleasant compromise.

Shame about it not having comments though.... :-)

The author suggests using MessagePack, which I hadn't seen before but looks really cool.

http://msgpack.org/

It sure looks impressive, thanks for linking

This does look pretty cool. I think the author is forgetting about so many reasons why JSON is so widely used, however.

One huge benefit of JSON over msgpack or something similar is READABILITY.

So MessagePack wants me to ditch one of the most portable widely used data transfer formats in the world to save 33% on message size? No thanks. I'm willing to bet that using Content-Encoding: gzip on the JSON would negate most of the size difference anyway.

But that also uses double precision floating point number representation and UTF-8 for strings, so the only benefit (according to the arguments given in the OP) seems to be text vs. binary-based.

> But that also uses double precision floating point number representation

I don't think JSON requires the number representation to be double-precision, that can be up to the parser to decide how to represent a number. I think most people misrepresent this because the number implementation in JavaScript is double-precision, but that's not necessarily true for JSON. Am I wrong about that? Does JSON require the number format to be double-precision?

I had to check but yeah, that's true, the ECMA spec even says explicitly in its intro: "JSON is agnostic about numbers", they are "only a sequence of digits".

(Unrelated observation from looking at the spec again: there's no standard/recommended decoding behaviour if an object has several members with the same key.)

>I do not have a magical format to propose to replace it

So why did you even comment in the first place?

Seriously, he just lost his own argument right there.

He did say he uses msgpack which he claims is a step in the right direction, although it too has its issues.

I don't agree with you. I think that someone can point out problem without providing the solution at the same time.

He didn't just point out a problem, he actively tells people to "stop using JSON" without providing an alternative.

But when he said "Stop using JSON" he needs to provide a better alternative, otherwise what is the point of the complain? If he don't have a better alternative, so he should rename to "JSON sucks" or something like that.

You're right, but that's so easy to do unless the problem matters and is something that people are not easily able to realize on their own.

In this case, I'd say JSON not being the fastest it could possibly be doesn't matter too much. As a result, pointing that out isn't particularly valuable, but could be a bit more valuable if there was an easy-to-adopt solution also being proposed.

If you're going to post against such a widely used format, you had better have a very solid reason, and an even more solid alternative.

I think he means not having something to replace in general doesn't mean not having something to replace it with in particular instances. Maybe there isn't a good general all-singing-and-dancing format.

There are language specific serializations if you never leave that language environment. There are protobufs, ASN.1, messagepack, thrift, bert, avro.

I am personally am not rushing to replace it but I can see the point he is making.

    > *  It has to be valid UTF-8, meaning it's incredibly
    > slow to validate.

If make basic mistakes about what the format allows, the rest of your comment is liable to be junk.

> It's text-based, meaning it's incredibly slow to parse.

This is precisely the reason it is used so heavily. Easily readable format to reason about. And incredibly slow is a pretty relative term, when you're waiting on database calls or doing other complex logic that is orders of magnitude slower, who cares how "slow" json parsing is.

> It has to be valid UTF-8, meaning it's incredibly slow to validate.

I think being able to embed all sorts of different characters and languages is, again a plus. See argument above about performance.

> Its numbers representation is double-precision floating-point, meaning it's incredibly imprecise and limited.

This argument I don't get. I'm pretty sure I might be missing something, but in my experience you can just put a plain old integer and any parser in any language will extract as an exact integer. Nobody is converting a "1" in json into a double/float. Maybe somebody can elaborate and what the author might have actually meant?

So, really, the argument all boils down to, it's slow and wasteful. Well, while that's true I think it's pretty much been established time and time again that Moore's law has made it possible to value programmer time over CPU overhead, to a reasonable extent (i.e. if the overhead you're adding overtakes Moore's law and makes infrastructure particularly expensive). If you have a format that is human readable, easy to understand, and simple, that helps tremendously in software development and it would take order of magnitude performance hits to really make it bad tradeoff (and even the, if you weren't getting a lot of traffic, who would care?), not just 2x or 3x.

We live in a web based world, and that world is fundamentally based on a text based protocol (http) and text based messaging formats. There are plenty of valid and good reasons why it happened the way it did.

*Final Note: I would like to see some empirical evidence to compare the wasted CPU cycles and energy that JSON uses compared to if all messages were sent with msgpack or something like it instead. While my own inclination is that number would be dwarfed by the overall energy used in computation I would prefer to see evidence rather then conjecture if you're going to make a point like that.

>> Its numbers representation is double-precision floating-point, meaning it's incredibly imprecise and limited.

> Maybe somebody can elaborate and what the author might have actually meant?

I think JSON numbers are not necessarily double-precision, at least I don't think I've ever seen them specifically described this way. In JavaScript, however, numbers are double-precision, there are no real integers in JavaScript. So maybe the author misrepresented JSON numbers because JavaScript numbers are double-precision. But that isn't necessarily the case AFAIK.

Ok, I understand your argument, now tell me a better format to create an API, Thanks ...

Turtle!

If msgpack is able to match JSON for serialization/deserialization, couldn't you make an API in JSON and offer msgpack as an alternative format?

In a project I'm working on, we do that with Protocol Buffers. Our main client uses protobuf but we still provide JSON for other clients. We also use the JSON API for debugging.

Here the reasons why i use JSON:

* It's text-based, meaning it's readable and easy to parse.

* It has to be valid UTF-8, no hassle with other formats.

* Its numbers representation is double-precision floating-point, i can choose what i want.

Who cares? Use what you want and what solves your problem at the best way.

Servers aren't CPU bound anymore, there is no need to worry about that...

Well, it depends on your scale.

It depends a lot on the service that you provide. I provide compute services. Most of my servers are CPU-bound most of the time.

Yes, because XML is so much more compact.

Oh, wait....

I don't see xml mentioned once, I'm sure he would say it's worse. He'd clearly probably advocated some sort of binary format, since he mentioned msgpack. Did you even read the comment?

At least I can comment my xml config files. I hate that all my JS config is in JSON with no explanation in the file for why certain parameters were set.

Show me in that post where the word XML was used.

Oh, wait, you haven't read the post.

Oh, now that you mentioned it, I'll use XML, It's very cool and compatible with new technologies, like a new thing of Microsoft called C#.

Reply on news.ycombinator.com