Better Than JSON? | Hacker News Reader

Better Than JSON?

wiki.alopex.li

128 pointsfanf2148 comments

On XML:

  > Not sure anyone really knows how XML happened. It’s 
  > basically the W3C’s fault, I think? It’s okay for some 
  > things but in the end I’m not sure it’s something anyone 
  > actually wants to use, it’s just going to be one more of 
  > those mistakes of the past.

Look, I was doing web dev when XMLRPC was in. For simple API stuff, JSON ended up being worlds better. No fiddly XML preamble, no schemas, no envelopes, just enough structure to be able to quickly pull out a couple of fields and we're set. Beautiful.

But let me tell you how much I appreciated XML and schema validation when we were working with fifteen different partners all using a standardized XML format with a schema. These are cats who were putting XSDs into their IDEs, clicking generate to get a class, and then just not caring.

So when our integrations broke because they were sending XML payloads that failed schema validation, our conversations were so easy.

Before you say "yes but technically you could have done the same with $OTHER_THING" -- sure it's possible, but XML is ubiquitous. Shit, our tools were in Ruby.

XML can be a bear, but sometimes you need a bear.

"Sometimes you eat the bear, and sometimes the bear eats you."

You only need to serialize faster than the guy next to you, or something

Much agreement here, and from the exact same usecase. Probably the easiest B2B API integration scenario I worked with in the past was XML + XML Schema validation. And I still think XML is more readable in many cases than JSON, because while verbose the hierarchical structure is very easy to grasp, sometimes I misread an entire JSON document at first because I missed a quick "[" or "[{" (yes I have also misread XML documents because of the sheer weight of text you are presented with, so I'll call that one a wash).

I do agree with the XML criticisms around writability and complexity, though I think if you made a Venn diagram of the actual complexities inherent in managing a multi-client document-exchange system and the complexities of XML+Schema, you'd find that they're similar--and that many people who start building such an exchange with JSON documents end up stumbling into the same amount of complexity except in an ad hoc fashion.

It's funny, I wonder if other people have the same surreal experience: as the mass zeitgeist moved away from XML I was more or less like, okay, gotta go with it, as the driftwood goes with the currents. It's interesting how the zeitgeist functions. You can tell it's happening when people look at your functional system and are like, "Uhh, why aren't you using X...?" as though it's self-evident that you should drop everything and rewrite. Where X was, over the years (dating myself) {"Java", "PHP", "Node", "Rails", "Jquery", "REST", "Angular", "React", "Thrift", "NoSQL", "Protobuf", "Hadoop"}. Some of which I quite happily used, BTW, so the interesting part is the penetrative nature of the conversation, e.g. how the quite-functional-but-not-hot technology becomes the sick gazelle falling behind the herd, even as it is not empirically sick.

The problem I think is the extreme cargo-culting that exists with any popular technology -- half the arguments against the current flavor of the month really just begin with "no, this is not sent from god, and there do exist tradeoffs". Which is a fairly unreasonable starting point for a conversation between educated parties.

And then the ratchet turns, and the pendelum swings in reverse, and suddenly that thing sent from god was instead discovered to be sent from satan... and you're still trying to explain that tradeoffs exist

    XML can be a bear, but sometimes you need a bear.

Phrase heard at conferences in the 90's/2000's:

    "XML is like violence. If it doesn't work, you're not using enough of it."

Made me laugh. :-)

> "XML is like violence. If it doesn't work, you're not using enough of it."

That same saying would also work replacing XML for soap (but not SOAP).

Yes. Can't agree more.

This coming from someone who hates XML, avoids it at nearly all costs - there are times it's the right tool for the job. And you list that time. If you have a 1-1 API contract, don't use XML. If you have 1-N, don't use XML. But if you have N-N API contracts? Where you're talking to half a dozen other services, and they're all talking to you? XML is an excellent choice.

One of my very, very, very first jobs (1999) was at a startup that was leaning very heavily into XML for dealing with structured data. A massive amount of time was spent dealing with issues related to hand-editing XML, so I went deep into building schemas, utilities to validate XML according to the schema, and finally building an editor that was capable of providing some XSD-enabled automagic.

Good times.

> So when our integrations broke because they were sending XML payloads that failed schema validation, our conversations were so easy.

Fine, but with for example, Protobuf, you share a .proto file and so there's zero percentage chance of this happening.

No, with Protobuf, I send my counterparts a .proto file, and they 1) Don't know what the hell to do with it, not just because they don't know Protobuf but because 2) it wasn't included in the docs/dictionaries from the standards organization that set this all up in the first place.

XML files are little bit easier to debug

I hate xml so much. But I had a exam at my university a year ago where they would let me type a validation schema for a xml file in notepad++ and a checker on a computer with no internet. The fail rate for that exam was high and the course was outdated, api am glad they updated the curriculum now. I just hope to never work with xml data again.

It's a shame the original article didn't mention XPath which is a great feature when you're in the XML ecosystem. (Obviously JSON has jq, but that's not a standard - so there's no library version - and has some weirdness compared to XPath).

JSON has the equivalent JSON Pointer described in RFC 6901 (https://tools.ietf.org/html/rfc6901).

Basically, use the best tool for the job. If you're working with complex data structures from many sources, having data, validation and transformation languages (XML, XSD and XSLT) which have been co-developed and refined to basically be able to validate any third-party tool is invaluable. Sometimes extra safety is worth both the extra space and time.

JSON schema achieves the same result and is fairly standard actually.

I was asked by a company to sort out complaints which were coming from recipients of their XML. The big win was doing validation against the schema on the outgoing XML!

Also worth looking are EDN [0] (mentioned briefly under S-Expressions), Transit [1] and Fressian [2][3].

[0] https://github.com/edn-format/edn

[1] https://github.com/cognitect/transit-format

[2] https://github.com/Datomic/fressian/wiki

[3] https://www.youtube.com/watch?v=JArZqMqsaB0&ab_channel=Cloju...

Yeah, the author didn’t specify why he thought EDN wasn’t more than “a good start.” I’m curious what it’s missing!

If you follow the link, you'll see that there are no examples of data in the format and the README declares things like that it will provide a BNF when the specification ready. The context seems to imply that it is in an early implementation stage and not ready for non-enthusiast consumption. Also, last updated in 2014.

What's missing is major projects using it I think. Note how for every example he lists the big players keeping it alive.

From recollection it’s the native data transmission format for Datomic (which is driven by Cognitect/Nubank). I think there is a little bit more to the bias’ of EDN (mostly driven by language preferences in this case).

EDN has a few limitations regarding encoding custom types (tagged literals), and also performance in general. I understand both Transit and Fressian are possible evolutions addressing these points.

I would love edn if it was viable as a standalone format. In practice, programming in clojure it's plain JSON vs transit vs fressian. I tend to go with just JSON.

After half a decade of "better than JSON" mentality, fiddling with thrift, protobufs, custom serialization methods, I came to realize JSON/HTTP is usually the right tool for 90% of jobs.

Agree. Handful of cases where thrift or grpc helps with an interface or streaming, but they seem to be the edge cases.

Maybe, if you're not paying for data transmission costs.

Binary formats win for the following cases:

- payload size

- serialisation/deserialisation speed

- simplicity of client / server code (e.g. replacing a full blown HTTP server with a simple ZeroMQ one)

Sure, having payloads that are human readable is great for initial debugging/verification, but once things are put live it's just an unnecessary expense.

Definitely not for simplicity of server/client code. Using a Json serialization library basically means implementing a POJO at this point.

I was more referring to the overkill nature of needing a HTTP server rather than the ease of integration of JSON parsing code.

I remain confused. Many libraries support generic data contracts that enable both xml and json objects from the same objects.

This is true. There usually are a few cases an org would have where latency costs, de-serialization and deserialization costs outweigh the simplicity of http/json, and it usually happens in a business critical part of the company. This leads to a standardization policy where ALL projects now use grpc/thrift/etc... as an RPC standard, even if when it doesn't make sense.

"simplicity of http/json" always elicits snickering from me.

HTTP, by itself, is one of the more complex layer-7 protocols known to man. UTF-8 is not simple. Converting a decimal string representation of a floating point value into IEEE 754 is not simple (particularly compared to... memcpy). In a bizarre example of the tail wagging the dog, the ARM ISA has added an instruction specifically for parsing JSON...

There's really not a lot of value in having different serialization formats, so the "even if it doesn't make sense" requires a specific context for determining what makes sense.

I guess the ARM instruction you are thinking of is FJCVTZS, which implements the special ToInt32 operation required by JavaScript. https://stackoverflow.com/questions/50966676/why-do-arm-chip...

It is nothing to do with JSON.

You think an instruction with Javascript in the name, that converts JavaScript floating-point to signed fixed-point, rounding towards zero, isn't used when parsing JSON?

It is, but oddly enough Javascript numbers and JSON numbers are different and mutually incompatible. JS allows ±INF and the NANs, since it uses IEEE754 double-precision floating point. JSON doesn't, and technically allows arbitrary precision floating point (though implementations are allowed to define truncation behavior to a format at least as large as an IEEE754 double).

> and technically allows arbitrary precision floating point

technically, "arbitrary precision decimal"; floating point is a storage representation, to which JSON is formally agnostic, though RFC 8259 recommends that limiting to the range and precision representable with IEEE 754 doubles is a good idea for interoperability.

Yup. One of many ways that parsing JSON is anything but a simple thing is the ambiguity around how numbers, which really isn't a hard thing to get right.

With compression, payload size differences aren't that significant, and don't necessarily favour other formats (e.g. almost none of the formats have particularly compact representations of strings, and single decimal digit integer values are common enough that integer fields are not that wasteful in JSON).

I totally hear you on serialization/deserialization speed, though it is amazing how inefficient some of the "machine readible" implementations can be, and how efficient people have been able to get JSON parsing, its horrible design disadvantage in this area is lethal.

The simplicity argument is an amusing one to me. I think a lot of people build simple client/server code for JSON, but these "simple" implementations have all kinds of rough edges that prove problematic long term. Once you start doing things right, JSON proves to be far more complex.

Are you including the size of the typically useless HTTP headers when you talk about 'payload size', or just the actual payload in the HTTP request itself?

I really mean 'typically useless' in the scenario where the communication is being done internally, not when you're actually communicating over the public internet. There can be quite a bit of wastage in this scenario. I really don't care about the 'Date' header or the 'Server' header, but the HTTP spec does, so they're there.

Thanks to all that complexity in HTTP (particularly HTTP2), there is no need to pay that headers size cost in most cases, particularly ones where bandwidth tends to be a concern. Yes, naive and "simple" implementations totally waste a TON of space in headers, and even if you exploit all that complexity HTTP is still inefficient in terms of network bandwidth consumption, much as even compressed JSON is still inefficient with its use of bytes.

All I'm saying is that while JSON is horribly inefficient for some payloads, the reality is that most serialization formats tend to be efficient for specific kinds of payloads, but still have plenty of common cases they don't attempt to be efficient with. For example, machine readable formats like protobuf, thrift, avro, etc. employ some variation on varint encoding to keep small integers compactly represented, they encode strings relatively inefficiently; They encode field type & number, followed by a varint for length, followed by uncompressed UTF-8 encoded payload... and I've found that if anything it seems once you go through the compression ringer it often isn't much different in terms of space consumption vs. JSON... and sometimes it can be less efficient. Given how often fields end up being encoded as strings (even though they should be enums, ints, etc.), this has resulted in a lot of surprising cases for teams who have tried switching away from JSON to reduce payload sizes. If you cared about efficiency, you'd probably have a special encoding mechanism for short strings, and then a reasonably compact (but fast to parse) representation for longer strings, probably using something like SCSU encoding instead of UTF-8.

...and I've reached the exact opposite conclusion. You aren't unusual, but I've never been able to understand how people get there any more than they understand how I have come to my conclusion.

Provided you never need to use integers > 53 bits, dates, binary data, comments or validate what you're sending or receiving it's totally fine.

I-JSON specifies how to represent dates and binary data in strings. https://tools.ietf.org/html/rfc7493

JSON Schema http://json-schema.org/ is widely used for things like Swagger / OpenAPI.

I like JSON Schema and I wish Typescript did something more like it with it type definitions.

I liked this CBOR vs MsgPack article from a while back (which also talks about size on the wire, not just performance):

https://diziet.dreamwidth.org/6568.html

SBE (Simple Binary Encoding) also makes some interesting performance claims.

My pet peeve with both protobuf and capnp is the lack of an option type; they seem to be designed for languages with type systems that include null, rather than more modern languages like Rust or Kotlin.

Another vote for CBOR. I had actually designed my own binary JSON encoding out of frustration with BSON (which nobody should use IMO). My format was focused on compactness, but I dropped it when I found CBOR which was very similar and even slightly better in places. One notably thing about CBOR is the support for both Indefinite and Definite-Length Arrays and Maps (typically streaming generators need the former whereas parsing the latter can be more efficient).

Interestng read, didn't know most of this. Not that it matters in the end: at one point we just needed something simple which would just work and MessagePack does just that :)

Btw JSON is often a popular choice, but although this is not explicitly mentioned in the main article as a con which is an oversight imo, the actual JSON standard does not support NaN/+-Inf for floating point. That's insane, which is why most implementations have an option to allow that anyway, but not all, which can be quite the showstopper.

Protobuf has fields that may not be required (although it is advised not to use it). How is that different from Option?

In some sense JSON is unimprovable, because some properties of JSON are reached their limits.

The idea of JSON is to offer two important ways to aggregate, which are different enough that they deserve separate treatment - ordered array and associative array. They are economically represented, with [] and {}, and internal structure is also made with small costs - commas (,) between elements and colons (:) for keys in associative array. We can argue that both of those features are required.

Chosen primitives - numbers, strings, booleans and null - are also selected from what's regularly and consistently used, and the choice is supported by two decades of JSON application. Roots of this selection are in design of JavaScript, which in turn relied on common practice for basic primitives.

If JSON is considered as what I think it is, structure out of basic primitives, it's at the optimal point. Crockford's decision to avoid JSON versioning makes a good sense.

Applications of JSON - human-readable texts, performance-optimized communications - can suffer from e.g. lack of built-in comments, non-extensible "type system", non-optimal bandwidth usage (if direct ASCII or even UTF-8 is used for JSON encoding, not some other compressed approach). They however can be fixed by building on top of JSON. For example, parties can agree upon reserved keys and compression schemas, constraints in form of grammars (JSON-Schema). This flexibility stems from the fact that JSON is at the very foundation of data structures, so "better than JSON" is to an extent like "better than two's complement number representation".

I think https://github.com/rentzsch/lich gets the basic primitives right as well, and in a binary-friendly format, but it has never been pushed by any organization, so it’s essentially unknown.

Again, I think this is based on some starting assumptions about what is an ideal state, and a lot of people would argue they are flawed assumptions. It's easy to get trapped into a context when considering what the ideal solution would look like.

People forget this, but ASCII (and therefore UTF-8) already has characters in it for field/record/etc. separation. One increases complexity significantly (and introduces inefficiency) by overloading other characters with their semantics.

There are much more simple & efficient ways to demarcate ordered arrays in particular (e.g. simple length prefixed structures).

Nulls are highly problematic for any number of reasons that are widely documented, and JSON unfortunately compounds this by having two ways of representing them (explicitly and by omission).

One might quibble with the notion that a format that didn't even exist in 2000 doesn't really have two decades worth of applications working on it, but setting that aside, the design constraints of JavaScript and the challenges that have emerged from it are far from ideal. The work arounds that have emerged (the whole Number fiasco, the "standardizing" on UTF-8 while using UTF-16 when escaping, line terminators, the backslash escapign of control characters, the inconsistencies between JSON & JSONP, the binary64 "good interoperability" rule, and comments being disallowed, and of course the various JSON Schema efforts).

While I agree that an ideal serialization standard would not have versioning, and Crockford's decision to avoid JSON versioning might consequently seem to make good sense... in practice this has lead to significant inconsistencies between implementations, many of which lead to interoperability problems that could at least be somewhat mitigated by versioning. (http://seriot.ch/parsing_json.php)

"Fixing" a bad abstraction by layering on a bad abstraction is a common house of cards meme. It is highly problematic to say the least. It's how "simple" JSON rapidly evolves to be at least as complex (and at least as error prone) as ASN.1.

> Apache is the tragic junkyard of open source projects

This made me laugh more than it probably should have.

Yeah, OpenOffice first springs to mind. Is like a retirement home.

Somehow this made me feel happy that GraphQL is part of the Linux Foundation

Apache, GNU, Boost

The kiss of death. It's a graveyard of things from a previous era, pure maintenance mode. I think library authors get tired of maintaining for free then hand it off to these life support organizations. The new way for open source software is a decentralized ecosystem where people fix software they actually use, so the cream rises to the top on its own and you don't need the sponsorship of these digital museums.

Everyone loves to hate on XML just because it can be verbose and some of the WS* protocols are complicated. But as a human-readable format it’s no harder to read or parse than JSON and legacy application support is orders of magnitude better than JSON. I’m not saying we should all use it but we shouldn’t be hating it either.

Make these people debug doyne old EDI documents and all of a sudden XML doesn't seem so bad.

who doesn't love x12, especially with its loops and segments?

The issue with XML is that it was mostly pushed upon developers without any deeper understanding.

So there's a ton of utterly crap XML applications out there with no rhyme or reason, and these crap XML enterprise apps are most developer's understanding of "real world XML". Similar to how terrible C++ code being all over the place is making people hate C++ the language.

-------

XML can be used very well however: its use in Jabber / XMPP is clear and concise. XML Namespaces are needed due to the extensions of XMPP, and various programs can interact with the protocol. Specifications are clear, the ability to "stream" data is proven, different parser strategies have been implemented. Its clear that XML in this instance works, and works pretty well.

Around 2009, I worked on an application whose web interface and API were the same url -- it was served as an XML data file, with an XSLT transform to make it pretty for the browser. It was so comfortable.

I really like the separation between the semantic content and the visualization there. HTML/JS/CSS throw everything together instead. But XSLT is nothing I want to have to work with..

>crap XML enterprise apps are most developer's understanding of "real world XML"

I mean if there's enough of it, it kinda becomes "real world XML". Right or wrong human's ability to actually use the tool or tech well is often where the bar is set. Could be a great idea but if folks can't do it... won't fly.

> without any deeper understanding

I can't count how often I've seen XML that looked like this:

    <container name="containerName">
       <attribute name="attrib1" value="value1" />
       <attribute name="attrib2" value="value2" />
       <attribute name="listAttibute" value="pipe|delimited|values" />
       <attribute name="containedAttribute" value="&lt;container name=&quot;otherContainer&quot;..." />
       ...
     </container>

Which was, technically, XML (it even conformed to a schema!)... almost like "malicious compliance".

XML's verbosity massively harms its human-readability in my opinion.

Disagree--XML to me is much harder to read, if you're talking about human readable.

After 20 years of training, my meat-based XML parser is a well-tuned machine, and I prefer it over any of the other wannabes.

Markup character and namespaces aside, I think what makes JSON easier to read is that it is more explicit about whether a child object is a singleton or a list, where XML requires you to consult the schema for minOccurs and maxOccurs.

Hmm no, XML is 10x harder to read than JSON, period. It's just a ugly format with so much metadata information which are not human friendly.

I think that's the main problem of XML it's not meant to be read by humans but people thought it was.

The primary problem with XML is that it's a document markup language being used to do data serialization. Schemas were bolted on after the fact to add enough metadata for data serialization, but that's a whole lot of added complexity on top of the already complex process of parsing a lightly specced document.

So you have an already complex document parser (which you are using only a fraction of the full functionality!) with the added complexity of a schema system layered on top. This is why JSON took off so fast, it ditched the whole document part and just gave you the data.

There's a huge difference between an XML document type designed by committee and something utilitarian used by an app developer. For example:

  <items>
    <item id="1" attr1="value 1" attr2="value 2" attr3="value 3" />
    <item id="2" attr1="value 1" attr2="value 2" attr3="value 3" />
    <item id="3" attr1="value 1" attr2="value 2" attr3="value 3" />
    <item id="4" attr1="value 1" attr2="value 2" attr3="value 3" />
    <item id="5" attr1="value 1" attr2="value 2" attr3="value 3" />
  </item>

It's not something that an IBM committee would have come up with but it fits the needs of an application. If XML has an "intrinsic fault" it's related to DTDs and namespaces which allow for the creation of horrible, complex monstrosities. XML can be every bit as stripped down and to-the-point as JSON, trading the curley braces for angle brackets. JSON's best point is that it doesn't support any of this needless complexity but it's still easy enough, with enough nested objects and arrays, to make something that's effectively not readable by a human.

> It's not something that an IBM committee would have come up with but it fits the needs of an application.

Items can be order-dependent in XML. So the "id=1" thing is still superfluous. See xs:sequence. (In contrast: xs:all declares something to be order independent).

If you were doing XML for your own apps, I think xs:sequence items simplify parsing grossly.

Ex: if you have Address, maybe you always want "Street", then "Zip Code", then "State".

    <Address>
    <Street> 1234 Blah Street </Street>
    <City> Springfield </City>
    <State> Whatever </State>
    <Zip> 55555 </Zip>
    </Address>

By declaring that Street -> City -> State -> Zip must happen in that particular order, you can grossly simplify the parser's job when reading such data in. (Or writing it out).

--------

If there are multiple People at this address, you could (and probably should) just list them out in order.

    <Address>
    <Street> 1234 Blah Street </Street>
    <City> Springfield </City>
    <State> Whatever </State>
    <Zip> 55555 </Zip>

    <People>
      <p> Marge </p>
      <p> Homer </p>
      <p> Lisa </p>
      <p> Bart </p>
      <p> Maggie </p>
    </People>
    </Address>

JSON does support namespaces (and comments) you just roll them by hand (ad hoc).

It’s made for documents. It just doesn’t fit to general data structures. For instance when do you put class members in elements and when in attributes?

Yes, XML is verbose.

It's also highly redundant (so it basically can't be hand written), undesrpecified (do I put this value as an atribute, value, or sub-element?), underpowered (how does it represent numbers? Byte streams?), complicated (have you ever created a DTD by hand?), and just out badly designed.

Of course, JSON and YAML share some of those same problems, and add some different ones. But the complaints about XML aren't some superfluous opinions about how it looks.

The under-specified thing I think is funny because it’s the source of pretty much all the bad XML you see in the wild. For being an entirely tree oriented format it’s tragic flaw is that XML can’t actually represent nested dicts (or arrays for that matter) which turned out to be the data structures people actually wanted to work with.

Oh there are lots of ways to beat XML into looking like it can do it but they’re all bad, break all the actual semantics of XML and turn a nice deserialization story of a tree where each node is a complete thing into mess.

Turns out that very few people actually needed a tree of heterogeneous string k-v pairs.

https://www.w3.org/TR/xmlschema-2/#built-in-primitive-dataty...

For comparison, time formats in JSON are all over the place.

The attribute versus child conundrum in XML specification is largely absent in JSON. Ordered versus unordered children is still a puzzle to sort out but more of a special case.

In XML, there was always some motherfucker trying to stuff CSV into an attribute.

And the difference between ID as specified and ID as implemented lead to a lot of problems, reaching its zenith (or maybe nadir?) in the XML Signature spec.

How would you represent CSV in JSON?

An array of maps - each map is a row, each key a column.

Or an array of arrays with a TOC containing the keys to index mapping. Keeps the column order and it's more compact because it omits the keys in the data.

I meant the specific case of someone trying to stick three values into an attribute but the other responder covered the general case.

If you need three values use an array (or in xml, child element)

JSON5 looks pretty great to me, I'm surprised it doesn't get more attention in the write-up. Take JSON and add trailing commas, multi-line strings, and comments. Done and done!

Is that what VSCode does for the settings.json and similar settings files?

I haven't tried multiline strings but it allows comments and trailing commas

Avro never gets any love, which is a real shame given its extremely powerful schema resolution rules[0]

You don't have to worry about maintaining numeric field tags, can remove fields, can make previously required fields optional (by promoting them to unions with null), or even change their type (by promoting them to unions of <old type> and <new type>)

You need to pass your schema around with your data somehow, but there's a file format specified for that, and you can still just dump to JSON if you can't be bothered.

The schemas are specified in JSON too... which in my view makes it more robust than the JSON + JSON-Schema combo

[0] https://avro.apache.org/docs/current/spec.html#Schema+Resolu...

To weigh in on the XML vs JSON debate, this article https://engineering.instawork.com/when-xml-beats-json-ui-lay... has been posted before and shows JSON and XML's strengths with different kinds of data, i.e. lists vs trees.

Also https://concise-encoding.org/

It's both binary and text, and supports all the common data types, and doesn't require a schema or extra compilation steps.

if you are sending data to "yourself" and performance is important, defining your own binary protocol can be very effective. every time I go to grok protobuf, thrift, et al. I end up just rolling my own format because it's so much simpler and I can take all the tradeoffs that benefit what I'm doing instead of what anyone might possibly do with it. this doesn't apply in many situations (i.e. sending data to outside parties) but I've experienced a lot of success with it that seems at odds with conventional wisdom on the subject.

We only do that on embedded platforms, meaning MCUs. Anything that runs Linux is better off with CBOR or MessagePack or even ProtoBuf. Dunno about FreeRTOS et all. I've only seen libraries for JSON. ESP32 and Cortex M4 are powerful enough for serialization formats but at the expense of wasting cycles, so you're probably better off with a custom binary protocol most of the time. But if you want to sell your product to integrators use a well supported serialization format. Nobody is going to write decoders for your custom binary format today.

S-expressions are the most logical representation of data structures. “CAR”/“CDR” are not a part of the expression syntax, so this argument against them won’t matter.

I didn't read that as complaining about CAR/CDR, but rather it was making a joke: the section header is "Cons".

Ah. Too sophisticated...

S-expressions are great once you've figured out functions, atoms, complex data types, etc. That's how Amazon's Ion[1] uses them: as a nice supplement to a serialization standard.

> “CAR”/“CDR” are not a part of the expression syntax

If you were implementing them for a serialization format, you'd leave out cons cells and car / cdr.

But in late-binding LISPs, which is where they're typically used, (a . b) is part of the syntax. It's worth calling out for why you don't want it: it's badly typed since it could be a 2-tuple, or it could be a list depending on what 'b' is.

[1]: https://amzn.github.io/ion-docs/docs/spec.html#sexp

S-expressions lack a standard representation of maps/dictionaries. Even LISP is inconsistent: it has both alists (list of pairs) and plists (list of alternating keys and values).

So S-expressions suffer a similar problem to XML: there is not a clear way to represent many data structures.

I find it curious that serialization formats like JSON get hit on for not having a native date/time format. Just use a UNIX timestamp and be done with it.

https://tools.ietf.org/html/rfc7493 says to use an ISO 8601/RFC 3339 timestamp, uppercase, with timezone, with seconds. IOW, Javascript's toISOString() or equivalent.

You need to parse that. For graphs with thousands of data points just use Unix time plus a timezone or use the browser's timezone. ISO 8601 would leave your graphs churning for seconds.

If parse time is a significant concern, why are you using JSON?

Because we feed the data to Js. No need for additional overhead parsing string based timestamps. Unix epoch is just an int. We don't need pre 1970 timestamps.

Pre-1970 timestamps work too:

  Welcome to Node.js v12.18.4.
  Type ".help" for more information.
  > new Date(-10000)
  1969-12-31T23:59:50.000Z

JavaScript can parse binary serialised protobufs.

Yup with external libs. JSON.parse is builtin. We do the renderin in browser so why bother with protobuf.

Evernote's API also uses Thrift, fwiw https://dev.evernote.com/doc/reference/

There's a widely-used Erlang ETF and BERT

See: http://erlang.org/doc/apps/erts/erl_ext_dist.html

and for an overview:

https://medium.com/@niamtokik/serialization-series-do-you-sp...

Wish the author would have checked out: https://borsh.io/

I've been using it lately but i'm i'll qualified to review it to any meaningful degree. It's neat, (reportedly) fast, but has seeming zero traction and thus makes me uneasy.

(I'm currently (configurably) using it in place of bincode, for a content addressed store. for some context)

Looks pretty interesting, thanks for posting.

We'd also have a use case where we think about using bincode, this seems worth to check out/compare as a possible alternative.

JSON is popular because of JS and web browsers. It's the native serialization format of the most popular platform, so naturally it is the most baby ducked. There are also pragmatic reasons such as browsers and various interpreters (Python) being faster at JSON than alternatives (thanks to years of hand optimization).

From an aesthetic point-of-view, XML is ideal. It reads like S-expressions or Lisp but with enough features to be a robust serialization format or even represent code. Validation alone puts it ahead of JSON. Too verbose? What kind of editor are you using that can't add the closing tag automatically or show you validation errors as you go. If type checking is good, then XML is good.

The main reason XML is hated now is that it's too difficult to learn.

People that care about performance will use protobufs/capnproto, which has all the safety of XML and more. People that want something to work out of the box everywhere without too much thought will use JSON. There is no room for caring about correctness for its own sake.

> Not sure anyone really knows how XML happened. It’s basically the W3C’s fault, I think?

Doesn't instill confidence in that content.

fyi: XML was subset by W3C (the SGML "extended review board" specifically) from SGML to become the generic syntax for web vocabularies (replace HTML syntax by XTHML, and define new SVG, MathML vocabularies). Didn't work out on the web, though.

> Doesn't instill confidence in that content.

It is pretty clear by the language used that it's all written tongue in cheek.

There's also the joke that nobody wants to take the blame for inventing XML.

SVG did. XHTML is incompatible with traditionally lazy nature of webdev: broken markup happens regularly, XML would make it non-displayable, so HTML keeps accessibility of broken pages.

One key element that I think is missing in this analysis is whether the format supports cyclic data out of the box. This is an important feature for many languages since it most languages besides Rust make it very difficult to determine if the data may contain cycles or not and handling cycles can be very expensive in terms of both time and memory for serialization.

Did a disservice to Thrift here. In my last job we had I think a hundred services communicating via Thrift. It's pretty fast, type definition is easy to read, has support for union types, and supports A TON of languages. We used the Java, Node, and PHP clients/servers...

When is XML never not too verbose?

(jk)

Very interesting article and I like the style in which it was written.

Another factor to consider is whether the serialisation code is intrusive or not. Being intrusive can be an issue for legacy codebases, and IMO is not paricularly welcome regardless.

It should mention canonical s-expressions[0].

[0]: https://en.wikipedia.org/wiki/Canonical_S-expressions

The list shouldn't mix those with a schema and those without.

Anything using a schema that is agreed upon between participants isn't like JSON at all.

All of them use schemas, the only difference is if the syntax is specified in the language and can be machine verified or if you're left to figure it out on your own. Actually the latter will be machine verified too, but the results of a mismatch is a core dump or an error in your own ad-hoc schema verification system.

Yes I obviously meant 'specified schema' which the post describes as 'schema-defined' with the alternative being 'self-describing'.

Edn should be on this list

It sort of is, under "S-Expressions", but I think it should have been in its own section.

I also hate XML, but there's actually one very fair use, where I'm really struggling with all other formats: User Interface Descriptions. HTML, Apple iOS (XCode), Windows Phone (XAML), and even Android XML (virtually all android apps) copied it for that exact reason, even though they could have started from scratch.

There are a bunch more possible entries in the human-readable category, one of the more popular contenders is TOML. https://github.com/toml-lang/toml .

There are more. I don't know if java dot-properties file is worth including.

The speaks to the context of serialization formats, which TOML isn't.

Hm. Would you call YAML a "serialization format" but not TOML? I'm not sure where to draw the lines of what is a "serialization format" (you can serialize to both yaml and toml, no?) but YAML and TOML feel to me like they are in the same space, human-readable representations of hash/array json-like data, ie serializations; not to you?

Arrow is rapidly become a data interchange format for Pandas and GPU compute, didn't see that on there

Good point, though I think Arrow is optimized for larger amounts of data. Not sure how it would hold up for small messages. Still, lots of people use JSON, etc. to store large amounts of data, so Arrow (or maybe more accurately, Parquet) is a fair comparison here.

I miss the security category.

All of the mentioned formats are now insecure by default, because they deserialize objects. (Just msgpack not, which has other problems). Even JSON, the most secure format originally became insecure in its 1st update. All the binary JSON variants, like BSON went bonkers.

This post seems to be conflating various things such as configuration files, client side message configuration, and wire format.

I am not sure what the takeaway is from it aside from the author's personal opinions on various technologies

I would have expected some mention to https://jsonnet.org/ too, which after a bit of practice can be brilliant.

I'm a huge fan of jsonnet, but this article is specifically about data serialization formats, not configuration languages. It could've been a worthwhile mention though.

You are right but I mean it since jsonnet has also function as data type, which is not a trivial matter.

Anything that lists BSON should include the fix: JSONB

https://news.ycombinator.com/item?id=7457645

In my experience with BSON it's ridiculously wasteful compared to CBOR. I haven't compared with JSONB, but it seems unlike that it improves on CBOR.

JSONB is a private internal data representation in PostgreSQL, designed to be faster to index than textual JSON. As I understand it, it isn’t guaranteed to be stable, it isn’t exposed outside PostgreSQL, nor designed for data interchange.

Just to be a broken record:

JavaScript is better than JSON because it has loops, conditionals, comments, modules, functional programs, typing, etc etc

Which opens a lot of opportunities for potential attackers embedding malware into data and executing it remotely.

msgpack users:

- Fluentd & Fluent Bit

- Microsoft SignalR https://docs.microsoft.com/en-us/aspnet/core/signalr/message...

lemme piggyback ./'__')\ https://stackoverflow.com/questions/57470117/name-for-junipe...

The only problem with json is that you can't do comments with the default implementation.

No trailing commas and multi-line strings are my other two big complaints. The ability to leave keys unquoted if they don't have any spaces would also be nice.

The best part is that these are a superset of JSON and could easily be added if there would just be more buy-in (lots of unofficial support already exists).

The author does mention JSON5 which implements all of those features.

better than JSON is JSOFF.

There is no such thing and this is blasphemous - shut it down.

It would be nice to be able to represent 64-bit ints--and have comments.

As for comments, I think JSON is only "human readable" by accident. It was never really meant to be read by humans, and was created in a way that implicitly discouraged people from trying to write or edit it by hand. Many parsers are picky about whitespace to discourage people from trying to create config files in JSON.

"comment-subject-author":"this is a comment, we dont need out of band communication, thats a bug, not a feature"

Ah, so now you're encoding out-of-band communication into your data? How is that not worse? This also completely breaks if your data is validated against a schema, or if your comment gets parsed as a data entry. For example, in Javascript projects you define your dependencies in package.json. I can't add a comment to explain why I had to add a strange dependency, because npm will interpret it as a dependency itself and then break.

Reply on news.ycombinator.com