Hacker News Reader

I think the reason JSON caught on outside of the javascript community is because it provided a succinct way to represent tree structures of common types (numbers, strings, dictionaries, and lists), using a syntax familiar to anyone who has used C, Perl, Python, Ruby, or Java.

S-expressions might have caught on for this purpose, but they lacked a killer app (jquery, and later rails). Like JSON, they are easy to parse and easy to generate, but being more loosely-specified than JSON, it's less clear how to map a given S-expression to a native type. As a glue format, JSON feels just right, even if it's a bit picky sometimes (e.g. trailing commas).

For anyone who wants extra features like comments, YAML is the oldest popular format I am aware of that is a superset of JSON. For any new format to succeed, IMO it needs to sufficiently distinguish itself from both YAML and JSON, and not just support a feature set that happens to lie somewhere between the two.

I think the reason JSON caught on outside the Javascript community is because it caught on so massively inside the Javascript community, at a time when there was a lot more crossover between people working on the javascript layer of web apps and people working on the server side (back then, almost always in a different language).

I was doing mostly Perl and Javascript when it caught on, and to this day I have very mixed feelings: it's wasteful of space but still doesn't allow comments; its type system is basically a technical-debt generator; and for all that people still get it wrong pretty often. On the other hand, it's more or less human readable for simple data structures and it's more or less everywhere.

My hunch is that for a new format to take off it wouldn't so much need to not fall between YAML and JSON: it would need to be the default format of something with such super exponential growth that even us oldies would have to use it.

Suppose simple > complex wins sometimes. JSON was a serialization format. XML was a markup language being used for serialization... and created a lot of extra work and bugs for developers. Markup languages can be incredibly useful (evidence: the web, SVG), but often bring immense complexity to simple problems.

> I think the reason JSON caught on outside of the javascript community is because it provided a succinct way to represent tree structures of common types (numbers, strings, dictionaries, and lists), using a syntax familiar to anyone who has used C, Perl, Python, Ruby, or Java.

I believe it's hard to explain JSON's popularity and wide adoption without talking about javascript. With javascript, JSON was right from the start an ‘eval’ away from being parsed. The barrier to entry to adopt it simply was never there. Once you start to expose JSON APIs to clients, other servers also start to need to consume data from those servers. Rinse and repeat until you reach mass adoption.

Oh yeah the "good old days" where one ran eval on the response to and XmlHTTPRequest.

> Like JSON, they are easy to parse and easy to generate, but being more loosely-specified than JSON, ...

They would not have been loosely-specified if they were specified, like JSON was :-) I mean this is taking things a bit backward. When tools use a data format based on S-exprs, they define more clearly what is or isn't valid (OCaml Dune, Guix, etc.)

I think JSON demonstrates that most of the time you don't need it, and doing that sort of thing badly is worse than not doing it at all.

E.g. XML has all that, and yet the average use of xml is much more fragile thsn the average use of json. Similarly asn.1 has all that, but does anyone actually like the fact the x509 certs use it? (To be clear, json would definitely not be appropriate for that case either).

There are both JSON [1] and XML [2] encoding rules specified for ASN.1... I have never seen it used in the wild, but realising it's a thing is horrifying enough.

[1] https://www.itu.int/rec/T-REC-X.697/en

[2] https://www.itu.int/rec/T-REC-X.693/en

I've used BER encoded ASN.1 at work quite a bit, and the reason you won't see it much are pretty much:

- HORRIBLE schema format.

- Very few FOSS libraries, most of which are terrible. I used Lev Walkins asn1c[0], which I patched slightly and coupled with some semi-generic code that walks the generated data structures to emit JSON (for debugging purposes).

[0] https://github.com/vlm/asn1c

Erlang's ASN.1 is awesome! Have a look at http://erlang.org/doc/apps/asn1/asn1.pdf! It could even be used as an introductory reading to ASN.1, to be honest.

I've implemented (a subset of) ASN.1 for an SNMP server I wrote for an embedded monitoring system, so I know the pain, but it was one thing to implement it for a compact binary format. Inflicting it on a text-based format just seems like it'd multiply the pain...

The resulting JSON was pretty clean, no worse than you'd get out of Flatbuffers or Protocol Buffers JSON encoding.

It was very handy to be able to use jq on the data.

What programming languages and platforms will your SDK be compatible with, will you also be around to port the SDK to new platforms in twenty or thirty years? Those are good reasons to agree on a standardized data format instead of a specific implementation. Pretty much every language on the planet has JSON parser libraries, and if not, it's fairly trivial to write your own (or convert the JSON data to something else).

> Pretty much every language on the planet has JSON parser libraries.

Yes, but they all implement the JSON RFC slightly differently, or implement it in a totally non-compliant way.

Then it's totally fine to pick another commonly used data exchange format which is better defined, that's still much better than hiding the data format behind an SDK.

However, we're coming from XML, which was better defined, and there are also binary "quasi standard" formats like protobuf, so those problems had already been solved.

Yet still "the world" has moved to JSON, which seems to indicate that those problems probably were not all that important for most use cases.

As long as you set a limit on integer sizes (in most cases an arbitrary low limit should be fine) and don't expect any precision on float (each conversion between text and binary representation there is problematic) and don't use extensions (like comments or trailing commas) you should be quite fine with most JSON libraries. This won't cover all things, but a lot.

It's not a great standard if every program/library using it must explicitly state its limits for such basic things as number values.

It's very easy to do JSON wrong, to not even be aware of doing it wrong, and do it wrong in a way that limits its interoperability (see: the default behaviour of the json library in Python). This is not theoretical either, just google for 'json nan github' to see the hundreds of production programs accidentally emitting JSON that cannot be ingested by RFC-compliant parsers.

I agree that as a standard it isn't great. However the fact that it is so successful shows that the minimalistic design has benefits for adoption.

And regarding integer ranges: Any user has restrictions on top of the generic format. Some fields have to be present for the application to work, some fields have to be a string, others an array. Some integer has to be between 0 and 100, some array has to have 5 elements. Given that 99% of integers in practice fit in a signed int32 there is little problem (97% are probably 0, 1% are 1, 0.5% are -1 and only the rest other values ...) If you are on the edge you have to know and work-around ...

> Any user has restrictions on top of the generic format.

The problem is that you are not guaranteed to know, as a JSON library user, that the library has not mangled the numbers it received on the wire prior to your application receiving it - so you don't know if you having .a set to 42 is the result of 42 being sent over the wire, or an implementation dropping bits that are outside the range of you library's support. The RFC does not mandate what a JSON implementation should do in case of numbers outside its' support.

> Given that 99% of integers in practice fit in a signed int32 there is little problem.

Until you hit that 1%, and you find this out the hard way, and you have no way of solving it because you don't control the emitting side. Again, this has happened in practice to me, when a Python library was emitting large numbers as numbers (as the RFC permits), while the receiving side silently casted to 32-bit floats, losing data (as the RFC permits). Both sides are right per the spec. Technically, everything worked as per spec. Practically, the product was broken.

99% of the time you might be okay. But the 1% of edge cases makes it that you can never rely on JSON, which makes it a bad interchange format if you care about reliability and safety. You _can_ make it work if you severly limit yourself and are deeply aware of all the possible issues that using JSON has (and there's a lot more). Or you can just pick some other standard that solves these basic things for you (eg. Protobuf).

Any half serious library will be able to handle 42. If I leave the signed int32 range as a designer of the API I should be aware of the problem and either clearly document what I do or find some alternative. (Like making it a string with quotes)

Random side note: ECMAScript has no integer type, but only Number, which is a float, thus when dealing with such numbers in a JavaScript frontend you are in Problem Land anyways ... which again shows that boundaries have to be thought of, independently from the specification of the data exchange layer.

> the receiving side silently casted to 32-bit floats, losing data (as the RFC permits)

Had to look that up. Was under the impression the rfc only specified the syntax. But here it is

> This specification allows implementations to set limits on the range and precision of numbers accepted.

I don’t think the implication is that it must be done by silently loosing precision though. A loud error would fit that description just fine. So in this case I would blame the implementation not the spec.

I really want something that’s strongly typed but doesn’t require code generation like protobufs do. Yaml doesn’t do it for me. The closest I can get is putting the type guarantees in the database and using GraphQL.

I made https://concise-encoding.org/ to deal with this:

- strongly typed

- ad-hoc or schema (your choice)

- no code generation step

- edit in text, send in binary

Why would someone choose this rather than msgpack or CBOR or protobuf or any of the other existing things in that space?

Because there isn't anything else in this space that:

- supports ad-hoc data structures or schemas per your preference

- supports all common types natively (doesn't require special string encoding like base64 or such nonsense)

- supports comments, metadata, references (for recursive/cyclical data), custom types

- doesn't require an extra compilation step or special definition files

- Has parallel binary and textual forms so that you're not wasting CPU and bandwidth serializing/deserializing text. Everything stays in binary except in the rare cases where humans want to look or edit.

That looks pretty good, actually.

I use JSON Schema to validate JSON documents.

https://json-schema.org/

Imho, statically typed languages are the ones that benefit most from schema. The current schema version is 12 but the implementations for Go, Rust, C++ and Java are all listed as draft 7. None of them support codegen either, just validation, so not exactly compelling.

> The current schema version is 12 but the implementations for Go, Rust, C++ and Java are all listed as draft 7

It's actually 2020-12, which is two versions after Draft 7 (they shifted from Draft n to YYYY-MM after Draft 7, and since then have had 2019-09 and 2020-12.)

And that's true of most languages, though there is some 2019-09 support. (It really doesn't help that there is also OpenAPI which baked in a variant—“extended subset”—of Draft 5 JSON Schema.)

OpenAPI 3.1 which was released recently, uses JSONSchema 2020-12 as the primary schema format. As a result, we can expect further consolidation of tooling, etc in the community.

I benefit greatly from schema validation in Ruby, ensuring that ingress-processing code does not receive e.g a String or Hash instead of an Array which would have things blow up way after the ingress edge when a call to an Array method fails, or worse, produce silently broken behaviour that may or may not blow up even farther down the road because both String and Hash respond to e.g #[](Integer).

Yeah but Ruby is a dynamically typed language. There's not much benefit to codegen since nothing is checked at compile time anyway.

I found code generation to be useful in Ruby with protobuf. This:

https://github.com/lloeki/ruby-skyjam/blob/master/defs/skyja...

gives that:

https://github.com/lloeki/ruby-skyjam/blob/master/lib/skyjam...

I would certainly enjoy having a DSL to write descriptive code to validate using JSON schema, but it would be even better if the Ruby definitions could be generated and persisted in Ruby files using that DSL.

Also, storing things in basic hash/array types works, but having dedicated types is useful, so that one can ensure not shoving one kind of hash in place of another unrelated kind of hash.

As for types themselves in general, there's RBS and Sorbet. One could have type definition generation as well for even deeper static and runtime checks.

Do you really want generated code to manipulate JSON? I'm not sure there is a demand for that.

Manipulating anything dynamic in a statically typed language is generally tedious and not type safe, so yes.

In any language I eventually need to validate. Whether I do it early, using a validator, or during processing the data at later is a choice depending on the problem.

Existence of a schema definition file and checking responses against is signalling that I can trust an API vendor to be at least aware of the requirements for clients. (Whether they randomly change the schema definition or ignore it is a second question, but at least somebody once thought about formalising and it's not an complete adhoc dump of today's internal data representation)

And there's Amazon Ion too - https://amzn.github.io/ion-docs/

"Amazon Ion is a richly-typed, self-describing, hierarchical data serialization format offering interchangeable binary and text representations. The text format (a superset of JSON) is easy to read and author, supporting rapid prototyping. The binary representation is efficient to store, transmit, and skip-scan parse. The rich type system provides unambiguous semantics for long-term preservation of data which can survive multiple generations of software evolution."

Strongly typed + no code generation is obviously doable in any dynamically typed language.

Apache Avro has support for parsing and utilizing schemas at runtime, even in C++.

For Apache Thrift you have things like thriftpy: https://thriftpy.readthedocs.io/en/latest/

I'm not aware of a type-safe mechanism for Flatbuffers or Protocol Buffers.

There are protobuf libraries without code generation, for instance: https://github.com/cloudwu/pbc (you lose the connection to the language's type system though).

Json does have schemas

Sure, they are just not that widely used. As in, I've not really encountered much json with schema in the wild in my career. I know it exists (in several forms). Support for schemas in mainstream parsers is pretty much non existent making their use very optional as you pretty much have to jump through hoops to do anything with them. Most people seem to opt not to do that.

I never got anything out of xml validation either. It just lead to this ridiculously verbose garbage xml that was even less readable with all the namespace, namespace declarations, etc. Very tedious to write manually as well. Also lots of weirdness doing e.g. xpath or xsl transformations against that. The whole SOAP / web services bubble eventually imploded when people figured that you could send tiny json objects instead via REST.

I have many issues. People sending invalid json to my APIs is not one of them. You get a bad request if you do. End of story. Don't don that if you don't like bad requests. It's not a problem that justifies a lot of over engineering.

Json, yaml, toml, hocon, etc. are basically all just variants of attempting to send human editable blobs of information. Fine for apis where all of the requests are created by programs. Unfortunately people also abuse them for things like DSLs that are authored by humans. The real problem there is not using a strongly and statically typed language that simply does not allow illegal things. A schema is just a stop gap solution when you don't have that.

Kotlin is actually great for creating proper DSLs. You get IDE auto-completion and red squiggly lines when you do it wrong. I think Rust also has some nice syntactical constructs for creating DSLs. Typescript might also emerge as a language that is very suitable for that (with maybe a few more features borrowed from Kotlin). Using a non compiled language with weak typing kind of defeats the purpose. Hence the endless ruby but not quite ruby like DSLs for things like puppet.

The JSON schema ecosystem is a mess. json-schema.org is one thing, openapi is another (it uses an 'extended subset' of json-schema.org, which is another way of saying 'incompatible'), there’s also typeschema, and a bunch of other pet projects.

There is no official JSON schema RFC, there is no official linking between json-schema.org and the JSON RFC. There is no push to make every language that supports JSON to also support a JSON schema system.

Finally, there isn't even a guarantee that all human-readable JSON document specifications can be expressed within a JSON schema in any sensible way.

OpenAPI 3.1 which was released recently, uses JSONSchema 2020-12 as the primary schema format. The incompatibilities has been completely resolved as of today.

XML schema is joy ?

Yes it does but as someone else mentioned, it can be really challenging to work with. Its difficult to visualize the structure of the document. Its difficult to program against. This was one of the problems which I wanted to solve when I created unify-jdocs. Agreed that it may not do all the things that JSON schema does but it does all those which are required 90% of the time. Along with other things like not having to generate any model classes (code generation) and being able to read and write any path in a single line of code. And being able to merge JSON documents into another. Strong typing has its pros and cons. In my opinion, for complex JSON documents, going the strongly typed way makes making change difficult in the long run. Think of JSON documents the size of 3000 JSON paths going 10 levels deep etc. You can read more about this Java library at https://github.com/americanexpress/unify-jdocs.