The Last JSON Spec | Hacker News Reader

The Last JSON Spec

tbray.org

88 pointsrobin_reala71 comments

I'd be happy about a JSON spec that allows for comments...

I'd rather have trailing commas, so every line can have the same format. Other people choose JSON for their config file format, and I don't have a say in it. I just have to do extra editing each time I reorder the lines or add a new one, because too bad. And I have to manually edit JSON when I'm writing code to handle other people's JSON APIs to test my code.

I'll probably never not have to manually edit JSON, so JSON's insistence that using the same format for your final line as for all the others won't ever be allowed will remain a never ending, unnecessary nuisance in the spirit of JavaScript itself: the "too bad, too late now" school of design. Except that in this case, even JavaScript corrected that particular nuisance and allows trailing commas.

All because IE6 wasn't supporting trailing commas at the time when JSON was written. Shortly after that we discovered that it was a bad idea to eval() JSON so the whole reason for browser compatibility became moot.

But who knows, maybe that is what was needed at the time to make JSON popular.

If you want a JSON-ish config format, I'd recommend to check out HOCON [1], basically a superset of JSON that fixes all he things that make it annoying for config files.

[1] https://github.com/lightbend/config

If we're proposing supersets of json, why wouldn't one just use yaml?

If the config is in the form of:

    {
        "file": "/etc/foo",
        "daemon": true
    }

Couldn't you add comments like so, all within the existing spec?

    {
        "file": "/etc/foo",

        "_daemon_comment": "default is false",
        "daemon": true
    }

The problem is that's not really a comment, it's a string which needs to be evaluated as data, wasting resources while never being used. In any sane language, comments are simply ignored by the parser or compiler.

Because Douglas Crockford didn't want people abusing comments in JSON by turning them into parsing directives, JSON doesn't allow using comments at all, even correctly, so everyone is left abusing JSON itself with workarounds that are objectively worse than just having comments.

Additionally, working against a json schema with additional properties false would cause a validation failure.

if you're using json as a config file you can choose a json parser that supports comments. there are a number of them

At that point you're not writing JSON though. Once you start bolting on non-standard bells and whistles, why not recognize that JSON was never meant to be used for config files, and switch to something that was, like TOML [0]?

0: https://npf.io/2014/08/intro-to-toml/

The reason why comments aren't allowed anymore in the spec:

https://plus.google.com/+DouglasCrockfordEsq/posts/RK8qyGVaG...

And a conversation about it:

https://news.ycombinator.com/item?id=3912149

>The reason why comments aren't allowed anymore in the spec

Is BS.

Exactly.

    {
        "__directive__": "phpstyle_stringly_typed",
        "boolean": "true"
    }

Nope, because that is still completely valid JSON. Sure it's kind of a shitty choice of structure, but every JSON parser on the planet will be able to read it successfully.

If you choose to layer some extra logic/rules on top of the actual parsing, that's fine - that would be entirely within the intended purpose of JSON.

Comments are FAR more likely to be used more like "preprocessor directives" to influence actual parsing rules, changing the syntax. Whereas your example conforms perfectly to the syntax, leaving parsing as-is and ensuring the logic is handled at a higher level.

But if you're already changing the syntax, why do you worry about JSON interop at all? Just invent your own language.

If you do worry about JSON interop, then JSON-with-parsing-directives-disguised-as-comments would be valid JSON as well - just treat the directives as comments and ignore them.

The resulting tree would probably not reflect the data that the author ment to encode (because you ignored the directives) - but neither would the tree from the example above.

It seems a bit like a baby-and-bathwater situation, honestly. It seems feasible to put in the standard something like that comments must be stripped out in the first pass and may never influence the interpretation of the non-commented JSON. A parser-implementation might choose to ignore that, but then it wouldn't be a standards-compliant JSON parser, in the same way it wouldn't be if it brazenly ignored any of the other sections of the standard.

You could, but it probably would be ignored. He was completely justified in making his decision, since the exact problem was worried about is rife in XML.

What's that supposed to prove?

First, I didn't say it can't happen. Just that it's a BS concern to be worried about in the spec.

For one, this thing he is worried can already happen without comments -- just needs a second pass after the parser (or of the parser).

Second, this can already happen without comments and a single pass parsing -- just needs out of band parsing instructions.

Those needing special parsing pragmas can get them anyway.

And the same problem didn't emerge in lots of other formats that allow comments.

No that is fine because there is no hidden information that standard JSON tools and libraries cannot access. It's including information in comments - which are meant to be ignored - that is the problem.

Seems like a good reason to me.

What about Zish? It's like JSON but with comments, a timestamp type, a decimal type and a bytes type:

https://github.com/tlocke/zish

(as the author, I'm biased)

My unsolicited comments. I really want to see a good JSON-done-right out there and your spec is maybe 90% there.

I'd rather only /* */ comments were allowed. By allowing //, you've now got two varieties of white-space the parser needs to be aware of. Because of this, I can't take multiple lines of ZISH and join them together into a single line. (Something both JSON and XML can do.)

What is a "line"? CR, LF, CRLF, LFCR, FF, VT, U+0085, U+2028, U+2029?

Am I required to treat ZISH files as "binary"? Could I convert all of those end-of-line bytes into my OS's preferred form without losing data?

The notation of ending a line with a \ also prevents joining multiple lines into a single line. Maybe formalize "string"+"string" as an explicit string joining notation? (Or if not +, some other character to notate string concatenation.)

Given you have the decimal type, is float necessary? Values with a "e" are still expressed with decimal digits.

Is there a limit to numeric types? What if my ZISH file contained a several hundred 9s in a row?

Sets seems unnecessary given you already have lists. You can take a list and treat it as a set after parsing with no loss of data.

I very much like the addition of timestamps and binary without needing a string. This is my big pet peeve of JSON.

Your examples include uses of null and true, but your spec doesn't define those values. (Or false for that matter.)

The description of map doesn't specify if the order of keys is significant. Is {"a":"b", "c":"d"} the same as {"c":"d", "a":"b"} ?

This is quite brilliant, thanks billpg. I'm also amazingly grateful that someone's taken an interest in Zish :-) These questions will all require some thought to properly understand and respond to, with no-doubt some changes to the spec. So I've opened an issue and put the comment at:

https://github.com/tlocke/zish/issues/2

Sets could be reordered to improve compression. Not a big benefit but it would depend on your data.

Personally I've never found that to be an issue when using JSON as a data interchange format.

The only place I'd like to have comments is in configuration files and in that case there are less verbose formats that do allow for comments.

I never really understood json for config files. It's not that readable. It's great for data exchange and relatively human-readable representation, but if you ask people to actually write and maintain it, why not yaml / toml / ...?

YAML suffers from significant whitespace, which makes me for one refuse to ever consider it. Personally I consider YAML less human writable than JSON for that reason.

TOML is tolerable, but all my preferred languages already have JSON parsers I'm familiar with, but while they may have TOML parsers, I'd have to go looking for onne.

HOCON[1] is a pretty reasonable compromise, IMO. I've really only seen it used by Scala and Scala-y Java, which makes sense considering its lineage.

[1] https://en.wikipedia.org/wiki/HOCON

JSON is readable enough. The format is simple - no inline functions, no circular references, just maps or arrays of key-value pairs. This makes it well suited for describing what a config file should, which is an initial application state, without much needless abstraction.

Config isn't meant to be code, and config files shouldn't be updated often, so a config file format doesn't need a great deal of expressiveness. Dealing with quoting keys and commas in JSON isn't more or less of a hassle than dealing with significant whitespace or whatever pecadillos those other formats have.

Although personally I really like Lua tables for config because they have an even simpler structure than JSON and you can add comments. I have to resist the tempation to turn Lua config files into applications, though.

It's also got issues though. Multiline text, undefined number ranges / interpretation for example.

yaml has different "standards" depending which parser you ask and it is dead strict on whitespace (some parsers dont like tabs at all, and heaven forbid mixing spaces and tabs in one file).

JSON doesnt care as long as it's valid, and there are parsers for every language and environment.

There's only one official source for specification: http://www.yaml.org/spec/1.2/spec.html

If the parsers do something different, that's an implementation issue, which inevitably affects all formats. This affects json as well: http://seriot.ch/parsing_json.php

Deserialisation is usually a one liner in your languages built in stdlib. Yaml/toml are not built in, though you may be able to try toml with your ini parser.

For the rest, it's just an equivalent of "import yaml" before you use a one-liner as usual. Is it really a big problem?

> Personally I've never found that to be an issue when using JSON as a data interchange format.

I had to diff on-wire JSON payloads to find a discrepancy, and boy was it a pain, whereas having those trailing commas would have reduced the diff to a quite direct and obvious pointer to the faulty change.

It doesn't solve your problem in the past, but if you run into this again in the future i've found it's really effective to stream your payloads through jq and have it the objects to compare by key first.

Convert to YAML and make unidiff from there?

You probably don't want comments sent on the wire, but if you like JSON as a config format but want it to me more human-tolerant, you can try HJSON:

https://hjson.org/

I like it more than any other config format, really.

this has many of the misfeatures of yaml, like ambiguity of values.

    {
      a: 5
    }

is that "5" or 5? with JSON it's unambiguous.

Goes to show that there is no perfect format for all use cases. The lack of comments is a good thing for interoperability in machine-to-machine communication. But it a bad thing for hand-written configuration files.

Why are no comments good for interoperability? Numerous formats can contain comments and I've never seen someone having a problem parsing them out, especially since they usually cannot be nested and/or allow you to skip until the next newline.

I personally dislike it the most when you want to provide a commented example of your data format, vanilla JSON won't allow you to do that.

You should have a look at JSON5 http://json5.org/

The fact that it has no support for Unicode at all -- neither literal nor escaped -- is just tragic.

Hm? The spec[1] has reference to Unicode all over the place.

[1]: https://json5.github.io/json5-spec/

If it's supported, then they should update json5.org, which says "TODO: Unicode characters and escape sequences aren’t yet supported in this implementation."

{ "_comments": [ "Hello." ] }

> "On the other hand, we still lack a decent high-quality JSON Schema language."

This turns data integration with JSON into a total nightmare. And to make matters worse, Swagger is a total shitshow. Combine the different flavors of JSON schema with OpenAPI/Swagger and you have are left with a regression from SOAP/XML that approaches the same complexity with with fewer features and no standards.

Even mf COBOL has a standard schema definition for data. For new projects I'm still cranking out XSD definitions and doing transformations to JSON if I have to. Even if they come out with a decent schema format I would probably skip it and jump straight to gRPC, Thrift, or some other IDL and just continue to treat JSON as an "also supports".

"we still lack a decent high-quality JSON Schema language"

Yes. Why?

The JSON Schema spec was abandoned by its original authors, and was picked up by a series of other people with varying levels of standards experience (myself included).

Personally, I had some promising support from my then-employer (including an IETF connection), but then it kind of petered out, and I was buried under other stuff and had to pass it on. The team who picked it up since them seem pretty on-the-ball as far as I can see, so I would still recommend it as a useful project to contribute to.

It is supported by some tools - last I heard, Visual Studio could use JSON Schema for autocomplete and validation, as well as a couple of databases. I used to get a few million NPM downloads a month on my validator, so someone's clearly using it.

As far as I know, Snowplow is one of the largest active users of JSON Schema, and we maintain one of the larger repositories of JSON Schemas on the internet, https://github.com/snowplow/iglu-central

After something of a hiatus, we are re-starting active work on our schema registry technology (https://github.com/snowplow/iglu). We'd love to be involved in new design work on the JSON Schema spec. How do we get the old (or new!) band back together?

There's also the excellent dtsgenerator which converts json schemas to typescript type definitions.

Unfortunately, power complexes. When more power is needed than json has (e.g. a schema language), there is xml and the xml ecosystem (e.g. xs), with all the tools, features and standards you could possibly want... actually, more. Thus, it is the complexity of xml that keeps json simple.

It is the same principle as keeping flies off the dinner table by dumping a pile of manure outside the front door.

No we don't. Typescript is _great_ for this.

  type Book = {
    title: string;
    author: Author;
    isbn?: string;
  }

  //etc

It's easier to read and write by humans than JSON-Schema, a large amount of programmers are already familiar with the syntax, and it's pretty self-explanatory to most programmers who are not.

Sure, it doesn't have constraints like "maximum string length" and "must start with a capital" etc, but I'd wager that those don't really belong in a schema language anyway. The structure and the types is what matters. Once you got that covered, doing some validation on the content of the values is easy. And you can use a Real Programming Language for it, instead of something half assed.

> doing some validation on the content of the values is easy

How can you validate a JSON string to see if it matches a TypeScript declaration?

FYI: I've put together an Awesome JSON (What's Next?) page [1] collecting all JSON next flavors incl. JSON5, HJSON, HanSON, SON, CSON, USON, JSONX/JSON11 and many more. Yes, comments (in JSON 1.1), please :-) Cheers. [1] https://github.com/jsonii/awesome-json-next

FYI: For the Microservices Austria meetup I've put together a talk titled "Meet Jason - The Universal Data Exchange Format - What's Next?", find the notes (transcript) @ https://github.com/geraldb/talks/blob/master/jsonnext.md Yes, comments (in JSON 1.1), please :-). Cheers.

Fine that they didn't another insecurity as in the latest updates. JSON is still the only secure transportation protocol if you don't use the RFC 9159 simplification, http://search.cpan.org/~mlehmann/JSON-XS-3.02/XS.pm#SECURITY...

From my reading, literally the only change between RFC 7159 and RFC 8259 is that the latter mandates an UTF-8 encoding. Checking Appendix A "Changes from RFC 7159" mostly confirms it:

* Section 1.2 has been updated to reflect the removal of a JSON specification from ECMA-262, to make ECMA-404 a normative reference, and to explain the particular meaning of "normative".

* Section 1.3 has been updated to reflect errata filed against RFC 7159, not RFC 4627.

* Section 8.1 was changed to require the use of UTF-8 when transmitted over a network.

* Section 12 has been updated to increase the precision of the description of the security risk that follows from using the ECMAScript "eval()" function.

* Section 14.1 has been updated to include ECMA-404 as a normative reference.

* Section 14.2 has been updated to remove ECMA-404, update the version of ECMA-262, and refresh the errata list.

* RFC 8259 requires UTF-8 encoding (for network transmissions)

* ECMA-404 has been upgraded from Informative (14.2) to Normative (14.1)

* Wording has been improved or fixed in some cases

You need to read between the lines and know the history.

Restricting to UTF-8 and disallowing the previous allowed UT-16 and UTF-32 BOM's (which my JSON library is the only one which did support that), is the most prominent change, yes.

What is troubling to me is that they now prefer relaxed RFC 7159 over strict RFC 4627. The RFC 7159 relaxed mode was a troubled update which harms security, and it should not be turned on by default.

And whilst they support now the insecure relaxed variant, they didn't clarify the still problematic points in the spec. MUST disallow duplicate keys, or MAY? It's not specified, or specified twice contradicting itself. Same for broken UTF-8 encodings. Error or warn or silence?

It's still much better than the YAML spec though. Minor nitpicks.

What do you mean by "only secure transportation protocol"? There's nothing inherently insecure in XML, yaml, asn1, bson, cbor, msgpack, out whatever other popular format. Implementations may fail, but that's a different problem.

* YAML is pretty much inherently insecure, custom tags are part of the YAML spec and are interpreted by default by most implementations, the YAML 1.2 spec only recommends defaulting to the "core" schemaor a lesser one and (AFAIK) at no point even suggests disabling explicit tags.

* XML is most definitely inherently insecure if you don't disable or strictly limit standard features (entity definition/expansion, external entity definition, DTD retrieval).

I don't know about the others, I do know that ASN.1 has a long and storied history of vulnerable implementations (and it's unclear whether there's ever been a secure one) which tells me that even if the format is not inherently insecure, in practical terms it is.

ASN.1's main issue is that it was created in the 80's, so most implementations were programmed before folks understood software security.

Also, it has had almost 40 years of additions, so it is now a large spec.

ASN.1's main issue is that BER/DER/CER are awful, and they are awful because they are TLV (tag, length, value) encoding rules. ASN.1 itself is a bit clunky, but has everything you'll ever need.

Too many people have reinvented ASN.1 badly. Protocol Buffers, I'm looking at you!

Both cases make good, secure implementations harder. They don't make the protocol itself insecure since you can make implementations that avoid these issues. There's nothing in the protocol itself that forces you into a situation of "choose between valid or secure".

> Both cases make good, secure implementations harder.

You literally can't secure XML without refusing to enable (or implement) standard features.

Are external entities required to be processed? I know multiple implementations require you to opt-in to download anything, so I don't think so. But happy to be corrected.

Of those you cited only msgpack is besides JSON inherently secure. All others are insecure by default. Read the link I cited.

BTW The latest 2 JSON specs also are now insecure by default, in its relaxed RFC 7159 mode, allowing values such as objects, arrays, strings, numbers, "null", "true", and "false".

Wait: default msgpack is also insecure, as you can trivially add packets to the front or end, overriding it. It also accepts cutoff or corrupted packets. Only my patched msgpack version was better, but only for prepending packets, not for added packets after. https://github.com/msgpack/msgpack/pull/114

> JSON is still the only secure transportation protocol

Ummm, canonical s-expressions[0] and ASN.1's various encoding rules[1] beg to differ.

The one thing JSON has that canonical s-expressions don't is maps/hash tables/dicts, but after working with both for quite awhile I've come to the conclusion that maps in the core transport language are actually a mistake.

[0] http://people.csail.mit.edu/rivest/Sexp.txt

[1] https://en.wikipedia.org/wiki/Abstract_Syntax_Notation_One#O...

S-expr are powerful, but too powerful. They do allow object deserialization, which is the insecure part.

> They do allow object deserialization, which is the insecure part.

Only in the exact same sense that JSON allows insecure object deserialisation. Yes, if you process a JSON object or S-expression, your processor can do all sorts of insecure things. But a non-buggy S-expression parser can't do anything wrong with (foo bar "baz"), just as a non-buggy JSON parser can't do anything wrong with ["foo", "bar", "baz"].

It's a lot of fun reading the spec and realizing you can hold the entire thing in your head, and probably also make a decent implementation in the language of your choice.

It would be neat to take that as a model and try to compose a new type of browser where each functional module must consist of a spec that is similarly succinct and easy to implement, perhaps with the exception of ECMAscript.

Reply on news.ycombinator.com