Fixing JSON | Hacker News Reader

Fixing JSON

tbray.org

139 pointsrobin_reala179 comments

Shameful confession: when I was first introduced to JSON, I was convinced it would go nowhere. "XML already does everything JSON does! And there's no way to differentiate between nodes and attributes! And there are no namespaces! And no schemas! What's the point of JSON?" And a couple of years later I looked up from the tangled mess of XSLT I was working on to discover that the rest of the world had moved on.

JSON is just javascript, and it leaves out everything else. That's its entire reason for existing, and it caught on because that's all that 99.44% of anyone needed.

Timestamps you can add today, without changing the protocol; just put them in your data if you need them. So I'm not sure what he's even proposing there.

Schemas: OK, he doesn't like JSON Schema's anyOf. Fair enough. There's no real proposal here for how to fix it, so not much to say here.

Replacing commas with whitespace sounds to me like replacing a very minor irritant with a constant full-body rash. Stuff like his example of "IDs": [ 116 943 234 38793 ] would lead to far more confusion and errors than the occasional stray trailing comma.

So I guess I pretty much vote no on this one thanks for asking

"Timestamps you can add today, without changing the protocol; just put them in your data if you need them. So I'm not sure what he's even proposing there."

He is proposing to add a timestamp type to the grammar. This would have the advantage that there would be one canonical way to have timestamps in your JSON. It would also mean that the parser would already validate them for you and you would not have to do that yourself every time.

I definitely see value in that.

> He is proposing to add a timestamp type to the grammar. This would have the advantage that there would be one canonical way to have timestamps in your JSON.

There is already a canonical way to have timestamps: encode them as defined in RFC 3339.

Done.

The proposals add nothing to JSON and don't make sense. I mean, screwing up with the language just to add a very specific data type that is used in only a specific corner case that isn't even supported in the use case that's provided as an example? Nonsense.

Are you proposing that timestamps as described in RFC 3339 be put inside quotes and just be string values in JSON? That's the simplest way I can think to do it, but I've had adamant protest to that idea.

If you are not suggesting that timestamp values be wrapped in quotes, then wouldn't you have to worry about every existing parser out there tripping on them?

The former I assume. It sounds fine to me. if one is worried aboutfragmentstion, create a standard on top JSON that provides rules for dealing with other types. Layering 101, people!

> There is already a canonical way to have timestamps: encode them as defined in RFC 3339.

That's not canonical at all. It's not even a de facto use of timestamps in JSON; most specifications I see call for ISO 8601.

> Done.

Well no, not done, because you then have to wrap it in a string, which essentially hides it from the JSON parser altogether and moves responsibility for parsing it out of the JSON parser into your application. As far as the JSON parser is concerned, it could be any old string, not a timestamp.

> a very specific data type that is used in only a specific corner case

I don't see how you can call timestamps a corner case – the article says "Among the RESTful APIs that I can think of, exactly zero don’t have timestamps." – and my experience is the same. Pretty much every API I've worked with has used timestamps in some form or another. They aren't a corner case at all – aside links, they are probably the most common data type without direct support in JSON.

> That's not canonical at all. It's not even a de facto use of timestamps in JSON; most specifications I see call for ISO 8601.

Nonsense. ISO 8601 is an ISO standard that defines representations of dates and times.

Date and time representations aren't timestamps.

RFC 3339 is based on ISO 8601 but is designed specifically to handle timestamps by including provisions that tackle interoperability problems that ISO 8601 does not handle.

> Well no, not done, because you then have to wrap it in a string

Nonsense.

JSON is just the languge that is used as a superset for other domain-specific languages. How the other language defined (and parsed) isn't handled by the JSON parser, obviously.

Considering Tim Bray's example, If a domain-specific language specifies that a "Capture time" string is followed by a string storing an RFC 3339 timestamp, that's precisely what the parser for the domain-specific language is expected to parse. The document remains valid JSON, and the domain-specific language remains valid.

> I don't see how you can call timestamps a corner case

Because it is. It isn't a primitive data type, nor is it required to implement generic data structures. Hell, some programming languages don't even support any standard date and time container or aggregate data type. Timestamps are a very specific data type whose application is limited to very specific corner cases that are already handled quite well by other means.

> RFC 3339 is based on ISO 8601 but is designed specifically to handle timestamps by including provisions that tackle interoperability problems that ISO 8601 does not handle.

So in other words, what I'm saying isn't "nonsense" at all, and they aren't fundamentally different things as you claim – you just prefer a standard that you think is better?

In any case, you're getting away from the point there. The point was not which standard was better, the point was that RFC 3339 is not canonical.

> > Well no, not done, because you then have to wrap it in a string

> Nonsense.

It's not nonsense in the slightest. You can't put an RFC 3339 timestamp into JSON without wrapping it in a string, at which point it is no longer part of JSON. All JSON sees is a string, not a timestamp.

> The document remains valid JSON, and the domain-specific language remains valid.

I never said that it wasn't valid JSON, my point was that as far as a JSON parser is concerned, it's a string, not a timestamp, so parsing a timestamp has to be handled by your application not the JSON parser.

> It isn't a primitive data type

When the subject of discussion is whether or not it should become a primitive data type, that's circular logic.

> nor is it required to implement generic data structures.

No, but it is required to implement a vast number of JSON-based APIs.

> Timestamps are a very specific data type whose application is limited to very specific corner cases

The "very specific corner cases" being every single RESTful API the author can think of? This is an incredibly common use case, I don't see how you can argue that it's a corner case when it's practically ubiquitous.

> So in other words, what I'm saying isn't "nonsense" at all, and they aren't fundamentally different things as you claim – you just prefer a standard that you think is better?

It's nonsense, because a timestamp isn't a date representation. This was already demonstrated. I don't understand why you decided to ignore this.

> The point was not which standard was better, the point was that RFC 3339 is not canonical.

No, the point is that the ISO standard you've quoted doesn't define timestamps. Hence, the example you provided to refute what I've said was nonsense.

> You can't put an RFC 3339 timestamp into JSON without wrapping it in a string

...and you can't encode a date without representing the year as a number, the month as another number, the day as anohter number, etc etc etc.

You, somehow, miss the point that a primitive data type is not required to represent timestamps.

In fact, you can represent timestamps in JSON by defining an aggregate type.

Timestamps as primitive data types doesn't make any sense if it's possible to use the types that are already available to represent it.

> my point was that as far as a JSON parser is concerned, it's a string

Somehow, you don't understand that JSON is only the superset language, and that JSON-based domain-specific languages represent specific subsets of JSON obtained by imposing other parsing rules.

> When the subject of discussion is whether or not it should become a primitive data type, that's circular logic.

You somehow already forgot that this particular data type is already representable by using another primitive data type.

> No, but it is required to implement a vast number of JSON-based APIs.

No, it's not.

Wrap it in a string. Done.

If that's too hard to do, just specify an aggregate data type.

Is it that hard to understand?

> The "very specific corner cases" being every single RESTful API the author can think of?

Somehow, no RESTful API was barred from being implemented in JSON because of this corner case.

I suspect you are, somehow, confusing "convenience" with "necessity".

Your arguments are good though I wouldn't exactly call it a "corner case".

meteor ejson format adds a consistent timestamp format with breaking current Jason parsers.

So I guess I pretty much vote no on this one thanks for asking

Yeah, this is precisely the kind of stuff people said would cause JSON to "lose" to XML -- how could you ever build anything without schemas and types and bunches of metadata to ensure you were using it correctly?

And this debate is not new; it's been around a long time, and I wrote about it at length literally ten years ago: http://www.b-list.org/weblog/2006/dec/21/i-cant-believe-its-...

> Replacing commas with whitespace sounds to me like replacing a very minor irritant with a constant full-body rash

Both vivid and accurate.

In order to combat trailing commas I normally place the comma on the following line, e.g.

    { "time": "3 minutes past four"
    , "age": 229
    , "sex": "monoecious species"
    , "appearance": "Tree-like.  It's a tree."
    }

    uint8_t *data    // Yada
          , *buffer  // Ya
          ;

    var javascript
      , variable = 6
      , declarations = [ "This is taking too long", "Yep" ]
      , mix = [ "Flour", "Sugar", "Water" ]
      , types = { 'old'     : (d) => { return d < new Date }
                , 'new'     : (d) => { return d > new Date }
                , 'borrowed': (o) => { return false }
                , 'blue'    : (c) => { return true }
                }
      , regularly = new Date().toISOString()
      ;

With such a format there is only ever a problem deleting the first line, which I find is much much harder to do without also noticing what you've done to the larger statement.

> In order to combat trailing commas I normally place the comma on the following line

Ah okay, so you're that guy whose code I have to reformat in every new job.

If you're having to do it on every job, maybe you're the unusual one

Weirder still - why do I follow him from job to job?

I like where this is going :-)

Ah, so you're the guy whose code I have to re-reformat back to the proper format.

I use industry standard IOCCC formatting rules. Doesn't everyone?

http://www.ioccc.org/2015/endoh1/prog.c

I highly recommend having a couple of reformatting scripts around - its a great way to find logical errors quickly.

Of course don't commit reformats where it diverges from the codebase.

I have seen this approach also used when constructing complex SELECT FROM and WHERE clauses in MySQL

That is horrible from a human understanding perspective

It makes perfect sense to me.

Which to be honest is what matters. However I would imagine that a lot of the problem grokking it is due to not being used to it.

Why?

It looks easier for me; I'm far more likely to notice a missing comma.

But I'm sure you must have a valid point, if you would elucidate?

I came across this formatting pattern in Haskell, but I still prefer trailing commas for one reason: I can trivially apply line-wise operations (e.g. sort or align) on the key-value pairs without breaking the syntax. When I sort your first snippet line-by-line, it becomes

  , "age": 229
  , "appearance": "Tree-like.  It's a tree."
  , "sex": "monoecious species"
  { "time": "3 minutes past four"
  }

and the syntax is broken. With trailing commas, the syntax always stays valid:

  {
    "age": 229,
    "appearance": "Tree-like.  It's a tree.",
    "sex": "monoecious species",
    "time": "3 minutes past four",
  }

With prefix commas it also always stays valid

    {
    , "age": 229
    , "appearance": "Tree-like.  It's a tree."
    , "sex": "monoecious species"
    , "time": "3 minutes past four"
    }

Is the first comma legal in Json or JavaScript? I'd have thought that'd be an error.

Valid but introduces undefined as the first item of the array

You are assuming that trailing commas are ignored as valid, which is true of JavaScript but not of JSON or of C (the other example).

Yes, I'm aware of this being an error in JSON (which is precisely the point of the discussion). I wasn't aware that it's a problem in C. Maybe gcc is more lenient here.

Trailing commas are allowed in C!

C89 v. C99 difference IIRC

edit: I was wrong: http://www.lysator.liu.se/c/ANSI-C-grammar-y.html#initialize...

No: tptacek did not bother paying attention to the context; as far as I know "uint8_t * data, * buffer, ;" (spaces added after * to avoid italics) would never be valid.

How is that the context? In the sense of the intersection between Javascript and JSON, trailing commas have pretty much always worked in C, which is something all of us who write code that generates C rely on, like a lot.

I googled and trailing comma was allowed since C89, I was wrong. For code that generates code I always did something like this cause of that mistake:

  { a, b
  , c
  ...
  , z
  };

The contextual example from C was "uint8_t * data, * buffer;" (spaces added to avoid italics), and no: a trailing comma is absolutely not allowed.

Replied here, to the first time I saw you write this:

https://news.ycombinator.com/item?id=12368022

I was thinking along very similar lines when reading the article by Tim Bray. JSON is designed for JavaScript compatibility, I almost stopped reading when he said:

> They’re inessential to the grammar, just there for JavaScript compatibility.

The entire article goes on to suggest extremely terrible solutions to the problems he is pointing out.

In order:

Commas: If comma's are an annoyance, is that to mean there is no annoyance with XML markup? You can forget a closing tag just as easily, probably more easily. Just get a JavaScript or JSON linter. Fixed. If you are really annoyed by commas, move to YAML or another format.

Datetimes: Just store your dates in a standardized timezone and format, how about [ISO 8601](https://en.wikipedia.org/wiki/ISO_8601). You can also use a [JSON-Schema](http://json-schema.org/) if you really want.

Schemas: There's lots of potentials here. However, again, if you're going to change JSON so much, why not just pick another data format that has better schema support.

For datetimes, do you mean that they be set as strings in the standardized formats such as specified by ISO 8601? Like:

{"item": "lawn chair", "order-timestamp": "2016-08-21T00:58:47Z"}

Yes, that would be a strongly recommended way. I could see using a unix timestamp as a numeric field as another way.

> JSON is designed for JavaScript compatibility,

Javascript compatibility is the least important part of json, lots of people use json with python or ruby or rust or java or c for things that will never interact with javascript, and anyways you don't want people to parse json with eval or confusing json with things that include string concatenation or function calls adding non javascript syntax helps make that clear.

There is a good reason to not change JSON(backwards compatibility, updating parsers, confusing formats, ect.), since you're not planning on parsing javascript with eval there is no good reason why any changes made in JSON 2.0 have to conform to a subset of javascript.

I think if breaking changes are made, there is no point, it would not be called JSON 2.0; it is an entirely new data format. When there are hundreds to thousands of alternatives, trying to "fix JSON" isn't the solution. JSON works because it is simple and easy to follow, it is robust and flexible, and has no restrictions. There is no good way to store dates in XML or many other formats, without the use of Schemas, and the syntax changes are minimally beneficial. It wouldn't save a lot of bandwidth with gzip, and the text data likely doesn't add up to a lot of disk space when compared to the data being stored; so removing commas which make parsing JSON easy and increases its adoption and usage, just so some people can have less linting errors doesn't seem like a large benefit.

>JSON is designed for JavaScript compatibility, I almost stopped reading when

Why does Javascript compatibility matter in 2016? The original idea for JSON might have been to parse untrusted data using js eval() but I'd hope that nobody is doing that anymore.

I don't believe the original purpose of JSON was to eval untrusted objects, since that would always have posed a security risk. It did however offer a very easy standard way to share and parse data back in 1999. In 2016, I believe continued backwards compatibility is a necessity; otherwise you are just creating a new data storage format. So removing commas doesn't seem like an acceptable improvement, especially when the recommendation comes out of a necessity due to human error while hand-editing. JSON is incredibly easy to read and write by hand, in comparison with XML it is far easier to parse and traverse through code do its significantly simplified structure. If however, you need more functionality, switch to a different format; why bother trying to mangle or force JSON to work where it doesn't. There are other technologies and standards out there, use them.

>It did however offer a very easy standard way to share and parse data back in 1999.

Not 1999. 2004. JSON wasn't on anybody's radar in 1999.

>JSON is incredibly easy to read and write by hand, in comparison with XML

Maybe for small shallow objects (and for those XML is also quite readable). Once size or complexity get a little higher, you're done.

> Once size or complexity get a little higher, you're done.

And what is better than JSON, when it gets complex?

I meant you're done with reading and writing it by hand.

> If comma's are an annoyance, is that to mean there is no annoyance with XML markup? You can forget a closing tag just as easily

That's not the same thing at all. The commas in JSON are equivalent to the whitespace between attributes in XML. A missing closing tag in XML would be equivalent to forgetting to close an object literal or array in JSON. So no, there's no equivalent XML annoyance to what he's asking for, XML is already fixed in the manner he suggests.

The issue is not about forgetting something. XML elements are self-contained, you don't need to separate them, thus you don't need to known the context. If you generate XML programmatically, each function can just push its object into the stream and be done. With JSON you need to know if your object is last or first at this level of hierarchy, so you know if you need to insert a comma or not.

Commas were meant to be used in code that you actually type. Using them as separators in a format that is meant to be produced and consumed by computers is clear, simple, and wrong.

> JSON is just javascript,

Today json is not really tied to javascript that much, I deal with it all the time in python that doesn't include or interact with javascript. JSON is just dictionaries and lists (and strings and numbers and nulls) this is pretty easy to express in dynamically typed languages and not that hard in statically typed languages if you give up on a bit of (static) typing. I don't see any reason why it should be limited to a subset of javascript, there are some good reasons for not changing it all though.

Why not just allow trailing commas?

Well, totally. Lua and C don't care. A minor change, breaking the strict-subset-of-javascript which is no longer important.

Well, not really. ES5 allows trailing commas in object literals too.

I loved JSON immediately, having hated XML for so long.

However, I'd agree with your conclusions. The minor irritations outlined here are less bad than breaking javascript compatibility.

Except you can put comments in your Javascript ... I'd love to have comments in JSON. It's not so important for machine generated files but when you hand-craft an example it's nice to annotate it.

> I'd love to have comments in JSON. It's not so important for machine generated files but when you hand-craft an example it's nice to annotate it.

Luckily, Douglas Crockford has an explanation and an almost prophetic solution[1] for your use-case: I removed comments from JSON because I saw people were using them to hold parsing directives, a practice which would have destroyed interoperability. I know that the lack of comments makes some people sad, but it shouldn't.

Suppose you are using JSON to keep configuration files, which you would like to annotate. Go ahead and insert all the comments you like. Then pipe it through JSMin before handing it to your JSON parser.

1. https://plus.google.com/+DouglasCrockfordEsq/posts/RK8qyGVaG...

I mostly agree, except I still like and use XML in many cases. XML Schema validation is just too useful as a first pass validation step to verify the "shape" of a message/document.

And to me, with any decent editor with folding, it is easy to read. And very importantly, allows comments. I have never understood all the XML-hate out there...

> I have never understood all the XML-hate out there...

Me neither. I think it's truly irrational.

While I respect tbray we need to fix engineers that try to 'fix JSON', absolutely nothing is broken with JSON.

It is nearly a perfect spec for the role it plays. It has taken off because it is so simple. I am constantly amazed that something so simple keeps being jabbed at with more complexity by engineers.

The job of an engineer is to make things that are complex simple, JSON did just that. Make a new standard for your fixes and watch them spiral out of control like XML. Making a simple format that works is damn hard and JSON is a nearly flawless attempt. It is baked into the ecosystem so let's not break it. It was a response to XML complexity/verbosity for use in javascript first, it is so damn useful that it is used everywhere now, why? simplicity.

Everything you need can be done with current JSON. Dates in quotes or ticks (another key for offset if need be), comments in separate files (do you really want © Oracle or Microsoft in all your data files?), strong typing GTFO (use a schema if verbosity is needed -- it supports all needed base types string, numeric, list, object), and it is Javascript Object Notation so it's purpose was to tie to Javascript, don't even attempt to break it from Javascript. If you need more use YAML or XML or a new standard.

If you are trying to change JSON, back away slowly, go add your complexity for minimal or worse benefit elsewhere...

In my opinion there are five problems with JSON in practice:

1. Trailing commas should be allowed. This would fix the entire comma ordeal the author is complaining about without introducing awkward whitespace logic.

2. Date time values should be a distinct type using the format of JS's Date.prototype.toISOString method. Had this been in the language when JSON was originally defined I'm sure it'd already be in JSON.

3. There should be a way to represent Infinity (for IEEE 754 compatibility) and possibly NaN.

4. Comments would be nice because JSON is often used for configuration and documenting JSON snippets can get awkward without them.

5. There's no real way to represent binary data other than base64 encoded strings. It would be nice if there was at least some hex format (e.g. `"blob": <dead beef cafe babe>` with optional spaces for readability) but I can see why some people might be opposed to the idea of adding this to the language

Although I'm personally not _as_ bothered by 5, the rest of these are spot-on. The other big annoyance for me is the inability to use anything other than string as an object key (numbers, for example, but others would be interesting/useful too I guess).

6. I'd love to see unquoted keys added to that list. Quotes could be used for cases when a key is not a valid identifier.

Yes and double yes to comments!

Otherwise it's useless as a configuration language

I used to think this, too.

Turns out, there is a really good reason for the lack of comments in JSON.

Which is?

To avoid them being used as parser directives.

I'm a bit wary of the second and fifth one. You could add an arbitrary amount of datatypes. Also I think the common argument against comments makes much sense: they could get abused easily for parsing directives etc. JSON isn't that nice for configuration files anyway IMO.

I would actually take away a feature of JSON: Unicode escaping. It's not needed anyway and astral plane characters being encoded as surrogate pairs is an added complexity for encoders/decoders. The selection of escape codes for control characters also is rather arbitrary.

The argument about abusing comments with parser directives is totally bogus. You can create parser directives with extra field names too! Worse, people create comments anyway using fields like "//" -- but they're much crappier than real comments, since they have to be a valid string value. We gave up a solution to a real problem in favor of a bogus solution to a theoretical problem.

Two more issues:

- No support for comments.

- No support for NaN and Infinities (i.e. lack of full mapping to IEEE 754).

Also at least on Chrome, JSON.stringify(-0) gives 0, though JSON.parse("-0") does give -0. Weird..

As far as schemas are concerned, I find it nicer to not use schemas but access the JSON through an interface that checks things on the fly (types, existence), effectively verifying the JSON while you're examining it. E.g. things like x.get_int('name'), which throws exception if it doesn't exist or isn't an integer. Possibly even with an error message explaining where exactly and what is wrong (like: foo.bars[4].bazes["abc"]: attribute "qux" does not exist).

The original JSON spec included comments. They were removed because people were using them as processing directives.

With the exception of trailing commas, all these things would break the fact that JSON is a subset of JavaScript. Breaking that would cause a lot more problems than it solves. It's just that the minor problems you have to live with seem worse than the major problems you don't.

What value is there in having JSON be a subset of JavaScript? In fact, it would be a good idea to make it not be a subset of JS, because that would prevent people from evaling it.

Most major browsers now support non-eval JSON parsing that it's a moot point - but it still requires it to be valid JSON, also be be usable in javascript out of the box.

JSON has the same thing going for it as the h.264 video codec - it may not be _the_ best/most open/etc. format around, but it is one of the few that is supported out the box by all major browsers and Just Works™ (with it's shortcomings).

Sure, but there's a difference between "we shouldn't break an existing format" and "it has to be a subset of JavaScript".

It wouldn't be JSON then. Give this new thing a new name and move on.

To be honest, I agree that there's really no point in changing JSON. I do think there's value in using something other than JSON.

It's also (almost) a subset of Python, and possibly other languages. The benefit is that it is trivial to remember, because you already know the syntax. And you can not just eval it (which you if course shouldn't do if you didn't write the JSON yourself), but more important you can just copy it into your code. Or start with a object literal in code, and when it grows, or you want to use real-world data, switch to an external data source.

There are a couple of projects which extend JSON. I believe an ideal JSON superset would be compatible with ECMAScript 6 object literals, and have comments and trailing commas.

JSON is pretty close to being a subset of Lua, as well.

Well = vs : is a bit different. And requiring quotes around keys.

> It's also (almost) a subset of Python

Yes, one of my main uses of JSON in my personal projects (and at one of my previous companies) is to serialize a Python dict to disk or network (and to read a Python dict from disk or network, of course). If I keep the members to int/float/str/dict/list, it works beautifully, and if I need to serialize anything more complex, I can always use jsonpickle.

Using JSON in javascript to transport data from some backend language to the browser in a simple and easily-parseable format is probably the most common use case for it - and no one has to worry about JSON support on the frontend as long as it's a valid subset of javascript, because it "just works" as long as javascript works.

Breaking that would only lead developers to write code to unbreak it.

To be pedant, JSON is NOT a subset of Javascript [1], using U+2028 or U+2029 inside a JSON string is perfectly valid, but is not in Javascript.

[1] http://timelessrepo.com/json-isnt-a-javascript-subset

Don't forget also that large integers are perfectly valid JSON (which imposes no semantics on numbers except that they are representable in decimal), but do not survive translation to JavaScript, which has only IEEE floats.

the loose specifications of json types is my biggest gripe as the maintainer of a json library. if json mandated that all strings be representable as utf8 byte sequences (after resolving escapes) and that all numbers be representable as ieee 754 doubles it would be much easier to provide sane and consistent defaults for encoding and decoding. as it is all implementations have to make explicit choices as to how to handle things like escaped strings that represent invalid utf8 byte sequences (like `"\uD800"`) or numbers unrepresentable with a double (like `1e400`). there isn't even any sort of consensus on how to decode a number representable as an integer. some implementations produce integers, others floats

Isn't the simple solution to decode e.g. a number as a special type, say JSON_number, which can fully represent what's on the wire? Then the user can access the number via one of a number of conversion functions depending on the native data type he/she wants (e.g. JSON_number_to_float). Is there really a huge benefit to eagerly performing this conversion?

that's a simple solution yes, but also a verbose one. the primary appeal of json as a data exchange format is that it maps to ubiquitous types like objects/hashmaps, arrays, strings and numbers. if you need to hint the decoder or manually perform conversions it loses a lot of appeal as a user

> What value is there in having JSON be a subset of JavaScript?

* Virtually zero learning curve if you already know JS.

* It can be pasted into and out of a working program.

* Constrains the format from becoming bloated.

> Virtually zero learning curve if you already know JS.

What if you know other languages but not js?

> * It can be pasted into and out of a working program.

Again only true for js

At this point, I'd settle for JSON5 (http://json5.org/), which says any JSON that's a valid ECMAScript 5 object is valid JSON (including trailing commas, unquoted keys, single quote strings, and comments). I don't think it's very popular at this point though, but I guess I've never tried to use it.

This is a good idea. Also browser makers won't feel like they're supporting another format. It's just a modernized version of JSON.

Ugh no. Adding all that crap complicates writing a correct parser. JSON's value is in that it is easy to parse.

If editing JSON is your problem, use a better editor. If you really want to use JSON as a config file format, use something like YAML instead.

Or TOML, if you find YAML's syntax too messy.

JSON is not optimized to be written by humans (I'd argue even JS is not well optimized for that </joke>).

There are other formats for that, like YAML and TOML[1].

When JSON is needed for human-to-computer stuff, I'd say TOML is pretty much always a better choice. A common use case for TOML is config files. And I see a lot of config in JSON lately; this makes me sad. (Though not as sad as the config-in-XML thing the Java ecosystem suffered from.)

1: https://github.com/toml-lang/toml

TOML is a good idea. Maybe it will catch on and sometime within this decade we will finally have a standard format for config file </wishfulthinking>

TOML is kinda ugly compared to YAML.

YAML is a pain to parse. And as a human, having to remember quoting string values like "yes" and "on" to prevent them from being interpreted as booleans in YAML really grinds my gears. I don't care if it's prettier; YAML can't disappear quick enough.

I wrote a haiku some years back to demonstrate how broken the Ruby implementation was.

    YAML:
      takes a no for a 'no': no
      spring,summer,fall,winter: 3,6,9,12
      the game is: on

Running it through ruby 2.3.1 yields..

    {"YAML"=>
      {"takes a no for a 'no'"=>false,
       "spring,summer,fall,winter"=>36912,
       "the game is"=>true}}

I agree. That's partly why I wrote this: https://github.com/crdoconnor/strictyaml

e.g.

    >>> load(x, Map({"YAML": MapPattern(Str(), Str())}))

    {'YAML': {'spring,summer,fall,winter': '3,6,9,12',
      "takes a no for a 'no'": 'no',
      'the game is': 'on'}}


    >>> load(x, Map({"YAML": Map({"spring,summer,fall,winter": Str(), "takes a no for a 'no'": Bool(), "the game is": Bool()})}))

    {'YAML': {'spring,summer,fall,winter': '3,6,9,12',
      "takes a no for a 'no'": False,
      'the game is': True}}

Think of it as ini 2.0.

Counter-point: you can tell if the JSON you received is intact or was cut off, because it won't parse properly without the closing braces. You don't have that safety net with YAML or TOML.

That's a really good point. I opened an issue: https://github.com/toml-lang/toml/issues/429

I'm not sure how dropping comma separators would work in practice, because carriage returns aren't required between array or hash elements in JSON.

So this would also have to be legal:

   { "foo": "bar" "baz": "quux" }

I'm not sure that's better, personally. It's also syntactically invalid in Javascript:

   > "bar" "baz"
   SyntaxError: Unexpected string

Let's just remove periods from sentences to They clearly aren't needed since there's a capital letter there

If anything I'd rather trailing commas in lists be defined as valid, and separation there is useful like you say

You could use "line-break" instead - but I also don't see really the point in removing comma's in the first place.

I know this is from Tim Bray, one of the primary authors and/or maintainers of XML, RDF, Atom, and JSON, but while I too find some annoyances with JSON, I accept them because:

- JSON was never designed to be hand-editable. It's a lucky accident that it's more palatable to edit than XML, but there are other formats like YAML, TOML, and even .ini that fulfill that purpose.

- While JSON was originally envisioned as a subset of Javascript and intended to be materializable by JS eval(), this very quickly turned out to be a bad idea. Once we all switched to using parsers, there is no more compelling need to remain compatible with Javascript, other than human recognizability of syntax, and, y'know, compatibility with 'old' JSON; and

- JSON's greatest strength is that it has such a good impedance match to native 'abstract' data structures supported in just about every contemporary programming language: strings, numbers, lists, maps. When people say 'simple', I don't disagree, but I'm convinced they're actually referring to this feature. Adding additional types -- yes, even timestamps -- would break this property, as datetime handling in most languages leaves a lot to be desired.

The first language that comes to mind when I hear "datetime handling in most languages leaves a lot to be desired" is, slightly ironic considering JSON's origins, javascript. If you ever touch a date in JS you HAVE to pull in something like moment, full stop. JS Date is a steaming pile of shit.

To go on a tangent, Java8 dates are "awesome" if you've used Joda-Time before, for the rest of us it's a pain in the backside. I found myself spewing lots of code to just do Moment one-liners. That made me appreciate Moment much more, to the extent that I sometimes just eval a Moment script with Nashorn at times.

On the JSON part, I agree with everyone, my go-to language for most things is JavaScript, and I'm okay with dates in JSON. I think it's RFC3336 as everyone says (2016-08-20T21:12:26.231Z).

Date parsing could be better when moving across languages, but I mostly prefer sending around millis since Epoch. Considering that people mainly look at their JSON for debug purposes, anyone seeing 1471727663538 should recognise that it's a date.

Other commenters put it well that the author should rather propose a new language, JSON is fine the way it is. Typed successors like Protobufs don't have native date parsing, but I guess being binary, they take away the commas too. I'm comfortable hand-writing JSON or editing it when needed. Most text editors help out by highlighting errors and keeping track of parentheses.

To continue the tangent, perhaps you read my mind, because I believe that Java8 and Joda (and their ports) are the only two libraries that do datetimes properly, and every other one confuses and/or conflates human calendar and clock concepts with absolute point-on-a-timeline timestamps, and the like.

But this illustrates the point: datetime is hard and we don't agree, so even going so far as retrofitting a native RFC 3339 datatype into 'official' JSON is, in my opinion, too far.

I'm fairly new to date manipulation in Java, between LocalDateTime, Instants, TemporalAdjusters etc. I get confused, but I'll catch up.

The comment I replied to mentions Moment for JavaScript, which I would add to your list. It's beyond excellent, great documentation as with most good quality libs, and very intuitive.

I agree with your point and opinion. Someone mentioned on another thread that engineers are there to simplify things, 'official' JSON has done a darn good job at that, no surprises across languages there.

Objective-C (or more precisely Foundation) does dates correctly, but the API are not so great.

I thought that the Date API was closely copied from Java? I do agree it is nasty to use.

Tim's ideas are just dangerous - they ruin compatibility for questionable gains. Why not just invent his own standard and call it something else, rather then trying to bust what we have already.

If so when it was copied Java had shit DT handling. The lack of a concept of timezones and auto-converting a UTC time to local time make life a living hell as soon as your code has to span more than 1 TZ.

Mandatory double-quotes for keys just to get rid of required commas does not make it better.

There are better relaxed JSON's available than this one:

* https://hjson.org/ does not need those double quotes and can handle without commas just fine.

* https://github.com/phadej/relaxed-json also allows comments, trailing commas, simplier strings as in YAML.

* RJSON http://search.cpan.org/~miko/JSON-Relaxed-0.03/lib/JSON/Rela... is similar.

The proposal doesn't fix JSON it introduces an entire new format. One which I would not be able to use without a 3rd party library that parses it. If I'm already using JavaScript that is one of the many benefits it has in the first place.

Timestamps are easily included when I want them, I tend to prefer UNIX time as it's just so much more impact proof. With this proposal can I use UNIX time if I want?

I don't have any problem with JSON. If I did I would have a problem with JavaScript itself, and while there's plenty to complain about I wouldn't propose mixing more formats into it as a solution. There isn't enough care being taken not to introduce additional complexity into the JavaScript world.

Dave Winer's remark from 2006 offers some historical perspective:

"I've been hearing, off in the distance, about something called JSON, that proposes to solve a problem that was neatly solved by XML-RPC in 1998, the encoding of arrays and structs in a format that could easily be processed by all programming languages. The advantage being that you could easily support the protocol in any language that supported XML and HTTP, which, at the time, was quickly becoming all languages."

http://scripting.com/2006/12/20.html

Then there was someone, much younger than Dave Winer, who wrote a response to this (during 2006), and who said (paraphrase) "When I'm older, I look forward to my XML moment; my horror at something that the young kids are doing."

I wish I could find that second quote, but I worry it is lost in the mists of time.

If you're going to drop the commas in the dictionary definitions, why not drop the colons too?

    { "key" "value" 42 37 }

It's a rhetorical question; I think removing the commas is a bad idea. Troubles while hand editing json shouldn't be an overriding use case that drives the syntax.

All of those complaints about json and nothing about how you can't have comments?? I feel like Tim Bray and I aren't even on the same planet.

One way to get a comma-free JSON-like format is to write the document in hjson, which is a superset of JSON.

http://hjson.org/

var config = HJSON.parse(fs.fileReadSync('config.hjson'))

Another more obscure, and more powerful serialization format that does away with commas (commas are treated as whitespace!), has a date type, and much more is Rich Hickey's (of Clojure fame) EDN. https://github.com/edn-format/edn

{:a 1, "foo" :bar, [1 2 3] four}

Major irritant: Commas

I thought the major irritant was having to add double-quotes around keys? Commas never bothered me personally, but having to use double quotes is actually irritating.

I was thinking the same thing - maybe I haven't hand-edited enough JSON but I have never felt like commas were any sort of issue at all

It comes from the flow of edition: remove the last element of a list (which typically is on its own line), forget about the fact that there is a comma at the end of the previous line.

But both optional commas and optional double-quotes are provided by Hjson <https://hjson.org/>. They do not advocate its use in protocols, however.

Inability to add trailing commas creates two problems:

- You can forget to drop/add comma when copying an existing line - Adding/Removing last element shows up as two changes in diff

JSON was fixed in 2014:

http://blog.cognitect.com/blog/2014/7/22/transit

https://github.com/cognitect/transit-format

Sounds nice, is anyone using it? Is it somewhat readable by humans?

Two quick observations:

- Getting standards passed is hard (especially for popularly used standards), but can be supported by latent usage in the community. You can create a plugin for a few languages that automatically allows the timestamps and doesn't require commas. Get a bunch of people using it, and you'll be on your way to actually getting it in the standard in the future (still debatable in this specific example, but broadly, usage never hurts for getting something into a standard)

- If your concern is your own frustrations (rather than benefits of the vast majority of people) - I highly recommend a simple editor-level mapping that translates from your preferred DSL to the underlying standard

I agree with those who say JSON should never change, but should one day be replaced. It behooves us to think about what that replacement should be.

My own peeves:

* Numbers are stored in decimal but usually interpreted as binary with no guidance about rounding.

* The concatenation of valid JSON blobs is never valid JSON.

* The empty string is not valid JSON.

The last two can be "fixed" by a format called "arraySON" where the global blob represents not a JSON object (dictionary), but the contents of an array plus an optional trailing comma.

Or given some of the ideas on this thread, perhaps a manadatory leading comma.

I'd see a trailing or leading comma as a potentially null, or undefined. It's intentionally against spec. I personally see the concatenation an issue in particular with streaming. Streaming JSON objects and parsing them on the fly is a pain, usually requiring a SAX parser. To concat two objects though, I feel like you could just merge the two, if you are okay with appending two arrays, if you had to merge two JSON objects you could. Numbers I seen issues with, I've had to store numbers as strings before, there is guidance however, the official ECMA-404 spec for JSON says there can be any number of digits following a "." in numbers, then optionally followed by an "e", "+" or "-" and more digits. For parsing then, it would be limited by the data types available to the parser. Since JavaScript itself is 17, so that would be the default maximum for JSON IMHO. However, different libraries for different languages may not have 17 for the native decimal types.

> * The concatenation of valid JSON blobs is never valid JSON.

You might want to check out JSON Lines:

http://jsonlines.org/

Is there are formal defintion of the format anywhere?

If you did all of that you'd still fall way short of what you have been able to do with XML for probably 10 years. XSD defines structure in terms of inheritance, composition, strong types, numerical ranges, enumerations, referential integrity, namespaces.

Raise your hand if you miss hand-editing or reading XML.

Requiring that a markup language (that must be sent across the wire between multiple buggy parser implementations) be hand editable is a recipe for disaster, and will never be as flexible as building/modifying an appropriate AST in a scripting language.

There's not even a good reason to read HTML anymore. You hit Ctrl+Shift+C in Firefox (or I think it's J in Chrome) and you view the actual machine-parsed DOM structure. The only reason human-editable/readable formats were ever necessary is a lack of appropriate dev tooling.

I have to agree, I've been working with JSON for 10 years and nearly never touched JSON by hand.

Raising hand. If your IDE understands XSD, it is far easier to do than JSON. Tab completion on all your fields and instant validation of values.

I'm sure Tim Bray would have some opinions on that.

Those things aren't a goal or desired feature of most JSON use. There's a reason people hated SOAP. XML is better suited for documents. For everything else, XML doesn't work very well.

Personally I would rather it require trailing commas like Go does, any optional syntax is a step backwards IMHO

The best way is to require one comma between elements but allow one trailing comma after the last element. This way you can both do multi-line and one-line lists nicely.

I never understood why the double quotes around property names are mandatory. For example, in JavaScript, I can write

    { foo: 1, bar: 2 }

but JSON syntax insists on

    { "foo": 1, "bar": 2 }

This makes JSON less easily readable by humans, and harder to write or edit by hand.

Anyone know why JSON was designed this way?

I'm no expert, but I'd imagine fully dropping the quotes wouldn't allow you to use the same syntax as a map/dictionary in Javascript, whereby the keys could be [almost] any type, and string keys could contain characters such as - or even : which otherwise would prove ambiguous to a Javascript parser.

I don't know the history of why design decisions were made in JSON, but one might be the allowance for spaces in keys and mirroring the usage within arrays (where commas and spaces may be used in a string, but shouldn't be confused with a record separator).

Per my other comment, there are a variety of plugins on the editor side that will make JSON look like this (e.g., vim-json ). To write out to JSON from that format, you can obviously just write a plugin (in JS this is one line).

The stricter syntax allows JSON to be directly parsed outside of JavaScript in other languages like Python. This is a nice feature and one of the selling points of JSON.

My assumption is that quotes are needed for keys with spaces.

    > x = {}
    > x["foo bar"] = 1
    > x
    { 'foo bar': 1 }

That only explains why they are permitted, not why they are mandatory.

Because using certain keys could break your JavaScript (and JSON was designed to be a subset), like

    { delete: "me" }

If you don't want to keep a list of known keywords around, just insisting on quotes is the easier way.

[EDIT: See also this YouTube video, where Douglas Crockford explains it himself: https://youtu.be/-C-JoyNuQJs?t=5m]

Unquoted keywords are valid ES5.1 object properties.

    alert(JSON.stringify({delete: "me", for: 1, var: 2}));

results in:

    {"delete":"me","for":1,"var":2}

The only browser it does not work in is IE8 and earlier.

JSON significantly predates ES 5.1

JSON values low variance between different representations of logically equivalent documents. It's not a primary concern (e.g. the explicit lack of key order allows a lot of variance), but it seems to trump syntax convenience just about everywhere except whitespace. Going all unquoted-keys would also be low variance, but this would restrict keys from "any string" to "valid javascript identifiers" or something bespoke in between.

I don't know who the OP is but I'll assume he's an authority in JSON since he talks about how he wrote some RFCs and stuff.

Anyway this is why I think there should be a better way to move tech standards forward than leaving it up to some small number of people who are way too ahead of the curve that they think everyone else has the same problem they do.

In my opinion there is nothing wrong with commas. I am sure the OP had his own issues while working too much with JSON, but for the 90% of the population it's not a problem and even a better way to represent data.

As an example, [116, 943, 234, 38793] is 100% clear and intuitive. Anyone who has seen any array representation will get what this means. But change it to [116 943 234 38793]? There are so many ways you can interpret this. I guess the only people who would think this is better than the 100% intuitive [116, 943, 234, 38793] option are JSON nerds who think too much about this (and when I say JSON nerds I don't mean JSON "users", but people who try to come up with the standard)

As someone who really hates what's going on with javascript (with all the new syntax that needlessly sacrifices simplicity and intuitive nature for the sake of making things shorter), I can't really think nicely about these opinions.

I think these standards that are used by a huge population should move forward in a more democratic way. Or at least let more "normal" people make the decision.

The author, Tim Bray, is one of the co-authors of the XML specification; https://en.wikipedia.org/wiki/Tim_Bray

Anecdotally, I've really enjoyed using ION[0] at Amazon. It has all of the features the OP wants, besides semantic whitespace which I'm not a fan of.

[0] http://amznlabs.github.io/ion-docs/index.html

Commas irritate me too. But I think removing them from The Serialization Format Known as JSON is a mistake.

My workaround is to use CSON, since I'm already using CoffeeScript. It add the syntactic sugar where I'm hand-editing JSON, yet it doesn't impose my tastes on the rest of the world.

Dude really? Commas are not a big deal and you're talking about a major change to get rid of them; making white space matter. Currently white space doesn't matter. In order to get rid of commas it has to. That should not be glossed over or treated lightly.

I don't agree with the "timestamps" part, for several reasons:

1) the author does not mention in his example the use of timezone offsets (which are a part of RFC3339), that's a bit irritating

2) Timestamp parsing and emitting sucks, no matter the language, because the format specifiers are wildly inconsistent across platforms. I prefer the good old UNIX timestamp, because with it there's no complex massaging needed in order to operate with the timestamp.

Also, the author is missing comments entirely - no matter if shorthand // or long-form /* */ gets included, JSON desperately needs commenting ability; but on the other hand I'm worried about compatibility with older systems...

1) The author does. Z means UTC.

2) The JS Date type has a "toISOString" function that emits an ISO date time string like the ones used in the examples. Since everything else is already based on what JS does, why not just move that behaviour into the JSON spec as well?

Just use yaml?

Or if you still want something easy to parse, s-expressions.

Could you please elaborate? I'm vaguely familiar with S-expressions from my cursory inquiry into lisp (planning to take the plunge in November), but I don't see how they can be applied in this context.

S-expresions are really just a way of representing a tree of data (in Lisp code, the S-expression literally represents the abstract syntax tree that is used for the code).

Take the JSON example:

    {
      "IDs": [116, 943, 234, 38793],
      "Settings": { "Aperture": 2.8, "Shutter speed": "1/250" }
    }

You could rewrite this as an S-expression as:

    ((IDs (116 943 234 3879))
     (Settings
       (Aperture 2.8)
       ("Shutter Speed" "1/250")))

Thanks! I figured it would be simple, and indeed it was :)

YAML is not as simple as one might think. The spec[1] is 80 pages long after all...

1: http://yaml.org/spec/1.2/spec.pdf

I feel like YAML might be a bit to complicated in the other direction, with many ways of encoding quite complex things and the connected risk for bugs. (E.g. the Python-binding, pyYAML, can't even read its own output correctly in all cases)

Commas aren't a huge deal for me personally but if there were to be a "json2", dealing with them and removing the double-quoting requirement for keys would make authoring a little easier. Ultimately these are issues that are easily solved with a linter though.

A standard date type would be huge however, and I'll add to that built-in support for comments. These two things would expand the scope of what you could do with JSON. JSON is already used in so many places for config files but without comments it can be a poor choice.

JSON is fine for what it was intended to be: a way to serialize fundamental data structures to disk or over the network.

If you want something optimized for human editing, such as for a configuration file or a DSL, we already have YAML for that. Hell, JSON is a valid subset of YAML, so you don't even need two parsers.

YAML including the entirety of JSON is not a positive argument in favor of YAML.

I think JSON is great, because it's simple. Who really needs something more oriented for humans, take a look at HOCON.

https://github.com/typesafehub/config

I'd argue that getting rid of commas would lead to an error-free experience. Say I use UUIDs:

    { "884fbf28-b285-49c2-8335-e42aa3fc896d" "f08bc223-2b49-4d49-a308-7bb93691cef8" }

    { "884fbf28-b285-49c2-8335-e42aa3fc896d", "f08bc223-2b49-4d49-a308-7bb93691cef8" }

Another example is long ids:

    { 214748364721474 89234789 7248374293843 23874827 }

    { 214748364721474, 89234789, 7248374293843, 23874827 }

Should we take into account text format and things like font-spacing? I say yes, because user experience is what's valuable here.

Edit: formatting

Json is flawed in any way possible, imo. It was intended to be edited and read by human; it can not comments, trailing commas, non-quoted keys. It was planned for universal exchange; it doesn't contain dates, full-floats, byte arrays in readable form. It can't have backreferences for graph exchange or per-packet string interning.

Even "obvious syntax" argument is wrong. "Key": value is not what comes to mind unless your parents spoke to you on js.

Why it is still here: legacy that was easier to read-write than sgml-flavoured monster. Other arguments are nonsense.

Hand-editing JSON may not be the most important way of interacting with it...

In that case, the arguments in favor of breaking long-established compatibility for the sake of hand-editing should be particularly strong.

If personal forgetfulness about editing around commas or date strings are strong arguments, then they should surely also apply to every programming language that uses commas or lacks the concept of a date literal. After all, the code of programming languages exists to be edited, and JSON is merely a means of serialization.

Regarding timestamps, the at-sign is already used by GNU tools, as a prefix to indicate actual UNIX timestamps (seconds since epoch): https://www.gnu.org/software/coreutils/manual/html_node/Seco...

I would like timestamps introduced as a first class citizen, and can be done relatively cheaply. As the OP said, they are very common in protocols, and it would be nice to have 1 way of encoding them. Especially 1 that doesn't involve an arbitrary offset...

The other suggestions are interesting, but rather distracting IMHO.

JSON is a beast to deal with for anyone who's manually editing it. All the comments here about special formatting strategies to avoid comma problems, or how to workaround the lack of a timestamp type just prove the point that JSON needs to fix these things.

We are using https://hjson.org/ to describe metadata of xml/json API and how to parse the result. It solves all the problems mentioned in the post.

Too much effort for too small a payoff.

Would trailing commas and timestamps be nice. Yes. Would they be so nice that dealing with the json to json 1.1 transition would be worth it. Not really.

On removal of commas I'd just disagree.

JSON isn't broken. There isn't anything to fix. :/

I miss AS3.

Or you know, use something else like YAML.

Removing commas makes minification less useful.

Just turn it into a Python dict already. ;)

Timestamps should include timezone offset.

His example uses RFC3339 which does include timestamps.

Reply on news.ycombinator.com