Douglas Crockford: Why I removed comments from JSON
plus.google.com
plus.google.com
{
"//": "I approve of the choice to omit comments from JSON."
}> I removed comments from JSON because I saw people were using them to hold parsing directives, a practice which would have destroyed interoperability.
I do not read something on the lines of "you can also use a comment as a property". I wonder how you read that in the first sentence?
>I do not read something on the lines of "you can also use a comment as a property". I wonder how you read that in the first sentence?
lol, please re-read my comment. I said the first COMMENT, not sentence, had the same suggestion.
:)
Secondly, the second you start putting documentation in your JSON, be prepared for services to break if you ever change that documentation. It may be silly to do "if (json._doc_.indexOf...)", but stranger things have happened and if you are making an API it is generally a bad tactic to require your clients not to be silly.
"But I intend to have it in there solely because I can't use real comments -- I don't plan on shipping these". OK then probably no problem -- but at this point they're really no better than normal comments (with respect to the issue that Crockford brings up) since theres no reason you can't put compiler directives in the "comment" properties:
{
/* do something special mr. proprietary compiler */
"compiled_property": "magic"
}
vs. {
"//" : "do something special mr. proprietary compiler"
"compiled_property": "magic"
}
So the possibility of diverging incompatible implementations remains -- however I think the real reason this didn't happen wasn't the lack of comments but rather that JSON's biggest selling point is almost universal compatibility.This isn't bigotry. This is just the way things are.
FWIW, English isn't my first language, Portuguese is, but I have native fluency in English.
Almost all of them can read English remarkably well, but some of the emails I get from them... Often grammar and syntax are nowhere to be found, and the vocabulary is occasionally amusing (a couple of the memorable gems have been "USB sticker", presumably thinking of "USB stick" (and still ambiguous in the context of the message, what they were talking about was a USB WiFi dongle), and "redeploy" referring to a reboot or power cycle).
Indeed, I don't think this situation is all that different from Medicine or Biology. Parts of anatomy, diseases, and taxonomic classifications will forever be known by their Latin names. Likewise, programmers will probably still be referring to "if" statements and "for" loops long after no English speakers remain.
Or, in reverse; I knew what those words did in BASIC before I knew what they meant in English (which isn't my native language).
... will always will be English centric ...
I don't think you can claim that. Maybe for the next few decades or centuries but beyond that who knows what will happen. Practically every tome, text, blog or article of any
real value in the field is published in English ...
Is it possible that you just don't know they exist? You probably don't know enough to make such claims, as an English speaker myself, I know of a few books that I would want to read but they haven't been translated yet. There aren't many, but then again I don't know about others because of me not knowing the language.I would agree that a good coder must be able to express clearly and unambiguously complex processes in mother tongue and his programming language of choice but excellent grasp of English is only a "nice to have", when your local community is developed enough.
As for breaking services - you could make these JSON parameters optional and very clear that the documentation won't affect the way the service runs.
If you have documented the API sufficiently, then it's not a bad tactic to require clients "not to be silly". Clients who misuse APIs can't complain if they don't use them properly.
{
"// compiled_property" : "do something special mr. proprietary compiler"
"compiled_property": "magic"
}And what if you want multiple comments in the same scope (say, a comment before each attribute)? From the RFC: "The names within an object SHOULD be unique." In practice parsers will just override values, as JS does, but I think it would be legal to reject JSON with duplicate keys.
> "but I think it would be legal to reject JSON with duplicate keys."
No it would not. `SHOULD` has a very specific meaning:
SHOULD This word, or the adjective "RECOMMENDED", mean that there
may exist valid reasons in particular circumstances to ignore a
particular item, but the full implications must be understood and
carefully weighed before choosing a different course.
See http://www.ietf.org/rfc/rfc2119.txtAn object with duplicate keys must be parsed, but the exactly result will be implementation specific (undefined).
JSON isn't a great config file format, if your config file is meant to be human-edited. But the solution is simple; use a format designed for human editing.
Config files need UI design too.
During Akka and Play 2.0 development, we approached this by starting from the hand-rolled parsers found in Akka 1.2 and Play 1.2 (both had ad-hoc parsers to support a "pretty" config file format). We took the aesthetic preferences of the hand-rolled parsers seriously, and came up with HOCON: https://github.com/typesafehub/config (scroll down to "JSON Superset"), https://github.com/typesafehub/config/blob/master/HOCON.md
The HOCON format is a superset of JSON and also happens to be mostly compatible with the Play and Akka 1.2 ad hoc parsers (which were independently developed, so two data points on what people wanted).
HOCON is roughly similar to YAML in complexity. Like YAML the spec is pretty long ( http://www.yaml.org/spec/1.2/spec.html ). And the Typesafe Config and SnakeYAML jars have pretty comparable amounts of bytecode in them.
HOCON or YAML would be terrible formats for an API or something like that, and they're painful to implement, but I strongly prefer them to JSON for human-maintained configuration.
Anyway, I think the genius of JSON was its focus on machine interoperability rather than trying to do everything, that's why it works so well for machine interoperability. But it doesn't mean you have to use it for everything.
"JSON (JavaScript Object Notation) is a lightweight data-interchange format. It is easy for humans to read and write. It is easy for machines to parse and generate."
Humans are mentioned before machines. Comments improve human readability. Furthermore, comments are very simple to implement, and don't lower the ease of parsing.
It's supposed to be human readable, but ultimately it's for machines. It's just easier to debug.
1. Machine parsing/generation
2. Human parsing/generation
3. Ability to transmit it in a parsable format through HTTP
#1 conflicts with both #2 and #3. It conflicts with #1 because binary data formats are much easier to parse, and are generally smaller on the wire than plain text formats. Also, it conflicts with #3 for the same reason, since HTTP is a plain text format, and JavaScript generally deals with plain text better than binary data.
#2 conflicts with #3, because comments are nice for human readability, but should never be sent on the wire, because they are not necessary to parse the sent data. If your comment is important, it should be a data element that's sent on the wire.
The end result is a compromise. Since two of the concerns effectively make comments useless, the compromise should be towards those two. In addition, the reason given by Crockford makes quite a bit of sense. JSON, much like XML, is a format for the transmission of data. Everything in a given JSON object should be important. It's not a programming language, which means that control directives are unnecessary.
(1) I don't like comments, but removing them won't solve anything: People who like comments can just embed them as funky properties.
(2) I like comments, but embedding them as funky properties is annoying and awkward, and I don't want to.
Now look at (1).
Now look at (2).
Now relax your eyes like you're looking at a stereoscopic poster until (1) and (2) merge together and you achieve a Zen-like understanding -- or a view of a schooner.
What could possibly make somebody want to do that? Are there any examples around of people doing that?
So this wouldn't surprise me one bit. Would be interesting to see some real-world examples though.
And people do all kinds of nonsense with HTML comments. A very bad idea is often the fastest to implement.
(Better yet is to create an explicit place for metadata. I almost reflexively use {"metadata": null, "payload": ...} now instead of putting my payload right at the top level, because wouldn't you know it, sooner or later some metadata always seems to wander in. And if it doesn't... shrug. If you're in a place where you can afford JSON in the first place the extra cost is probably way below the noise threshold for you.)
If you do that with the metadata explicitly stored as a separate key-value pair in your blob, then this doesn't happen; the meta data is never silently discarded when you, say, take all the key value pairs in the JSON blob and send them out down the wire to a client. If you want to strip the meta-data, you have to do that.
I know this isn't Python, but I think the Zen of Python is on point here: "Explicit is better than implicit."
If you've stored comments as regular data, you haven't lost them but you've just transformed them in the output.
TANSTAAFL.
I'd submit that an incomplete understanding of the source data is not necessarily a problem. It's often a design goal. Generic tools have a limited understanding of the source data by design. I don't want my JSON parser/formatter/minifier/etc. to know about some silly parsing rules you added as comments. I want my JSON parser to understand JSON as it's defined.
Your nonstandard comment directives are the problem, not the fact that I didn't write a custom tool.
I quickly get visions of version numbers, customized namespace declarations, typedefs, strftime date format strings...
https://developers.google.com/closure/compiler/docs/js-for-c...
{
/* if IE */
browser: "IE"
/* else */
browser: "standard"
/* endif */
}
Pretty terrible and still possible (but admittedly harder) without comments.Then again, we have certain types of logic stored in a database table, loaded through fixtures... so my two cents may be worth much less than what they appear.
1. Internet Explorer has conditional comments - http://msdn.microsoft.com/en-us/library/ms537512%28v=vs.85%2...
2. Sprockets' directive processor uses directives in comments. https://github.com/sstephenson/sprockets
Also, Emacs uses comments to set file-local options. There's a long tradition of overloading comments to achieve metalinguistic ends. JavaDoc and Doxygen are great examples.
Even when handed a decent macro language with whizzy namespaces and a great DOM, I imagine that some people will still stoop to gross and convenient hacks.
/* jslint nomen: true, debug: true, evil: false, vars: true */
Adding that capability would raise the complexity of the parser, because comments would have to be made part of the data structure that is built and transformed. For instance, it would be harder to embed the data structure for the JSON in JS objects.
But yes, for a parser that's ingesting data for immediate processing and has no need for comments, there's no discernable win regarding parsing simplicity.
Not to mention N unit tests to cover comment removal verification.
"One interesting story about leaving things out: as we got closer to releasing JSON I decided to take out the ability to do comments. When translating JSON into other languages, often times the commenting piece was the most complicated part. By taking the commenting out we reduced the complexity of the parsers by half—everything else was just too simple."
This more or less prevent anyone from using JSON for configuration. Unless, of course, you parse it with eval.
Using properties as comments is awkward at best: it doesn't match what people are used to, either the key or value ends up being a dummy, and consumers that iterate over properties have to be smart enough to ignore special doc properties (which can make automatic validation against a schema more difficult). The solution of running it through JSMin is pretty unsatisfying too. It's never wrong, so how about we always do that?
If it's related to configuration then there should be documentation regarding what is and isn't supported. If it's simply data you're sending across the wire then there should be documentation somewhere; you wouldn't want to waste bandwidth transmitting comments.
Note in this case, nothing would be transmitted over the wire. JSON is just being used as a convenient format. Even still, it can be handy to exchange annotated data over the wire.
What happens when you accidentally delete a value you didn't mean to? You would refer to documentation. Honestly all configuration files _should_ be well documented, including postgres so I don't see the issue here. Why would you want to include every single possible option, probably commented out, when you could simply grab the ones you need from the documentation? I would prefer having a lean configuration file that shows exactly what I'm using rather than 400 commented options; that's a MESS to maintain.
>Even still, it can be handy to exchange annotated data over the wire.
I cannot disagree more. It is not useful AT ALL to exchange annotated data. Unless you're using an ad-hoc system (which obviously wouldn't scale), you're going to be handling this through APIs and services that already know and understand the JSON file format. So the only purpose of comments at this stage in the game is using up extra bandwidth.
Right, but what if you want to document why you personally are using a particular option, rather than just what an option does? I agree that a lot of the time config comments are unnecessary, but they do have a place.
I'm a bit puzzled by documentation seems like a bad word around here; it has its place and I'm not suggesting writing up a huge document. It takes only a couple of minutes to drop some text in Word / Google Docs / Wiki / your flavor and put it someplace accessible for your entire team.
In my experience, comments are less likely to go stale than wiki docs.
Re: transmitting annotated data. All I can say is not every application of JSON is for APIs for mobile devices. When I have a connected gigabit network and gzip data, I may not be that concerned about an extra 50 bytes of annotated data. But, again, that even presupposes that its only application is for computer-to-computer communication. The value in comments are human-to-human.
JSON comments would be a short-cut, sure, but that information should be within reach and should NOT only exist within the JSON anyway. So the helpfulness seems really limiting here.
>Re: transmitting annotated data. All I can say is not every application of JSON is for APIs for mobile devices. When I have a connected gigabit network and gzip data, I may not be that concerned about an extra 50 bytes of annotated data. But, again, that even presupposes that its only application is for computer-to-computer communication. The value in comments are human-to-human.
You're right in that the value for comments are human to human; I just cannot picture a scenario where you're actually transmitting data over the wire and including comments. I am not limiting this to mobile device; any service end point should ignore any comments and they will never be seen anyway.
As I mentioned before, if you're ingesting files in a very ad-hoc manner then of course comments could be useful but that's not a typical use-case of JSON. JSON is typically used as an interchange format for end point to end point communication and comments in ANY type of file in that scenario are useless.
How does that follow? Usually "data", 5.23423, needs commenting more than most. What the hell is that? Why 5 decimal places?
Also comments in config files are for many things; when file was created, change history, by whom, who to contact with problems, warnings not to edit as it's managed via chef/puppet.
You would already know the answer before seeing the JSON file so I'm not sure why you care.
For instance, you're not going to be receiving JSON data over email and then putting it into a system manually. Instead, you'll have APIs that handle the JSON formats for you and simply ingest the data.
If you're processing a large volume of data using JSON as the interchange format, why on Earth would you want it to include comments? No service on Earth does this that has any volume of users.
>Also comments in config files are for many things; when file was created, change history, by whom, who to contact with problems, warnings not to edit as it's managed via chef/puppet.
This is not the job of a comment. These go stale and all are available via whatever version control mechanism. However, keep in mind you're talking a very specific edge case in a development environment. Typically these don't matter, at all. If you need comments within the dev system for whatever reason, you can just strip them out. Puppet would obviously have appropriate permissions so no one can simply modify them anyway without knowing they're messing with puppet.
Shipping items, however, should simply have documentation regarding what configurations you product does or does not support.
Your comment about Puppet is especially troubling. Puppet doesn't "have appropriate permissions", it's a root-level tool for managing system state. The files it manages may have all kinds of ownership and permissions, some of which are not ones you can just mess with (e.g. the system will throw up its hands and bomb out if they're wrong), and all of which are still modifiable by anyone with root access. There's no way to differentiate short of comments in the files themselves.
I was referring to JSON as an interchange format, which is what it is.
Arguably Amazon should have chosen something different for templates but I think that's an issue beyond simple comments and outside the scope of the argument. Regardless, it's hard to image anyone thinking it's a good idea to send comments over the wire especially if the end point is some sort of service.
This is highly arrogant comment. There was no need to be rude. I've been developing and working with actual data for over a decade now. Yes, in the real world sometimes comments are transmitted over the line but I'm sure you can agree that isn't a good idea. Yes, many times you don't get good documentation but you make it sound like that's acceptable and anyone who thinks otherwise doesn't have real-world experience.
In all honestly I would expect comments generated from some odd software packages but it's been an incredibly long time since I've seen data transmissions that contain comments. In environments dealing with petabytes of data you can't afford to send comments with every single file.
As for Puppet, I think you misunderstand my point. Yes, it is a root-level tool. That doesn't mean any user should have the same permissions as Puppet. Why wouldn't you simply place configuration files people shouldn't modify in places where they don't have permission but Puppet does? Honestly, I thought that was standard practice.
I never stated such a thing but that is what it's designed for and the primary use-case for JSON. I would imagine most other cases are edge-cases.
>and the files puppet manages cannot be put off to one side in some magical place where root cannot access them.
No one said to push them off to the "side" or into some "magical place". It's really simple: puppet has permissions to the files, your users do not.
A system administrator worth his salt isn't going to be messing with configuration files any which way and if said administrator has ROOT access then they should already know that's the level where puppet works and they could screw something up.
So I'm not really sure what your point was...
You asked nothing relevant to the conversation. Instead you asked about me ever working in a production environment and nothing more.
That questions accomplishes nothing but being insulting. I hate to sound crass but you're acting like an ass.
If you wish to bring something topical and relevant to the conversation then by all means do so but insulting someone gets nothing accomplished other than trolling...in which case you already won. Congrats.
>The point being that the things you sum up sound very much like textbook-knowledge and theory
No, they don't. Everything I stated is from real-world experience. I cannot fathom what kind of environment you work in where service to service communication includes comments going over the wire. But, as you can see above in my other responses, I've already covered all of these aspects. Nothing theoretical.
Problem at line 1 character 25: Expected exactly one space between 'function' and '('.
var crockford = function(){ while(1); }
Problem at line 1 character 27: Expected exactly one space between ')' and '{'.
var crockford = function(){ while(1); }
Problem at line 1 character 27: Missing space between ')' and '{'.
var crockford = function(){ while(1); }
Problem at line 1 character 29: Missing 'use strict' statement.
var crockford = function(){ while(1); }
Problem at line 1 character 34: Expected exactly one space between 'while' and '('.
var crockford = function(){ while(1); }
Problem at line 1 character 35: Unexpected '1'.
var crockford = function(){ while(1); }
Problem at line 1 character 37: Expected exactly one space between ')' and ';'.
var crockford = function(){ while(1); }
Problem at line 1 character 37: Missing space between ')' and ';'.
var crockford = function(){ while(1); }
Problem at line 1 character 37: Expected '{' and instead saw ';'.
var crockford = function(){ while(1); }
Problem at line 1 character 37: Unexpected ';'.
var crockford = function(){ while(1); }
Problem at line 1 character 39: Cannot convert 'array[0]' to object
Problem at line 1 character 39: Cannot read property 'disrupt' of undefined
Huh?But it is easy enough to pipe the JSON thru a filter...
- # in column 1
- # in column 1, backslash at end of line escapes the newline
- # as first non-blank
- # as first non-blank, backslash at end of line escapes the newline
- -- as first non-blank
- C-style multi-line slash-asterisk comment asterisk-slash
- C++-style //
- Python style doc strings
etc.
Then somebody will attempt to write the comment filter that will handle them all, somebody else will make a JSON parser that round-trips all different versions. Net result: a JSON 'standard' that is ugly and leads to parsers that are larger then necessary, not 100% reliable (how do you guess whether a backslash at the end of line escapes the newline?)I would rather have a simple standard with one kind of comment.