That was definitely not my take-away from the article. More like "JSON is not a data format you can rely on blindly if you are using an esoteric edge-case and/or an alpha-stage parsing library." I haven't ever run into a single JSON issue that wasn't due to my own fat fingers or trying to serialize data that would have been better suited to something like BSON.
I'm not saying JSON is bad. Personally I think the sloppiness that this post enumerates is part of its success, and part of why it has so much support in so many languages. The more you nail down the semantics, the harder such widespread support gets. Right now you get a nice subset of the JSON spec that works in a huge variety of languages, where the JSON support for a given language usually converts the JSON into something fairly native for the language. This wouldn't be possible if you nailed down the semantics as hard as something like Protobuf does. For instance, a ton of statically-typed languages will let you encode and decode integers into JSON. But JSON doesn't have an integer type. Strict JSON support ought to forbid integers. But it's sloppiness lets us all just sort of ignore that and get on with life.
If you have a need for precision, avoid JSON. And people probably generally need more precision than they realize and probably ought to reach for JSON a bit less often than they do. But on the other hand, the whole thing does mostly work, right? That can't be ignored.
JSON is agnostic about numbers. ... JSON instead offers
only the representation of numbers that humans use: a
sequence of digits
This mostly causes issues for code that needs to be data-structure-agnostic (in the Go case: decoding into a interface{}, rather than a struct with "json:" tags), but there are edges cases where you can get a surprise. For example, since all numbers are technically both integers and floats, something like {"visitorCount": 42.0} is perfectly valid, and a client has to know to coerce the number into an integer if it wants to deal with it sanely, even though the meaning of that number might be nonsensical if treated as a float.
[1] http://www.ecma-international.org/publications/files/ECMA-ST...
IMO this wasn't originally a bad thing at all. There are so many different types of numbers, with so many different behaviours (does `1` == `1.0`? Not in statistics class) that trying to work it all out in JSON would have been a fools errand. The problem is that so many JSON parsers are overly aggressive about turning JSON Numbers into something specific (usually floats) that when writing a JSON API we have to assume it will be consumed in this lowest-common-denomitor way.
By the way, I don't think that all JSON numbers are technically both integers and floats. `42` and `42.0` are clearly different according to the spec, IMO. I'd be curious to get your thoughts on this.
Ruby, IMHO, errs on the side of consistency (fractionless numbers become Fixnum, so encode(decode("1")) => "1"), whereas Go goes the opposite way (all numbers become float64, so encode(decode("1")) => "1.0", unless you enable the magic json.Number type).
JavaScript is an interesting scenario, because JS doesn't even have integer numbers, which means it doesn't have any choice in the matter. Interestingly, Node.js/V8 truncates the fraction if it can: JSON.dump(JSON.parse("1.0")) => "1".
It's a mess.
data Sign = Positive | Negative
data Digit = Zero | One ... Eight | Nine
data JSONNumber = JSONNumber
{ _sign :: Maybe Sign
, _integerPart :: [Digit]
, _fractionalPart :: Maybe [Digit]
}
Of course, users of the parsers could ask to have a number converted to a float, but that wouldn't be the default.
We're a long way from that though. At the moment I don't even feel like I can write a protocol that expected implementations to distinguish `1` and `1.0`. This should not be the case.
Indeed you can't, among other reasons because that distinction doesn't exist in JavaScript, but ... why would you want to make such a subtle distinction?
If you really want what you described above, you can get it by using a string.
Well, no. They're numbers as humans think about them.
(Cue the saying about programming being a job where you hate yourself for not thinking enough like a computer)
I should have said "JSON number's aren't machine numbers" (e.g. float, double, integer). But serialized, human-readable numbers are still numbers, so I was wrong.
With Unicode string handling I already wouldn't expect that, not to mention whitespace. However, I would expect that dec(enc(dec(string))) === dec(string)
> {"visitorCount": 42.0} is perfectly valid
The problem here is not necessarily that the standard is informal, but that languages differ in the number types they offer. Treating "42.0" as something different from "42" is after all just another convention and not a universal one. In other words, having to coerce something to an integer is more a property of the environment than a problem with JSON.
I suspect that when this problem does occur, it goes undetected for longer than it should. In the remainder of this article, we examine potential improvements to our handling of long."
https://www.techempower.com/blog/2016/07/05/mangling-json-nu...
This gets even worse if your software is an integration layer between two services you do not control.
The best you can do is:
- interpret the spec to the letter.
- for every fragment of a statement you write, consider whether it might conceivably go wrong, and handle those cases (in the simplest matter because 'handling' means writing code, and that code, too, needs to go through this process).
For example, a json parser must be prepared to handle missing values, extremely long keys and values (integers may have thousands of digits, think long about the question whether 64 bits always is enough for storing a string length, etc.), etc.
- if you are truly paranoid, have very stringent security requirements, or expect to be heavily attacked, run the parser in a separate process.
- fuzz your implementation.
I'm struggling to think of any realistic scenario where this isn't true!
Let's do a quick estimate: one can read in the order of 2^24 bytes/second from disk. A day has on the order of 2^16 seconds, so that's 2^40 bytes/day.
=> You will need 2^24 days to read 2^64 bytes. I think that's around 50k years. That an attacker will try to generate a buffer overflow this way is a risk I would take, even if I thought the hardware had room to store that string.
The only way I can foresee a real risk is when an optimizer can optimize away the computation of a string whose length it is asked to compute by an attacker.
That's still very much far-fetched, and if no string gets allocated it's hard to see how it could become a security issue, but it could be a reason to be extra careful, for example when providing online access to a C++ compiler, with its template metaprogramming capabilities.
Seems like an odd thing to say in general. Perhaps "JSON parsers are not libraries you can rely on blindly", but that seems true of everything.