Comment by jerf | Hacker News Reader

Comment by jerf | Hacker News Reader

Parent Full thread

The sum totality of all the issues raised in that post is not an esoteric edge case, even if each individual element is an esoteric edge case. If you haven't encountered any of them in your real code yet, there's two basic possibilities. Either you aren't using JSON very hard at all... or you have encountered them and you just didn't realize it. You will, sooner or later.

I'm not saying JSON is bad. Personally I think the sloppiness that this post enumerates is part of its success, and part of why it has so much support in so many languages. The more you nail down the semantics, the harder such widespread support gets. Right now you get a nice subset of the JSON spec that works in a huge variety of languages, where the JSON support for a given language usually converts the JSON into something fairly native for the language. This wouldn't be possible if you nailed down the semantics as hard as something like Protobuf does. For instance, a ton of statically-typed languages will let you encode and decode integers into JSON. But JSON doesn't have an integer type. Strict JSON support ought to forbid integers. But it's sloppiness lets us all just sort of ignore that and get on with life.

If you have a need for precision, avoid JSON. And people probably generally need more precision than they realize and probably ought to reach for JSON a bit less often than they do. But on the other hand, the whole thing does mostly work, right? That can't be ignored.

JSON doesn't have an integer type, but it certainly supports integers. Within, obviously, implementation defined limits. I'm with you up to "if you need precision, avoid JSON". Actually JSON is fine for the kinds of precision most use cases require, and when it isn't, you probably know it.

JSON's number support is a source of problems, because the standard is so informal [1]:

    JSON is agnostic about numbers. ... JSON instead offers
    only the representation of numbers that humans use: a
    sequence of digits

This poses a problem for some languages, and tends to break things. You would expect encode(decode(string)) == string, but languages deal with numbers differently. For example, in Go, if you decode into a map[string]interface{}, you will get float64 by default. If you decode "42" and then encode it back to JSON, you'll get "42.0". (This is the reason Go has a special type you can use, json.Number, that preserves the value as its original string value.)

This mostly causes issues for code that needs to be data-structure-agnostic (in the Go case: decoding into a interface{}, rather than a struct with "json:" tags), but there are edges cases where you can get a surprise. For example, since all numbers are technically both integers and floats, something like {"visitorCount": 42.0} is perfectly valid, and a client has to know to coerce the number into an integer if it wants to deal with it sanely, even though the meaning of that number might be nonsensical if treated as a float.

[1] http://www.ecma-international.org/publications/files/ECMA-ST...

Exactly. JSON Number ARE NOT ACTUAL NUMBERS. They're really restricted strings (or, as you quote, a syntax for representing numbers).

IMO this wasn't originally a bad thing at all. There are so many different types of numbers, with so many different behaviours (does `1` == `1.0`? Not in statistics class) that trying to work it all out in JSON would have been a fools errand. The problem is that so many JSON parsers are overly aggressive about turning JSON Numbers into something specific (usually floats) that when writing a JSON API we have to assume it will be consumed in this lowest-common-denomitor way.

By the way, I don't think that all JSON numbers are technically both integers and floats. `42` and `42.0` are clearly different according to the spec, IMO. I'd be curious to get your thoughts on this.

The spec is too vague; it's "agnostic". A number can include a fractional part, but it doesn't say if that means that fractionless numbers are to be treated as integers. It leaves that to the implementation, which is a terrible idea for an interchange format.

Ruby, IMHO, errs on the side of consistency (fractionless numbers become Fixnum, so encode(decode("1")) => "1"), whereas Go goes the opposite way (all numbers become float64, so encode(decode("1")) => "1.0", unless you enable the magic json.Number type).

JavaScript is an interesting scenario, because JS doesn't even have integer numbers, which means it doesn't have any choice in the matter. Interestingly, Node.js/V8 truncates the fraction if it can: JSON.dump(JSON.parse("1.0")) => "1".

It's a mess.

Thanks for your analysis (and the Ruby and JS reports). The more I think about it the more I think parsers should not parse to numbers by default, but instead to (using a Haskell-based pseudo type):

  data Sign = Positive | Negative

  data Digit = Zero | One ... Eight | Nine

  data JSONNumber = JSONNumber
    { _sign           :: Maybe Sign
    , _integerPart    :: [Digit]
    , _fractionalPart :: Maybe [Digit]
    }

Actually this should include exponents as well -- basically parsers should just report exactly what the spec allows, without coercing to float or whatever.

Of course, users of the parsers could ask to have a number converted to a float, but that wouldn't be the default.

We're a long way from that though. At the moment I don't even feel like I can write a protocol that expected implementations to distinguish `1` and `1.0`. This should not be the case.

> At the moment I don't even feel like I can write a protocol that expected implementations to distinguish `1` and `1.0`.

Indeed you can't, among other reasons because that distinction doesn't exist in JavaScript, but ... why would you want to make such a subtle distinction?

If you really want what you described above, you can get it by using a string.

I respect the pragmatism of your reply. At the same time "distinguish decimals and integers" is about as far from a subtle distinction as you can get.

The distinction may be familiar to us, but if you ask someone on the street, "1" and "1.0" are the same number. The habit we have of giving them different types is somewhat arbitrary.

It sounds like what you're saying is that it should be deserialized as Java's BigDecimal, or whatever the equivalent of that is in a given/language framework; and if there isn't one, then the JSON parser should provide that equivalent.

> JSON Number ARE NOT ACTUAL NUMBERS.

Well, no. They're numbers as humans think about them.

(Cue the saying about programming being a job where you hate yourself for not thinking enough like a computer)

Almost two weeks late, but I thought about this more and I totally agree with you.

I should have said "JSON number's aren't machine numbers" (e.g. float, double, integer). But serialized, human-readable numbers are still numbers, so I was wrong.

> You would expect encode(decode(string)) == string

With Unicode string handling I already wouldn't expect that, not to mention whitespace. However, I would expect that dec(enc(dec(string))) === dec(string)

> {"visitorCount": 42.0} is perfectly valid

The problem here is not necessarily that the standard is informal, but that languages differ in the number types they offer. Treating "42.0" as something different from "42" is after all just another convention and not a universal one. In other words, having to coerce something to an integer is more a property of the environment than a problem with JSON.

"Perhaps all the numbers we use are safe. Are we actually mangling our numbers? Probably not...but will we know? Will anything blow up, or will our application be silently, subtly wrong?

I suspect that when this problem does occur, it goes undetected for longer than it should. In the remainder of this article, we examine potential improvements to our handling of long."

https://www.techempower.com/blog/2016/07/05/mangling-json-nu...

I like CBOR (http://cbor.io/). It's a standard (RFC-7049), it can encode 64-bit integers, IEEE floating point, text (UTF-8), binary data (anything non-UTF-8), booleans, null and undefined. You have arrays and maps (and maps can have any value as an index, not just strings) and you can semantically tag data as well. The RFC is one of the better RFCs I've read, covering all the details and plenty of encoding examples to test against.

Reply on news.ycombinator.com