Comment by niftich | Hacker News Reader

Comment by niftich | Hacker News Reader

The github for this project doesn't use this terminology, but the keyword here is 'canonicalization'. That's the process in which you transform a document which can take any number of forms to a specific document that only takes one form.

You do this to ensure that documents that contain the same "information" will look identical on the wire, not just after-the-fact once you've processed them into some execution-specific data structure.

For example, X.509 certificates use ASN.1 DER, a restricted subset of ASN.1 BER where each value takes a deterministic form, so that two certificates that contain the same information will look identical when serialized into bytes. This is a stricter application of ASN.1, while, say, when you talk to a directory service over LDAP, you can speak BER, the looser encoding, because the exact bytes by which you're making yourself understood don't really matter.

Son is a canonicalized form of JSON, and the author maintains a page of JSON subsets and supersets [1] that lists two other canonicalized encodings, both of them containing silly flaws in their design. Son is a better effort.

[1] https://housejeffries.com/page/7

A stricter subset of JSON doesn't interest me without more specification surrounding how numerics should be decoded.

How do I decode 18446744073709551615? What is the behaviour of a decoder that can't handle numeric types of that size? Should it cause a parse error? Should it truncate to 1.84467e+19? Should it overflow to 4294967295?

Presumably if you want a canonical textual representation, the parse error is the only acceptable solution.

> What is the behaviour of a decoder that can't handle numeric types of that size?

Is this actually relevant? It sounds like an implementation detail to me and it seems to boil down to using arbitrary size bigint. If you can't handle a bigint because it's too big for your machine, it sounds like a problem with the system, not parser.

We can play this game all day. What if I pass an integer that takes 32GiB of memory to store? Also the system's fault if the resources aren't available?

I'd say so. This is kind of a security question - the data is valid from the protocol's perspective, it's the contents that are malicious. It's up to the developer to decide how to handle that - if it's super important to be able to process such big numbers, modify the parser to handle that (for example by storing them on the HDD). Same goes for another serialization format: bencoding. It supports arbitrarily nested data structures and most implementations are recursive, which can easily result in stack overflow if you mess up the implementation. But I wouldn't blame the standard - that's the price you pay for flexibility. At the end of the day, you need to sanitize your input anyway and keep your edge cases in mind. Same goes with XML bombs...

I would think the JS part of JSON would preclude any massive integers. It’s still an IEEE floating point at the end of the day, so your max is (2^53)-1

If you managed to load a JSON string into memory that contains a number that takes 32GiB of memory to store after parsing, I don't think you're worrying much about resources.

The behaviour of the encoder and decoder are just as important to specify as the bytes across the wire. This is the whole problem with JSON in the first place.

Decode it as a number with exactly that value as the numeric type the application requests. If the number is out of range, indicate that to the application (which should already have logic to handle numbers out of the range it cares about).

If the application doesn't request that you decide it, don't. Leave it in some internal format which does not lose precision. Limit yourself only to memory limits the application requests.

It's no different than decoding a string...

That's great, but most JSON libraries don't give you that level of access... they just decode in to a bunch of nested maps/dicts consisting of your languages native data types. So one day you swap out your JSON library and the behaviour of your application silently changes.

Shouldn't they return a list of warnings/errors on the parse? Just blindly accepting a machine parse of some random input data is how you crash programs with bad input.

Use a better JSON library :)

(There are competent JSON implementations, though they're few and far between. PostgreSQL's is the only good example I can point to off the top of my head.)

Here’s what Rust’s `json` crate does for the `Number` type: https://docs.rs/json/0.11.12/json/number/struct.Number.html

It stores it as positive, mantissa and exponent, allowing you to pull it apart in that way if you need to. Meanwhile, normal usage is via JsonValue (https://docs.rs/json/0.11.12/json/enum.JsonValue.html) which allows you to attempt to convert it into an f64, i32, u8, &c. as you desire, returning `None` if it doesn’t fit inside that type (e.g. -7 won’t go into an i32, 576 won’t go into a u8).

By what’s being talked about here, that looks to be a competent JSON implementation.

Doesn't most of the Rust ecosystem use serde though?

It looks like serde-json does the same: https://docs.serde.rs/serde_json/struct.Number.html

Well, not the same; it eagerly interprets it as PosInt(u64), NegInt(i64) or Float(f64) rather than storing the mantissa and exponent. This is enough for JavaScript compatibility (its Number type being essentially a 64-bit float), but it’s not quite as flexible as the json crate’s approach.

Wonderful. Good to know that about Rust, I haven't used it but everything I read about it looks good. Yes that is exactly the approach I meant (and a similar mechanism to that used by my own C++ JSON library I use for side projects).

This is what unit testing is for. Test the methods that validate and return your expected object structures. This catches other things like a structure changing (as simple as a property being modified), ensuring you can handle malformed input (even if it's valid JSON) and any other data validation rules you have.

For context, let's start with what JSON is. All the JSON spec really lets you do is say "this sequence of code points is JSON", or "this sequence isn't". It includes a few interoperability suggestions for how you might avoid JSON that will be hard to interpret, but it's extremely stark on parsing and generation.

For a pretty epic summary of this, see this Reddit comment: https://www.reddit.com/r/programming/comments/59htn7/parsing...

You have a comment downthread about how inadequate this is, which I totally agree with:

> The behaviour of the encoder and decoder are just as important to specify as the bytes across the wire. This is the whole problem with JSON in the first place.

Unfortunately since the goal of Son is to be a subset of JSON, there's not much we can do about encoding/decoding within Son's spec. Instead I hope to build a sane foundation on which other tools can be built. Limiting the scope of Son seems like a good way to go about this. In this case, that means not specifying ranges for numbers, so that future tools such as schemas have as much flexibility as possible (for instance to include arbitrary precision numbers).

EDIT: I'm mixing together two different things in my last paragraph. I don't want Son to know about Floats, Doubles, etc since JSON doesn't. That doesn't mean that we couldn't specify a range on number size though. The reason that is left out is to allow maximum flexibility for other tools to build on Son -- I'm trying to drop extraneous parts of JSON, but as few useful parts as possible. This needs more elaboration within the project's docs.

"In this case, that means not specifying ranges for numbers, so that future tools such as schemas have as much flexibility as possible (for instance to include arbitrary precision numbers)."

If the allowed range of numbers isn't specified, then when you emit data in this format, you can't be sure it will be read correctly on the other end...

This is definitely a problem, but the place to solve it isn't Son.

Son is intended to be a very simple project that grabs all the clear wins. Do we really need both `0` and `-0`, stuff like that.

There are too many different things JSON Numbers can represent for us to have a clear strategy for all them in Son. Int64s, doubles, floats, arbitrary precision numbers (eg https://hackage.haskell.org/package/scientific-0.3.5.2), etc. That makes this a good for for the schema layer (eg JSON Schema or whatever you're using), not the base specification layer.

> Do we really need both `0` and `-0`, stuff like that.

Well, yeah since signed zeros are a thing.

Chrome:

> JSON.parse('-0.0') -0

> JSON.parse('0.0') 0

python:

In [3]: json.loads('-0.0')

Out[3]: -0.0

In [4]: json.loads('0.0')

Out[4]: 0.0

> This is definitely a problem, but the place to solve it isn't Son.

But then what problem does Son actually solve? How is a canonicalisation format that's not actually canonical useful?

You take a hit (e.g. by forgoing fast existing libraries, control over pretty printing, and by being forced to sort on serialization which screws up streaming), but gain essentially nothing, not even the advertised benefit of your stuff not randomly changing as it moves through the stack. The likely main benefit of canonicalization is for security and crypto, but unless you use son without numbers (or create your own subset of son) it's kinda useless for that. Of course you can just encode all your numbers in strings or something, but at the point where you're doing your own number parsing logic (which is the hardest bit), why not just use some well-designed actually canonical format like csexp?

This isn't really a true canonicalization since it doesn't define an ordering. I was actually hoping it was. The matrix project [0] invented their own json canonicalization so that they could have servers meaningfully sign it. Was hoping some external standard of json canonicalization existed I could recommend them switching to.

[0] https://matrix.org

You could argue that Matrix’s canonical JSON form /is/ an open external standard ;) Meanwhile, Mastodon looks to have been implementing its own canonical JSON form recently too for signing purposes.

What do you mean by ordering? This is easy to miss in the linked "spec" (I did too at first) but there is a sentence:

"Object members must be sorted by ascending lexicographic order of their keys."

They need to tighten that a bit, though. “Lexicographic” isn’t clear enough even ignoring locale dependencies.

Looking at the reference implementation, it contains

  encode = encodeUtf8 . generate

So, the sorting is done before conversion to UTF-8.

Relevant line, I think is

  genObject hm = "{" <> foldl' go mempty (sortOn fst (HM.toList hm)) <> "}"

So, it uses Data.String’s sort order, which seems to be to lexicographic by Unicode code point (https://stackoverflow.com/a/3126287)

⇒ implementations cannot sort the UTF-8 byte st sequences lexicographically. I think that’s a bad choice (if it was a choice and not an oversight)

I’ve never written a single line of Haskell, so corrections welcome.

Have you seen the relevant reference in RFC3629, page 2? It's explicitly listed there as a feature: "The byte-value lexicographic sorting order of UTF-8 strings is the same as if ordered by character numbers"

Agree that specifying the keys to be ordered >by unicode code points< instead of >lexicographically< would be less ambiguous though.

I definitely meant ordering by Unicode code points. Someone very helpfully opened an issue and we're trying to figure out the right wording there: https://github.com/seagreen/Son/issues/13

Looks good. I still foresee interoperability problems between implementations, though. It just is too easy to mix up the ‘sort by key’ and ‘escape various control characters’ steps (CR sorts before ascii characters, but “\n” sorts after it)

Even if the spec requires it, I fear implementations will also canonicalize strings differently, breaking sort order.

Actually, it does define an ordering:

> Object members must be sorted by ascending lexicographic order of their keys.

It also specifies that object keys must be unique, which is another canonicalization issue with JSON.

yep. I completely missed that. My bad.

Reply on news.ycombinator.com