Hacker News Reader

> This problem happened with Matrix when they started having to layer JSON around base64-encoded-encrypted-JSON and generally the looseness of the format infecting the entire data model of the protocol.

Indeed, this has already created considerable problems for the request and event signing and getting different Matrix implementations written in different languages to interoperate correctly. I now firmly believe that cryptographically-signed JSON is always a disaster waiting to happen, no matter how hard you try to canonicalise it.

Well, it's a webapp, so everything there is designed with web technologies.

Yes, for all the great things browsers do, it sure is a shame that the only data format they can handle is JSON.

But just because it's a webapp doesn't mean you have to use JSON for everything, it's the other way around, using JSON for everything means that the only place the data model feels natural is the web.

Which is strange because JavaScript has had the Uint8Array that makes manipulating arbitrary byte structures easy for a while, it doesn't feel any less natural than other dynamic languages in that regard. Defining a "signed message with a header" structure in terms of bytes like I described above seems like a no-brainer. If you want the payload data of your protocol to use JSON then that's fine, but that shouldn't dictate the cryptographic layers underneath.

As others have said, there are unfortunately not that many good alternatives for serializing data in web browsers (that are both performant and don't require bulking your app with the serialization implementation).

We've done our best to make sure that the way we're serializing objects can either be done in a consistent manner across implementations or, if done inconsistently, it doesn't matter.

I don't really buy the argument that additional serialization logic adds bloat to the application when you compare it to the many thousands of lines of code in modern JS frameworks for manipulating DOM elements.

The consistency is really an important part here, are you spending less time doing the testing and bugfixing to ensure that JSON ambiguity is not possible / is harmless than you would writing serialization for cryptographic types in unambiguous byte-oriented structures?

SPMessage serializations are never regenerated after they leave the client, so it's not an issue as far as I'm aware. JSON is supported universally in all languages and all platforms, and natively in the browser. Is there a specific serialization format you're advocating for?

I suspect it's more that JSON is (rather explicitly) not designed to be stable across all implementations, so there are subtle differences. Numbers don't have a specified precision, strings don't have a specified encoding, maps have no canonical order nor do they require unique keys[1], etc.

Nailing all that down so you can e.g. have a truly deterministic signing process often requires custom encoders/decoders and is rather fraught with problems even then, at which point why even bother with JSON. Use something designed to be consistent.

[1]: https://stackoverflow.com/a/23195243

We're aware of these issues with JSON, but they do not affect the integrity of Shelter Protocol, and by using JSON we get a lot of advantages in terms of speed and developer usability (support for tools like `jq`, schema-less serialization, widespread language support, easier debugging, etc.).

I was really excited when I tried out the scuttlebutt protocol a few years back. Distributed, peer-to-peer, offline-enabled social networking seemed neat.

But a healthy ecosystem of clients never emerged. When I looked into why, I discovered it relied on node's particular method of JSON serialization[0]. Are you confident that the shelter protocol has avoided this pitfall?

[0]: https://news.ycombinator.com/item?id=22910752

Yes, I am confident that we have avoided that specific pitfall because as I mentioned we do not ever re-serialize the JSON.

However, thinking on it more, there is one thing Groxx mentioned that has given me pause, and that is "Numbers don't have a specified precision". This could be an issue in the future if, for example, computers suddenly use 128-bit numbers and JSON implementations start to actually use such numbers. It's an edge case that might result in incompatibility with clients that use 64-bit numbers. So we'll consider the possibility of changing the serialization format after researching this more.

> So we'll consider the possibility of changing the serialization format after researching this more.

I asked Brendan Eich if JavaScript will ever change the way it represents floating point numbers (from 64-bit IEEE 754 values), and he said no ("don't break the Web"). And to represent larger bit values in JSON you're supposed to use a JSON string and schema to specify the type. I'm not sure exactly what he means by that, but I assume it means something like using JSON keys like "<BigInt>mykey" and string values instead of number values. Then manually parsing the string into whatever native 128-bit type or whatever representation you have.

So - JSON still seems to be the favorite here, but I totally get the anxiety around it because of how under-specified it is. If you avoid re-serialization nothing should break.

Avoiding re-serialization does avoid a ton of issues, yea. Though I would suggest building in validation for e.g. no duplicate keys immediately, because unexpected duplicates are a common source of bugs and security exploits (e.g. it's a fairly common issue with http headers when requests cross multiple systems).

After evaluating the various options, we seem to be settling on the idea that the protocol will specify that for duplicate keys, the handling must be the same as in browsers: meaning, the last key is the one that's used.

Interesting remark! Genuine question here: by what would you replace JSON if you can re-architecture it?

I wish there was some language-agnostic tool for defining byte-oriented data strucrures that could emit code to manipulate them in different languages. Protobuf and similar tools exist but they still have their own weirdnesses and aren't entirely zero-overhead. I've seen plenty of pseudo-schemas used in standards documents, in particular the MLS spec which I think was finalized recently.

Something similar to this?

https://bnfc.digitalgrammars.com/

I agree, having/using a well-defined standard that can have automated tooling to general language-specific implementations would be ideal.

How about bencode? c-sexp? netstrings? The important features I would demand are: able to represent the types you need (e.g. integers instead of strings of digits or floats) and designed to have a canonical presentation from the beginning. There are so many bugs around ASN.1/XML/JSON parsing and "normalisation" that it's simply inexcusable to repeat this mistake. If the designers didn't come up with a really convincing explanation for selecting the wrong serialisation format for a cryptographic protocol how likely is it they read and understood the prior art in other fields like cryptography? Just because a canonical JSON (de-)serialiser is only a npm/pip/gem install is no excuse to burden a specification with the bloat and attack surface. Even fix sized, flat, packed structs are a better solution. K.I.S.S.!