Hacker News Reader

Comment by johnrob | Hacker News Reader

I find JSON “array of arrays” to be a better solution while also remaining human-readable. Especially when formatted as one array per line, analogous to CSV.

View on HN

Right on. And with the inclusion of a JSON-schema header, it makes data parsing a breeze:

    { "$schema": "http://json-schema.org/draft-07/schema#", "type": "array", "items": {"type":"array","items": [ { "type": "number" }, { "type": "string" }, { "enum": ["Street", "Avenue", "Boulevard"] },{ "enum": ["NW", "NE", "SW", "SE"] } ]} }
    [[3,"some", "Street", "NE"],
    [4,"other", "Avenue", "SE"],
    [5,"some", "Boulevard", "SW"]]

With draft 2020-12, you can even have module-like based on [Compound Document spec](https://json-schema.org/draft/2020-12/json-schema-core.html#...)

Now you have tables and relation. Only few people understand the usefulness of Compound Document, currently.

Parsing a CSV can be done in a line-per-line basis, while a an array of arrays in JSON is not valid until you reach the end.

How would any existing JSON parser handle 75GB of data in a single array of arrays?

Many JSON Parsers already have an incremental forward read mode (think SAX-style parser if you are familiar with XML parser styles) where you can ask for each array inside that array of arrays one at a time as it reads them. If something unexpected happens at the end of the large outer array such as syntax error you can decide at that point if you rollback what you've already read/operated on or not.

JSON.Parse() in the browser isn't that sort of parser obviously, but there options in many languages such as both major .NET JSON libraries (Newtonsoft and System.Text.Json) have ways to parse that way. Similar examples exist in other languages too.

Interesting. I don't know of any similar option in Python's json module, which is the one that I mostly use, but good to know this exists.

Yes, Python's bundled json module does not support that style of parsing. I saw several suggestions in a quick search to try NAYA [1] if you need that in Python (it's a no dependency Python 3 library) or one of the C-backed wrapper libraries NAYA mentions at the bottom of its Readme if you can afford native dependencies.

[1] https://github.com/danielyule/naya

Reply on news.ycombinator.com