Hacker News Reader

Comment by lufte | Hacker News Reader

Parsing a CSV can be done in a line-per-line basis, while a an array of arrays in JSON is not valid until you reach the end.

How would any existing JSON parser handle 75GB of data in a single array of arrays?

View on HN

Many JSON Parsers already have an incremental forward read mode (think SAX-style parser if you are familiar with XML parser styles) where you can ask for each array inside that array of arrays one at a time as it reads them. If something unexpected happens at the end of the large outer array such as syntax error you can decide at that point if you rollback what you've already read/operated on or not.

JSON.Parse() in the browser isn't that sort of parser obviously, but there options in many languages such as both major .NET JSON libraries (Newtonsoft and System.Text.Json) have ways to parse that way. Similar examples exist in other languages too.

Interesting. I don't know of any similar option in Python's json module, which is the one that I mostly use, but good to know this exists.

Yes, Python's bundled json module does not support that style of parsing. I saw several suggestions in a quick search to try NAYA [1] if you need that in Python (it's a no dependency Python 3 library) or one of the C-backed wrapper libraries NAYA mentions at the bottom of its Readme if you can afford native dependencies.

[1] https://github.com/danielyule/naya

Reply on news.ycombinator.com