Really? I just tried with a sample of one of my files, using python 3.6. With text in memory, 1M smallish strings decoding is 12s with ujson and 25s with the built in json. orjson and rapidjson seemed about the same for this test.
No, it's no.
>Clearly, no JSON parser provides a significant gain on parse speed of small JSON documents, when you define large as those where parse time is significant.
A rather bizarre deduction.
E.g. "define large as those where parse time is significant".
So let's say a large file is one whose parse time is > 10 seconds.
So, while looking at small, ie. < 10 second parse time files, if a JSON parser takes files that others load in 500ms and load them in 20ms, then that's "a significant gain on parse speed of small JSON documents"...
We didn't have to change any of the definitions...
I've been working with a 90gb JSON file recently. I'd probably start counting a file as large at around 100mb.
It's the perfect solution because you can use the same decoder + something like jq for debugging and manipulation on the command line.
It isn't quite "awk-ish", not quite cut/join/sort/uniq. But also not quite graphql? Which I suppose would be the "natural" language for a json-query tool?
I feel like I'm missing a piece - and missing out...
Filter out empty or null values:
https://stackoverflow.com/questions/39500608/remove-all-null...
I seem to either end up with unwieldy, long/verbose solutions - or shorter ones that does not work :)
I tend to get 90% of the way with jq and then awk/sed/grep my way to the desired result (jq -r).
I'm guessing that OP wasn't parsing a single 90Gb object, but a stream of many newline delimited JSON objects.
Plus the human element helps.
I wish that were true. Alas it really was a 90gb object. Actually, the first thing I did was split it by top-level key before processing further.
They can be huge changes. With slightly larger docs we had a setup that had to return batches of results back. The bottleneck was absolutely the single thread part of the API encoding and decoding JSON.
Even if the docs were just the size of my example that's 25s for 1m docs. With a few threads I can easily run 10k+/s of the useful part so that'd be 100s + 25s + encode time. With a few more threads hitting 50k/s that's 20s useful time + 25s + more.
With larger docs it becomes far more of a problem. More importantly the fix is a few seconds to implement.
A million iterations that total to order of ten seconds seems quite reasonable for a quick performance comparison. It's just underspecified for the rest of us to take away much beyond that.