If you use a language like Rust (which is really easy for these kind of jobs), then 1mb will parse almost instantly.
I've been working with a 90gb JSON file recently. I'd probably start counting a file as large at around 100mb.
I've been working with a 90gb JSON file recently. I'd probably start counting a file as large at around 100mb.
It's the perfect solution because you can use the same decoder + something like jq for debugging and manipulation on the command line.
It isn't quite "awk-ish", not quite cut/join/sort/uniq. But also not quite graphql? Which I suppose would be the "natural" language for a json-query tool?
I feel like I'm missing a piece - and missing out...
Filter out empty or null values:
https://stackoverflow.com/questions/39500608/remove-all-null...
I seem to either end up with unwieldy, long/verbose solutions - or shorter ones that does not work :)
I tend to get 90% of the way with jq and then awk/sed/grep my way to the desired result (jq -r).
I'm guessing that OP wasn't parsing a single 90Gb object, but a stream of many newline delimited JSON objects.
Plus the human element helps.
I wish that were true. Alas it really was a 90gb object. Actually, the first thing I did was split it by top-level key before processing further.