Hacker News Reader

Comment by dietr1ch | Hacker News Reader

cat file.whatever | whatever2json | jq ?

(Or to avoid using cat to read, whatever2json file.whatever | jq)

That's not really random access, though. You're effectively just searching through the entire dataset for every targeted read you're after.

What might be interesting is to have a tool that processes full JSON data and creates a b-tree index on specified keys. Then you could run searches against the index that return byte offsets you can use for actual random access on the original JSON.

OTOH, this is basically just recreating a database, just using raw JSON as its storage format.

> What might be interesting is to have a tool that processes full JSON data and creates a b-tree index on specified keys. Then you could run searches against the index that return byte offsets you can use for actual random access on the original JSON.

I did build that once. But keeping track of the index is a pain. Sometimes I was able to generate the index on-demand and cache it in some ephemeral storage, but overall it didn't work out so well.

This system with RX will work better because I get the indexes built-in to the data file and can always convert it back to JSON if needed.

Well, JSON had no random access to begin with, so maybe that's on needing JSON.

Maybe a query over the random-access file then converted into JSON would work?

Right -- that's exactly the point. JSON is sequential data, and has no mechanism for random access. The previous comment described piping the JSON into jq as a solution for querying specific bits of data out of a JSON stream, which is consistent with what you're suggesting here, but still isn't a solution for random access.

Or in this case, just do `rx file.rx` It has jq like queries built in and supports inputs with either rx or json. Also if you prefer jq, you can do `rx file.rx | jq`

wow, on that case then using `jq` is just a presentation preference at the very last step unless jq is more expressive (which might be the case given how long it has been around?).

right, the jq query language is much more complex and featureful than the simple selector syntax I added to the rx-cli. But more could be added later as needed or it could just stream JSON output. It would be pretty trivial to hook up a streaming JSON encoder to rx-cli which could then pipe to jq for low-latency lookups. The problem is jq would need to JSON parse all that data which will be expensive.

Reply on news.ycombinator.com