Hacker News Reader

Comment by ashvardanian | Hacker News Reader

There is a simple way and there is a hard way.

Simple - memory-map the whole file with `mmap`. We also suggest using `madvice`, to inform the kernel that you will be accessing the data just once, strictly in the sequential order. And then simply give it to SIMDJSON. With `ondemand` parser, your memory usage will be proportional to the depth of the deepest tree, not their number.

Harder way, would be to parse file in chunks, locating continuous sequences that form a single document.

There may be more options, but I'd have to check the docs :)

View on HN

Regarding the hard way, this little utility does a great job of splitting larger than memory JSON documents into collections of NDJSON files:

https://github.com/dolthub/jsplit

Thanks! Yeah I've seen both general advice but since I'm not incredibly familiar with the API I've had a hard time creating code that works.

If you have any code examples of either of those options I'd love to see them!

Edit: Also, I'm curious: what are the downsides to doing the mmap route?

Downsides - unpredictable latency for IO when the parser crosses a page boundary and encounters an unbacked page. Unpredictable IO issue size (might be smaller IOs than ideal). Unpredictable caching behavior (OS may drop the pages before the application is done with them, only to have to reread again).

You can mitigate this somewhat with explicit MAP_POPULATE and mlock.

Also, generally higher overhead, more memory use for very large files (the page tables are not free), and no way of handling errors.

> more memory use for very large files (the page tables are not free),

They're... kinda free (in terms of capacity). A PTE or PDE is 64 bits per page. Even with 4kB pages, this is 8/4096 = 0.2% overhead. Churning the page tables and TLB as you walk the file is expensive, or at least historically was expensive (used to be that removing an entry required flushing the entire TLB; I don't think that's true anymore).

If you can use MAP_HUGE_2MB, this drops off to 0.00038% and the TLB impact goes way down.

> no way of handling errors.

Yeah, this aspect is really significant. Thanks for mentioning it.

0.2% really isn't free. If you have a data set of 10 TB (one small-ish HDD right now, or one large SSD), that's ~20 GiB in page tables alone. If you can use 2MB pages, then sure, but I would assume that also limits your page-in granularity.

Oh, you're right. I wasn't remembering we needed PTEs to cover the data not paged in as well as the data in RAM. Sorry.

Well, here's an idea. Do you HAVE to have the file as a big json file? How did it end up like that?

When you source the data, can you instead output the overall doc as "part" files that are aligned along subdocument boundaries?

THen you don't have to solve that "find the subdocument terminator" on stream-back.

You are PROBABLY building the json doc via some DB dump, so just change the code that dumps the data.

If you need to reconstruct it as a single json doc for some reason, it is about as trivial as it gets to reconstruct the data as a single document.

Or programattically, a "file" object that just does the reconstruction on the fly from the source part files as it is read, that should be relatively trivial in lots of languages.

Reply on news.ycombinator.com