Hacker News Reader

Comment by nijave | Hacker News Reader

Not sure if anything exists but I wish something would do in memory compression + smart disk spillover. Sometimes I want to work with 5-10GB compressed data sets (usually log files) and decompressed that ends up being 10x (plus add data structure overhead). There's stuff like Apache Drill but it's more optimized for multi node than running locally

View on HN

If you're not afraid of exotic languages, I encourage you to have a look at the APL ecosystem, and especially J and it's integrated columnar store Jd.

I have just embarked on an adventure to do just what you describe in... Racket. But it's nowhere to be seen yet.

I'm an epidemiologist and I've been wanting to make my own tools for a while, now. It'll be interesting to see how far I can go with Racket, which already includes many pieces of the puzzle.

Would you mind pointing me at a resource that explains how J/Jd handle it by comparison?

https://code.jsoftware.com/wiki/Jd/Overview

Jd only packs int vectors, though. So if you're hoping for string compression then I don't know of any free solution. Jd heavily leverages SIMD and mmap. Larger-than RAM columns can be easily processed by ftable. I use Jd for data wrangling before making models in R. Of course, the J language is not for the faint of heart but it's really well-suited to the task.

I wonder if Arrow (https://arrow.apache.org/faq/) would work for this:

"The Arrow IPC mechanism is based on the Arrow in-memory format, such that there is no translation necessary between the on-disk representation and the in-memory representation. Therefore, performing analytics on an Arrow IPC file can use memory-mapping, avoiding any deserialization cost and extra copies."

Dask is popular for this purpose. No memory compression, but supports disk spillage as well as distributed dataframes.

If youre doing OLAP style queries you should look at DuckDB, it's hella fast and supports out-of-memory compute (it's not exactly "smart" but it handles spillover)

I feel like the HDF5 libraries could be helpful here if you could figure out how to get compatible compression.

What do you mean with smart disk spillover?

When the library attempts to load something from disk that doesn’t fit into memory, it’s transparently, and (usually) without extra intervention from the user, swaps to memory-mapping and chunking through the file(s).

Particularly useful for when you’ve got a bunch of data that doesn’t fit in memory, but setting up a whole cluster is not worth the overhead (operationally or otherwise) and/or if you’ve already got a processing pipeline written in a language/framework and you can’t/don’t-want to go through rewriting it for something distributed.

How is that different to virtual memory?

In memory representation also tends to have some data structure overhead so 5GB compressed -> 50GB uncompressed -> 100-250GB as a data structure (seems 2-5x is pretty normal). It starts out seeming pretty innocuous but quickly explodes. In addition, some things like Drill can do some automatic indexing/metadata recording which can reduce the amount of data it needs to access

Memory mapping lazily loads the file, completes immediately, and scales arbitrarily; using disk-backed virtual memory would require the entire file to be read from disk, written back out to disk, and then read in from disk again on access; it would also require swap to be set up at the OS level, and the amount of swap set up puts a hard limit on the size of the file.

You've described the difference between using mmap vs relying on the operating system's swap mechanism. But neither of those is quite the same as having an application that's aware of its memory usage and explicitly manages what it keeps in RAM. Using mmap may be useful for achieving that, but mmap on its own still leaves most of the management up to the OS.

Thanks for clearing that up!

gzip, zcat, parallel... That sort of thing?

Yeah but starts to fall apart for anything beyond basic search/filtering (like grouping, computed fields)

How is that different from, say, opening up a gzipped file with a reader object in python?

It's not but implementing intelligent data access each time can become pretty tedious (sure you can write libraries and tools, but I'm basically asking if those already exist)

For something like AWS CloudTrail logs, 5GB is 40k 100-130kb gzipped json files so hit single-core CPU bounds almost immediately (just reading/decompressing/json parsing off an SSD). CPU scale out model in Python is processes so now you're copying data between processes if you want to parallelize it so now you hit IPC bottlenecks just using the standard library multiprocessing/concurrent futures stuff

5GB compressed /probably/ won't fit in memory so now you have to deal with that, too unless you have a way to keep it compressed (which would come at the cost of additional CPU usage)

tldr; it's non-trivial to actually fully use the hardware

Stepping back a bit, if you're CPU bound in a single thread decompressing data off an SSD, does copying the compressed data into memory first actually buy you anything?

If you truly need the dataset fully loaded into memory for performance reasons, then it's presumably because of the need to do lots of random accesses, where the read latency would otherwise harm you. The tricky bit is the fact that it's generally hard to randomly access compressed streams of data. You need to compress the data in a way that makes random access possible, likely to the detriment of compression ratio. Unless you also use the same compression format to store the data on disk, then you're back to having to decompress (and recompress) the whole file sequentially anyway in order to build the data structure in RAM.

I've seen purpose-made compressed log formats that support efficient seeking. I've never seen them loaded into RAM in their raw compressed form, though. Generally they do have a corresponding library to make accessing the log data easy.

maybe give Vaex a try. it's along those lines – out-of-core data frame library, but also works on a single machine

https://github.com/vaexio/vaex

You can memory map Arrow

I think ClickHouse does that.

I actually started looking at Clickhouse a couple weeks ago but got a bit side tracked trying to grok how distributed tables work. It looks promising but there's a bit of a learning curve (seems some of the performance also comes from its use of arrays but best I can tell my use case should just use regular tables)

ClickHouse performance is principally due to column storage, compression, and ability to parallelize processing. Arrays can improve performance in some specific cases but are more commonly used to help deal with semi-structured data or perform custom processing on values within groups.

If your data maps cleanly to tables, that's in fact the best case with the easiest options for performance enhancement.

Reply on news.ycombinator.com