Hacker News Reader

Comment by antirez | Hacker News Reader

If you have mostly big objects (relative to hardware page size) and the encoding in memory and when stored on disk are pretty similar, this is indeed a good idea (to trust the virtual memory I mean).

Otherwise... no way, if you have data structures everything is fragmented around (and you want to use a lot object sharing, caching, ... for performance, without to mention hash tables that are very cool at filling at least 1 byte of tons of pages even with 2% of data inside).

Also data structures can often be serialized on disk using 1/10 of the space.

Using the VM is cool, but not so generally applicable. The proxy stuff is perfect. Also on-disk DB is perfect using the VM the other way around, to get a memory-cache for free, if you don't need strict consistency (see MongoDB).

View on HN

If your data structures can be serialized using 1/10 of the space and you can identify what you're not going to use for a while, then why not serialize it into RAM that way and let VM serialize it to disk if needed? You accomplish the same thing, but with less code and fewer system calls.

To me the big reasons to serialize to disk are as a way of sharing data between processes, and as a way of making data permanent. But not for performance. Because compared to RAM, disk has none.

Sorry I replied to the wrong comment, check one child later please ;) The best representation in memory and on disk are usually completely different, while VM will force you to use the same for both the worlds.

I can't find the "one child later" that you're referring to.

As for the best representation, I am not sure I agree. I agree that the best representation for frequently accessed data and seldom accessed data are frequently very different. Of course normally the former is in RAM while the latter is on disk. However it is not obvious to me that it is at all a bad thing to have some space used in RAM by infrequently accessed data. And if it is not, then you wind up with the solution I suggested. Just write out the data as you want it on disk, into RAM, don't touch it, and let the VM worry about when (and if) it needs to actually be paged to disk.

Of course this doesn't work if you're not using virtual memory. :-)

I've said this before, but I think the optimal would be to leverage the OS's vm by relocating subpage objects to create coherence by putting frequently coaccessed objects on the same page.

I'm pretty ignorant on systems programing, but I gather on most platforms there's a way to create a custom page fault handler, that could preserve your custom serialization format. Depending on how redis objects are linked however, allowing relocation may be complicated.

What are the difficulties in using some sort of compression algorithm like LZJB on the disk's page file?

If, as you say, data can be that easily compressible, the IO speedups from reading the compressed data from disk and space savings would outweigh the CPU cycles necessary to do the decode, but my brief Google search for compressed page file implementations came up with nothing.

A linked list is hard to compress in memory as the overhead is the metadata: pointers, malloc overhead, ... but when written on disk can be represented as prefixed length strings.

With VM the live representation and serialization format are the same.

just curious: is it not possible to have some form of unrolled-linked-list to mitigate this effects somewhat ? or perhaps you have already tried it, and it doesn't really fit the bill.

It kind of sounds like you're describing a sparse array.

> It kind of sounds like you're describing a sparse array.

hmm not really. this: [ http://en.wikipedia.org/wiki/Unrolled_linked_list ] is what i had in mind

Paul Wilson worked on compressed paging at the turn of the century [1] and tried to address this by introducing specialized compression algorithms for different kinds of pages (e.g., x86 instructions, pointer-filled data structures). Interesting stuff, but I don't think I've ever seen it commercialized, except in Newton OS where we used compressed code pages in binaries (read-only, not swap).

[1] http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.26.7...

Here's something from Android: http://code.google.com/p/compcache/

Reply on news.ycombinator.com