Hacker News Reader

Comment by amelius | Hacker News Reader

Very interesting.

I'm also interested in one particular kind of memory abstraction: abstracted access to memory mapped files. Here the problem is that the base address of the memory mapped area might change whenever you perform the mapping. And you would like to use standard data-structures (like maps, lists, etc.) inside the memory mapped area.

View on HN

Mmaped files are already available it seems (https://crates.io/crates/mmap/), although I've never tried it. You'd have to implement the interesting structures on top of that yourself however.

Hmm, it seems this library only provides the basic I/O.

The main problem is not opening the mmapped file. The problem is having a kind of pointer that allows arbitrary offsetting, and using this pointer inside standard data structures.

Rust has typesafe arenas, when one allocates small objects from a bigger chunk of memory and typesystem ensures that small objects do not outlive the chunk. I suppose mmap files could be treated similarly when assembling a pointer from an offset stored in the mmaped file is treated as an allocation.

Relative pointers are great in memory map data structures. All pointers are relative to current offset. That way it's almost always also possible to get away with just 32-bit pointers -- you get +- 8 GB range.

A library that can do that in Rust would be great.

A signed 32 bit "offset" value gives you 2GB in each direction, not 4GB in each direction.

Only if you use byte aligned relative pointers. For data structures you don't need better than 32-bit aligned pointers. So you get +- 8GB relative offsets. It's not all that unreasonable to use 64-bit or even 128-bit alignment, for +-16 GB and +-32 GB respectively.

Reply on news.ycombinator.com