Hacker News Reader

Comment by amelius | Hacker News Reader

I'm wondering if the following will be one day possible with Rust. I want to build a fast data-store with the basic data-structures offered by Rust (e.g., std::collections), which are stored in a memory-mapped file. The problem with mmapped files is that they don't always appear at the same address (this is not always possible, especially if you open multiple stores at once). So in order to use the standard data-structures, Rust must provide some mechanism to make this possible (I realize that this can be tricky). Alternatively, I could write my own data-structures, but of course, I'd prefer to use the standard ones.

View on HN

>I want to build a fast data-store

Usually this means you want to build things build on arrays.

>The problem with mmapped files is that they don't always appear at the same address

i.e. you use arrays and then index into the array instead of using pointers.

> So in order to use the standard data-structures, Rust must provide some mechanism to make this possible

Well, arrays exist in Rust. Or you're going to have to be more specific about what you're trying to do.

> Or you're going to have to be more specific about what you're trying to do.

I want to allocate arrays, maps, etcetera inside the region of a memory mapped file. Then, later, when I mmap the same file to a different region in memory, I want the arrays and maps to still work. That's all.

There are several mmap crates, but they won't let you transparently use std::collections on top of them.

I imagine this will be possible once the custom allocators work sorts out.

Just having custom allocators won't automatically fix the pointer issue: remapping the file later will leave the previously "allocated" pointers pointing into old memory, which may not be where the file was remapped.

The allocators and data structures would need to support using non-raw-pointer pointers types, including not even platform-pointer sized types.

Right. Makes sense.

As others have explained, it's definitely possible to construct those data structures in Rust and use them from a crate. But you can't just flip a switch to get the standard library collections to work that way.

You could probably get pretty far by copy pasting the standard library containers you're interested in and replace Box with something like OffsetBox which uses indices into your memmap arena instead of pointers to any addressable memory. Of course, implementing OffsetBox is an exercise for the reader. :)

You would probably have to write some unsafe code to get it to work. But it's not a blocker for a project.

Perhaps a solution would be to introduce the concept of relative pointers. I.e., the pointer becomes a signed value that should be added to the address of the pointer to obtain the address of the location pointed to. Not sure what the implications would be. Of course the standard data-structures would need to support such pointers.

After some searching, I found that Microsoft Visual Studio supports something similar, and exactly for the application I had in mind: [1].

[1] https://msdn.microsoft.com/en-us/library/57a97k4e.aspx

Do you have examples of how this works in other languages? I would assume they have specialized data structures for this purpose.

Here: http://www.boost.org/doc/libs/1_63_0/doc/html/interprocess/o...

Reply on news.ycombinator.com