Comment by zengid | Hacker News Reader

Comment by zengid | Hacker News Reader

This is really impressive work and I applaud the author for building something that I'm sure lots of folks will find interesting and possibly very useful, but I want to say that I think there's a reason reference semantics (using and sharing heap allocations) are hard in Rust: It's because references are just plain hard to use safely. Rust tries to make the gnarly things awkward, but what is the easy path?

If you look at the Hylo (formerly Val as of 3 days ago) language [0] that folks like Dave Abrahams have been working on, they're outright saying to just not use reference semantics if you can avoid it. In this talk at CppCon about Value Semantics, Dave argues that we should be decoupling object graphs from access to objects [1]. That's right folks, using `usize` indexes into collections, or adjacency lists, isn't just a hack to get around the borrow checker as people like Jonathan Blow have famously critiqued [2], it may just be the right way of doing things.

[0] https://www.hylo-lang.org

[1] https://youtu.be/QthAU-t3PQ4?t=2354

[2] https://www.youtube.com/watch?v=4t1K66dMhWk

I know of a lot of people trying to do "arena" GC in Rust using integer indices. I think that's an excellent approach, but it does mean you have to pass in the arena as a resource to every single function which produces an allocation. This isn't even necessarily a bad thing - the net result is a lot like Zig's allocator API.

I know there's a nontrivial overhead to using index-based references, especially since CPUs can't easily predict load operations. This gives me an idea: create a CPU architecture where pointers are actually (object, offset) tuples, enabling better on-metal performance with index-based references. I've neither the time, money, expertise, nor energy to implement such a thing, but it would be really cool if it existed - maybe CHERI is a close approximation, though I haven't looked very closely at it.

> it does mean you have to pass in the arena as a resource to every single function which produces an allocation

Implicit arguments! I'm more and more convinced that this form of controlled dynamic scoping dynamic scoping should be embraced by more languages.

> create a CPU architecture where pointers are actually (object, offset) tuples

Segments might yet see a renaissance (and CHERI might be indeed be a form of it).

> Implicit arguments! I'm more and more convinced that this form of controlled dynamic scoping dynamic scoping should be embraced by more languages.

This might be an odd comment, but this is sorta similar to React's Context mechanism. For what it's worth, it makes a lot of things easier but it can definitely also add a lot of surface area for complexity. Although it's certainly better than relying on globals all over the place.

Yes, the obvious way to do this is have the compiler add a context parameter to every function that allocates memory or dereferences a pointer (semantically). To run the program, the main function ala c initializes the context and passes it in to the actual main program implementation. When the program finishes, all the memory can just be released in bulk in the wrapper main that initialized the context.

How is that much different than just using malloc and then letting the OS clean up when the process exits?

By having implicit arguments, a framework can decide when to override the implictness and be explicit to use a different arena when needed.

Also, many process like servers are long lived, so the OS clean up strategy does not work for them, and they are exactly where you want some strategy for memory allocation like arenas to prevent memory fragmentation.

This is actually a serious proposal for the Rust language, it's called contexts and capabilities

Here's a blog post from 2021 https://tmandry.gitlab.io/blog/posts/2021-12-21-context-capa...

This isn't implemented yet, and the proposal never matured into a proper RFC, but there's high interest for some solution along those lines

And just a note, this is called Reader monad in Haskell

In other words, receiving an implicit parameter is a kind of effect. So another way to provide this feature is to have a full blown effect system (which is much more general and is probably overkill, but there's also some interest for that)

> just a hack to get around the borrow checker as people like Jonathan Blow have famously critiqued [2], it may just be the right way of doing things.

I dunno, isn't a usize index into a collection essentially just the same as a pointer? Ok it's a bit safer because you at least know the type of the object you're accessing is correct, but you can still get e.g. use after free bugs.

I think Jonathan Blow is right and wrong - it is a hack to get around the borrow checker, but also that's totally fine. The borrow checker still works for the other 95% of code you are writing. Nobody ever claimed it was the perfect solution to every problem.

Don't you also know the type of the object when dealing with pointers and references (assuming no casts)?

Though arenas are useful even outside of "hacks".

In the end, a pointer (or a reference, even) is "just" a usize index into the heap, which can be thought of as a heterogenous collection.

Yep. A pointer/reference is just an index into a giant shared mutable array of bytes, which is exactly what causes all of the problems :)

No. An invalid pointer can point to literally anything. Just because the pointer type says it points to T doesn't mean it really does.

With an index into Vec<T> you are at least guaranteed that the object will be a T.

You are guaranteed memory safety, but you aren't guaranteed to have a correct reference.

Reply on news.ycombinator.com