Why we didn't rewrite our feed handler in Rust | Hacker News Reader

Why we didn't rewrite our feed handler in Rust

databento.com

38 pointsmpweiher8 comments

I'm not a very experienced Rust person, but I suspect the buffer reuse concern could be resolved with a tiny custom allocator, so semantically it's a new buffer on each iteration of the loop, satisfying the borrow checker, but the allocation is free. There are a number of bump/arena/slab allocators that do this, like bumpalo, but it would also be possible to write a small custom one specifically geared toward "just continually reuse this exact pointer".

Anyway, I don't mean to nitpick; I know that what's given in TFA are high level examples representing broader classes of concerns, but I thought this one in particular was interesting for having some decent alternatives.

It's possible to cast the lifetime away. It's also easy to wrap it in a safe API: https://docs.rs/recycle_vec/latest/src/recycle_vec/lib.rs.ht... (libstd will likely add something like this).

However, the other issues they mention hit some well-known limitations of Rust, which don't always have a simple satisfactory solution. It's a valid criticism.

With a bit of `unsafe`, it's easy to force through self-referential structs incorrectly and violate exclusive ownership/aliasing rules (that's UB, in Rust!). Sometimes such code can be reorganized to avoid self-referential structs. With enough `unsafe` dark arts it's also possible to properly handle some self-references. However, in general it's typically a false positive in the borrow checker's model, and an annoying limitation.

I wonder if for issues 1 and 2 a small amount of unsafe code could just solve them. At least for 1 this would seem quite achievable.

I don't even think unsafe is necessary for (1), just a custom allocator, as I suggested in a sibling comment.

For 2, I believe the pattern would be to make a generic version of the data structure as its own crate, with all the required unsafe stuff in there, properly tested and hidden behind a safe interface. Obviously that does require the data structure to be well defined and for you to know upfront what all the internal references and transformations are going to need to be (but honestly... it's probably good for that to be the case regardless).

Last I checked a custom allocator required unsafe code for pointer handling, one can hand this off to a library but the unsafe code is still there (though maybe better tested given a mature enough library).

HN algo stripped the "Why" from the original title:

"Why we didn't rewrite our feed handler in Rust"

Thanks; I was wondering what that was about. Now I just wonder why, hah.

Interesting magic: https://users.rust-lang.org/t/reused-vec-in-loop/134116/2

I still find "instead of a little unsafe, we choose all unsafe everywhere" to be kinda silly reasoning.

Reply on news.ycombinator.com