Comment by josephg | Hacker News Reader

Comment by josephg | Hacker News Reader

Parent Full thread

I totally hear you about C friendly APIs not making sense in rust. GTK struggles with this - I tried to make a native UI in rust a few months ago using gtk and it was horrible.

> Also, a more Rust-friendly model would have incured higher memory costs.

Can you give some details? This hasn’t been my experience.

I find in general rust’s ownership model pushes me toward designs where all my structs are in a strict tree, which is very efficient in memory since everything is packed in memory. In comparison, most C++ APIs I’ve used make a nest of objects with pointers going everywhere. And this style is less memory efficient, and less performant because of page faults.

C has access to a couple tricks safe rust is missing. But on the flip side, C’s lack of generics means lots of programs roll their own hash tables, array lists and various other tools. And they’re often either dynamically typed (and horribly inefficient as a result) or they do macro tricks with type parameters - and that’s ugly as sin. A lack of generics and monomorphization means C programs usually have slightly smaller binaries. But they often don’t run as fast. That’s often a bad trade on modern computers.

The code is written in an embedded style, i.e. no dynamic memory allocation or thread creation/deletion after program initialization. It's also prioritizing reducing memory usage over performance since we are targeting memory constrained devices (and our performance target is 10 tps and we have like 100k tps). Thus we'd use trait objects over monomorphization. Dynamic collections are also off the table unless backed by a fixed-size arena on the stack or static mem.

We heavily use arenas. We also have runtime-typed objects used to represent dynamically typed data like that obtained from JSON/Yaml or over IPC. If we were to be more friendly to modeling in Rust, we'd likely require that all memory reachable from an object node be in the same arena, disallowing common patterns like having list/map's arrays in one arena and having keys/strings in another or in static mem (this allows reusing other buffers without forcing copying all the data, so backing arrays can be smaller).

You said above:

> Also, a more Rust-friendly model would have incured higher memory costs.

I'm not sure how modelling everything in a rust borrow checker friendly way would change anything here? Everything you're talking about doing in C could be done more or less exactly the same in rust.

Arenas are slightly inconvenient in rust because the standard collection types bake in the assumption that they're working with the global system allocator. But there's plenty of high quality arena crates in rust which ship their own replacements for Vec / HashMap / etc.

It also sounds like you'd need to write or adapt your own JSON parser. But it sounds like you might be writing part of your own JSON / Yaml parser in C anyway.

Arenas aren't the issue. Its objects with mixed lifetimes and mutability. I cant easily model the lifetimes of objects when there are cases like an instance where buffer memory reachable from the object has a different lifetime than maps/lists. Also these objects could be transively shared or mutable. In order to make a Rust friendly model, I'd have the tree be all shared or all mutable, and have all reachable memory have the same lifetime. This would often mean allocating the full tree into one arena. That is where the overhead comes from. Each arena would need enough memory to store the entire object; currently they can be smaller since they often only need to hold parts of objects, not the entire thing.

Reply on news.ycombinator.com