Coming from C/C++, I know that there are structs in memory and can visualize what they look like - surely I can just go and tweak them? (No I can't because either it wasn't safe to do that anyway, or I need to explain to the Rust compiler why it is safe, and that can be quite difficult.)
Coming from something like Java, I (probably) don't think too much about what the underlying structure in memory is, so I'm less likely to try to do this.
Either way, it feels like part of the learning curve is learning the "Rustic" way to do something - in general, you probably shouldn't be reaching for "unsafe" until you know what you're doing! ;)
I tend to agree with this sentiment. The moment I saw that OP was doing pointer arithmetic: I knew they were in for a bad time. It's not that you can't do it -- it's moreso that Rust's raison d'etre is to highlight the flaws inherent in that approach.
I hadn't thought about it, but I think you're right, my background in managed languages is probably why I didn't run into these sorts of issues until after I was already comfortable with Rust. I wasn't trying to make maximally storage-efficient collections right out of the gate, instead I was building application-level stuff on top of `std::collections`.
Which is what's being discussed.
To me it's not a problem at all, Rust is working exactly as designed. When you start writing code in the style that Ayende has: its memory layout becomes entirely non-obvious. The structure definition no longer lines up with how that memory is actually being used. You'll need documentation of the sentinels & invariants, you'll want diagrams of the memory layout (since it won't line up with the structure definition), and a careful understanding of every line of code packing the bytes -- that's just to get a basic understanding of what's actually going on.
This is fine, and you absolutely need to be able to do it, which is exactly why Rust provides `unsafe {}` (along with `std::ptr` and `std::mem`.) Sometimes you need that control you want a memory-efficient data structure, other times that's because you're reading registers directly off the hardware. In either case Rust is going to make you use `unsafe {}` which is a signal to say: "to understand how this is really laid out in memory you need to understand every single line in this block."
I would much rather be able to grep for `unsafe {}` and audit parts of a trie-map for memory safety, as opposed to auditing my entire program for memory safety violations. That's not a failing of Rust, it's functioning exactly as intended. Trying to squeeze every last byte out of a struct is neither obvious nor safe, it requires careful thought.
you mean auditing the entire implementation of the trie data structure, not the entire program.
Guess what?
In rust, if you use unsafe, you also have to audit the implementation of the entire trie datastructure.
And if you think you don't, then I submit you don't understand Rust safety that well.
The code in the blog post looks very much like C-translated-to-rust. Right now, systems programming mostly does look like C/C++ because those are the only two mainstream languages that work here (D, too, but D is like C++. There are also other languages). But systems programming doesn't have to look the same or use the same primitives.
Do you really need to use malloc to allocate memory in a systems language? Do you really need to use pointer arithmetic to deal with things in a systems language? Sometimes, but not necessarily often. Even the Rust operating systems have managed to minimize the amount of unsafe/C-like code.
Rust does expose this functionality, but if it's not something it expects you to use often it makes sense that it might not always be straightforward to use.
To add to this point, my answer would be a big NO, if one would look to the C alternatives that sadly became niches.
Sadly the adoption of C has lead to younger generations thinking that those are the only valid paths in system languages.
Thankfully new languages, including Rust, are changing that.
It is written from "I have one CPU" side. Even C is far beyond that with things like OpenCL...
In any case, Rust is definitely not written with "I have one CPU" in mind. A slogan that is sometimes used for Rust is "safe. fast. concurrent.", and it's one of the headline features on the front page: http://rust-lang.org/ . Rust's abstractions allows writing extremely high performance libraries like rayon, which, had better performance than even highly optimised C libraries like Cilk, in one benchmark I heard about. Building out that parallelism ecosystem takes time, so it does not seem particularly surprising that there are more (unsafe!) libraries for the 40-year-old language, than (safe!) libraries in the less-than-2-year-old one.
You get to use unsafe almost exclusively in the internals of such access where it shouldn't be necessary if there are ways to specify more properties than just binary "safe" and "unsafe".
It is extremely hard to prove such code correct in Rust.
The only way I can make this make sense, is having cross-platform abstraction around atomic orderings etc. (like the new C++11 std::atomics) that works on x86 and ARM and PowerPC etc.
Rust has such a cross-platform abstraction, almost identical to C++11: https://doc.rust-lang.org/std/sync/atomic/index.html
Please be more specific about what exactly you mean if I am wrong.
If you're adding performance considerations I think the answer is going to be yes.
polymorphism is of course an abstraction, the idea of cost-free is that you only pay for the abstractions you actually use.
This is not the same as maximizing performance.
The real question is whether or not you can do these things without giving the developer the control to maximize performance, and I think the answer is going to be no. The response will be that it's "good enough", but that's not actionable as we don't know what "good enough" means for any particular project.
You can build (or use, from the stdlib) safe abstractions over malloc that have the same performance. This should cover 99% of the use cases of malloc. For the few quirky cases where you still need malloc, you can still use it, but that means that the language doesn't need to make malloc-based memory management easy.
You can't do systems programming without them, what you can do is minimize their use by building abstractions, which other languages, such as C++ and D, already do.
So this idea that people somehow think you have to use the actual malloc call throughout your code is a bit of a strawman.
So yeah, (modern) C++ also contests this claim too. Rust contests a stronger claim about safety as well.
------
You're the one who said that "it's claimed to be a systems level language, and if it's really that painful to do systems level things," -- my point is that in both Rust and C++, "systems level things" don't need to involve pointer arithmetic and other similar things. They should be niche tools you reach for when the abstractions fail you, and the abstractions should not fail you that often. This is already true for both languages, I feel. It's not "niche" in C++ because the ecosystem needs time to evolve to that point, but for new code it probably works out.
Then there's the C constructor convention through opaque void pointers idiom.
This stuff is not new, not even close to new, NO ONE, not even the kernel developers, would make the argument that you cannot build abstractions around malloc.
If that's really what you're arguing, well you're not saying anything that C developers don't already practice.
smart pointers have been around for a while too. I see some codebases using them. But many don't.
> NO ONE, not even the kernel developers, would make the argument that you cannot build abstractions around malloc.
I never said that kernel devs or whatever would make that argument.
----------------
Look, you seem to think I've said something different than what I did here.
You said "if it's really that painful to do systems level things, then there's a problem somewhere.". I'm saying that Rust contests the claim that malloc/new/whatever must be easy to use for systems level things. Easy-to-use abstractions around them work well too. It's not that C++ doesn't have these abstractions, it does, and has for a while. My statement isn't one against C++.
My point is that the difficulty of using programming paradigms designed around malloc/new should not be a dealbreaker for a systems language, provided that the same tasks can be accomplished without raw allocation using a suitably generic abstraction. C certainly promotes malloc-based-paradigms. C++ has good alternatives, but a lot of the lower-level stuff I find avoids them because ultimately malloc/new aren't hard in C++. Rust makes them harder to use, but provides sufficient abstractions/abstraction-building-power that this should not be an impedance to systems programming. And, again, I'm not saying C++ doesn't -- I'm just saying that C++ also makes it easy to use malloc/new-based patterns, so you will continue to see those in systems programs even if they aren't necessary.
Now, interestingly, an abstraction for a vec-with-header doesn't actually seem to exist in the crate ecosystem, probably because folks don't like unconditional allocation. But it's relatively easy to build, might do it over the weekend.
(You can fake it in safe code using DSTs but it's not so great)
You may not like that they tend to be very thin abstractions, but they are abstractions, and the point is that the idea of building abstractions around malloc isn't new and certainly isn't unique.
You said:
>> You can build (or use, from the stdlib) safe abstractions over malloc that have the same performance.
While in defense of the statement:
>> Rust contests the claim that you actually need to do pointer arithmetic and stuff often in systems languages. You need to do it a bit to implement your abstractions, maybe, but it is not the first tool you should need to reach for.
This is presented as something unique to rust, which isn't even close to the accurate. This is why I called it a strawman. No one, not even kernel developers using C, are incapable of building abstractions around malloc to make it easier/safer to use.
The rest of what you say is inconsequential in my opinion. This isn't about C++, this is about your implication that building abstractions around malloc is something new or novel about Rust. It isn't. Not even the idea of safety around malloc is new, new[]/delete[] are themselves abstractions intended to make it less error prone to create/destroy contiguous blocks of memory.
That doesn't even get into the smart objects and RAII which have been around for a very very long time in C++.
None of this is new to Rust, and I think it's a bit disingenuous to imply that it is.
Right, I wasn't talking about just that, I was specifically referring to abstractions that make you change your programming patterns. C++ has those too (the move-based abstractions), but I don't consider new/delete to be them.
My point was sort of "being able to write code that looks like this is not necessary for systems programming". Being able to do the task at hand is, but it need not be done using code like that.
> This is presented as something unique to rust, which isn't even close to the accurate.
Fair. It's not. I didn't intend to mean that. I'm sorry.
I was a C++ programmer for a quite a while, some of which time I used modern C++, so I'm quite aware that these abstractions are not unique to Rust.
The reason I phrased it that way is that C++ doesn't really make it harder to use direct-malloc-based (or raw-pointers-with-new) programming, and most of the lower-level C++ code I've seen (this is anecdotal) still uses this. So C++ doesn't contest the claim the same way Rust does (it does contest the claim though), since Rust codebases doing the same thing (even the kernels in Rust) tend to move strongly in the direction of avoiding unsafe pointer twiddling as much as it can. This is a bit more nuanced (and based on anecdotal evidence on low level codebases I've seen) than I really wrote in my original comment, and I apologize for that.