Rust vs Zig Benchmarks
programming-language-benchmarks.vercel.app
programming-language-benchmarks.vercel.app
Maybe platforms like Leetcode and HackerRank can publish such statistics.
I'd find such benchmarks way more interesting than comparisons of milliseconds for solutions to some arbitrary algorithms that we'll never use in real life. I'm not saying they aren't useful, but they aren't interesting.
I think this shows that Zig isn't a slow language despite its relative youth, but it'd be much more useful if someone did the work to look through the code and provide commentary on comparable the entries are.
Reminder: With gcc/clang, -ffastmath makes it undefined behavior to run a calculation that results in an infinity or NaN. Due to the way UB works, the compiler can end up miscompiling not just the floating-point calculation, but also other code nearby (e.g. delete array bounds checks).
This is why Rust does not have any fastmath-equivalent: it would allow violating memory safety in safe code.
Not checking the correctness of the output sounds like a pretty bad oversight for a benchmark
Which shouldn't surprise anyone since the heavy lifting is done by LLVM and zig doesn't attempt to add too much semantics on top of it. (For the same reason I don't think benchmarking unsafe Rust makes much sense either)
https://programming-language-benchmarks.vercel.app/zig-vs-ni...
I love Nim so much, I wish it became more popular somehow. Unfortunately it has stagnated in popularity.
That isn't something the website does show.
Language A programs can be implemented in language B without being "independently written".
1.zig may have been based on 9.rs.
-The programs could have been written by someone who prefers to make small iterations and perform many submissions instead of someone who likes to make bigger changes with fewer submissions.
-The programs could be submitted by a novice who needs to make more submissions than would be required by a pro.
-The programming language may be older and has had more submissions.
-The programming language may be more actively changing and requires more updates in order to fix old programs.
-The programmers may also be making more submissions to improve other characteristics like memory usage, code size, code readability, compatibility, etc... that are unrelated to performance.
Language A programs can be implemented in language B without being "independently written".
As I see it, performance wise Rust has one advantage and one disadvantage compared to zig. Rust’s advantage is that it can add the equivalent of C’s noalias all over the place because of the rules imposed by the borrow checker. This can help the optimizer. And the drawback of rust is that all array accesses are bounds checked. (Well, at least in safe rust). But thanks to prediction intrinsics, the slow down from this is much less than I always expect. Bounds checks do bloat the binary size though.
So rust and zig trading blows benchmark to benchmark is about what I would expect to happen. And that’s exactly what I’m seeing here.
Slander.
Elephants are gentle, careful creatures. Probably would make good proprietors of a porcelain shop.
Unlike bulls and china shops....
Steers are probably OK though
In the years of Rust code I’ve written, I don’t think I’ve ever actually indexed into an array manually. If I have it’s been an incredibly small number of cases. I’m almost always iterating, which makes bounds checks essentially unnecessary.
> As a rule of thumb, we can say that a conditional jump is faster than a conditional move if the code is part of a dependency chain and the prediction rate is better than 75%. A conditional jump is also preferred if we can avoid a lengthy calculation [...] when the other operand is chosen
Never, ever, since 1986, have bounds checking been the major source of performance issues on applications I have written.
Rather ill chosen algorithms or data structures.
There's an open proposal to do this in Zig as well, with the ability to opt out at the individual parameter level (and with safety checks in debug builds).
https://github.com/ziglang/zig/issues/1108
Either way we can definitely thank Rust for blazing the trail. noalias in LLVM had never been stress-tested to that degree, and they were finding and fixing noalias-related optimizer bugs for years
You can see it in the function arguments in llvmir: https://rust.godbolt.org/z/a3c366nG6
Having said that, performance is a key feature for both languages and from what I can see the methodology seems legit.
I'd prefer to also have a C or C++ to use as a performance baseline and to get a clearer overall picture of where the current optimizers stands relative to each others.
Also, competition is good, I believe that we'll all benefit from it in the end, even if the heated debates and quasi-religious stances can be annoying, in the long run this is mostly noise.
(I'm assuming the Rust code isn't using "unsafe", if it is ten the "safe" zig numbers are be uninteresting)
> The goals of Zig are in contrast to the many similar languages introduced in the 2020s time-frame, like Go, Rust, Carbon, Nim and many others. Generally, these languages are more complex with additional features like operator overloading, functions that masquerade as values (properties), generic types and many other features intended to aid the construction of large programs. These sorts of features have more in common with C++'s approach, and these languages are more along the lines of that language. > reference (https://www.infoworld.com/article/3113083/new-challenger-joi...)
The provided reference doesn't really mention this statement, and C++ did not originate a lot of those features.
On a more abstract level, no having abstraction features doesn't always make a language simpler.
> A common solution to these problems is a garbage collector (GC), which examines the program for pointers to previously malloced memory, and removing any blocks that no longer have anything pointing to them. Although this greatly reduces, or even eliminates, memory errors, GC systems are relatively slow compared to manual memory management, and have unpredictable performance that makes them unsuited to systems programming.
> reference (https://docs.elementscompiler.com/Concepts/ARCvsGC/)
I am not sure how authoritative the given reference is, but even assuming it is, there is no mentions on manual vs GC speed, or anything about how GC systems are inadequate for system programing (what ever that means in this context). I know the the memory management styles trade-off are still hot debates, and a lot of progress have been made with regard to modern implementation.
> Another solution is automatic reference counting (ARC), which implements the same basic concept of looking for pointers to removed memory, but does so at malloc time by recording the number of pointers to that block, meaning there does not need to perform an exhaustive search, but instead adds time to every malloc and release operation.
This doesn't sound like the best way to describe ARC..., again the provided references doesn't substantiate this definition.
> Zig aims to provide performance similar or better than C, so GC and ARC are not suitable solutions. Instead, it uses a modern, as of 2022, concept known as optional types, or smart pointers.
I don't think smart pointer and optional types are the same. Quickly glancing at zig official website , this looks like a mistake
> Instead of a pointer being allowed to point to nothing, or nil, a separate type is used to indicate data that is optionally empty. This is similar to using a structure with a pointer and a boolean that indicates whether the pointer is valid, but the state of the boolean is invisibly managed by the language and does not need to be explicitly managed by the programmer. So, for instance, when the pointer is declared it is set to "unallocated", and when that pointer receives a value from a malloc, it is set to "allocated" if the malloc succeeded.
Mixing optional types and they way zig uses it to represent references.
I would expect Rust to be 2x-3x slower in real production code, solely due to approach to memory allocation.
Would you mind elaborating on this? I don't think I've read anything (yet?) that touches on performance differences due to that particular aspect of the languages.
> to use different allocators on a per-collection basis
It's far more than that. There are different memory management optimization patterns that are hard to implement in Rust, ie. splitting allocations to different arenas based on allocation lifetime.
> But Rust programs aren't allocating that much in the first place
Are Rust programs allocating mostly from the stack then?
> is going to triple the performance of arbitrary programs
The difference between malloc and any hand-rolled O(1) allocator is enormous, it's just people rarely benchmark this particular aspect.
There is a reason why Rust used to ship jemalloc and why #[global_allocator] is a thing. I would argue there's even more reason to avoid touching heap in the first place.
Edit: formatting.
Yes, overwhelmingly so.
The basic rule of manual optimization is that the upper bound for the improvement that can be gained by optimizing any one "thing" is equal to the proportion of the program's runtime that is devoted to that thing. E.g. if a flamegraph shows that 10% of a program's runtime is devoted to a specific function, then improving the performance of that function is going to improve the performance of that program by at most 10%. (We might call this "generalized Amdahl's law": https://en.wikipedia.org/wiki/Amdahl%27s_law )
Therefore, in order to improve the performance of a program by 2-3x by changing allocators, that requires that a program be spending at least 50% to 66% of its runtime on allocation. And it would be a highly anomalous Rust program that was allocator-bound in this way.
More specifically, arrays are about the fastest thing around, and handing out objects from within a preallocated array tends to give the best access performance by a large margin. From what I understand you basically have to sidestep the Rust borrow checker to achieve this. Which does raise some interesting questions.
I think the original poster was also saying that heap allocations are slow, which is true, but I agree it'd be easier to tell if your program is having trouble with that.
var foo_list = ArrayList(u8).init(foo_allocator);
try foo_list.append(1);
try foo_list.append(2);
try foo_list.append(3);
let mut foo_list = Vec::new();
foo_list.push(1);
foo_list.push(2);
foo_list.push(3);
> More specifically, arrays are about the fastest thing around, and handing out objects from within a preallocated array tends to give the best access performance by a large margin. From what I understand you basically have to sidestep the Rust borrow checker to achieve this.
Where did you get this impression? Rust has stack allocated arrays, and you can hand out references to them just as well as you can to any other owned type. I can think of no distinction that the borrow checker makes between stack- and heap-allocated types.
In some workloads short-lived allocations are common. Like memory is allocated for milliseconds and then deallocated - and X% of time will be spent just inside the allocator trying to find the best block to allocate. At this point you want to make sure that you don't spend more time in the allocator than allocations actually live.
I wonder how common business tasks can be coerced to this model.
Also I believe this is not a good excuse - the language should still provide a way to properly manage allocations even if it is not needed most of time.
> The basic rule of manual optimization is that the upper bound for the improvement that can be gained by optimizing any one "thing" is equal to the proportion of the program's runtime that is devoted to that thing.
Yep, and this is why you should always measure things before optimizing. But then again, you can save yourself a lot of time by not doing stupid things like allocating from heap in a hot loop and profiling just to discover such basic mistakes.
Rust makes it especially hard to use custom allocators, see bumpalo for example [0]. To be fair, progress is being made in this area [1].
Theoretically one can use a "handle table" as a replacement for pools, you can find relevant discussion at [2].
[0] https://github.com/fitzgen/bumpalo
[1] https://github.com/rust-lang/rust/issues/32838
[2] https://news.ycombinator.com/item?id=36353145
Edit: formatting.
And when you do know what you want, explicitly optimizing those parts is equally possible in the two languages.