Hacker News Reader

Limiting to the "top 5" vulnerabilities by number of CVEs feels like cherry-picking. It's true that spatial memory safety is lower-hanging fruit than temporal memory safety, sure; but the CVE list is dominated by vulnerabilities in web applications which are mostly already written in fully memory-safe languages (additionally, these vulnerabilities are typically mitigated by library/API design rather than by programming language design, in which case Rust gives you quite a lot more tools than Zig for designing APIs that have concepts of unsanitized/untrusted data, but I digress). If you filter for vulnerabilities relevant to applications that would typically be written in C/C++/Rust/Zig, use-after-free is easily within the top 5.

The exact positioning within the top 10 is also quite noisy: if you look at last year's list, UAF is in the #1 spot for actively exploited vulnerabilities, even beating out all the web ones: https://cwe.mitre.org/top25/archive/2023/2023_kev_list.html

Lastly, filtering on CVEs has a high selection bias: just because security researchers go for the easy vulnerabilities first doesn't mean that the harder ones can be ignored. As high-profile projects adopt tooling and mitigations to reduce the impact of spatial memory safety problems, it's common to see a sudden increase in CVEs related to temporal memory safety. This is not because use-after-free bugs somehow became more common or severe -- it's because once you eliminate the easy source of vulnerabilities, the attackers shift their attention to the new lowest-hanging fruit.

> Limiting to the "top 5" vulnerabilities by number of CVEs feels like cherry-picking.

The point isn't about limiting to the top 5. The point is that once you get to the things Rust prevents and Zig doesn't, there are quite a few more things that neither prevents, so it's just silly to draw a a particular sharp line between Rust and Zig because they perform exactly the same (in terms of sound guarantees; we're ignoring any softer effects) for most top weaknesses.

Even if you think that difference is so important that it justifies downsides that Rust may have in comparison, you still have to admit that Zig is much, much closer to Rust than to C by that measure.

And this "closer" matters because Rust's memory safety is also not absolute, and Rust proponents must accept that the cost of memory safety is an important factor, too, and sometimes not worth it, or else Rust wouldn't have been invented in the first place. After all, languages that are as memory-safe as Rust and more were more popular than Rust will ever be before it was even invented.

So Rust proponents must accept that eliminating dangerous vulnerabilities is good (unlike C), that productivity and cost do matter (unlike ATS), and that non-absolute memory safety is acceptable. And Zig satisfies all of these points, too.

The reason it's hard to find an objective metric to draw the line between Rust and Zig is because they're actually quite close to each other, at least on this front of trying to find a useful compromise between productivity and guarantees.

> Lastly, filtering on CVEs has a high selection bias: just because security researchers go for the easy vulnerabilities first doesn't mean that the harder ones can be ignored.

Sure, but then you might as well also consider softer effects. For example, maybe a language that's easier to review because it's more explicit, or a language that's faster to compile and is easier to test wins.

And I agree that we should consider all these, but then we start seeing why correctness is such a complicated topic, and we could speculate just as easily that it is Zig that "clearly" wins.

Anyway, it's perfectly fine for people to prefer Rust because they like it. But the attempt to find objective reasons for this preference is not based on any truly objective foundations, and just looks like some desperate rationalisation.

And BTW,

> relevant to applications that would typically be written in C/C++/Rust/Zig

If you think that Rust and Zig are designed to target the exact same domains, then some of the "softer" aspects I mentioned could play even a larger role. I mean, the portion of software written in low level languages has been declining steadily for a long time with no sign of a change in the trend. To me it seems that Zig has internalised the narrower and more focused and role of low-level languages today compared to what C++ imagined it would be in the eighties.

> it's just silly to draw a a particular sharp line between Rust and Zig because they perform exactly the same (in terms of sound guarantees; we're ignoring any softer effects) for most top weaknesses.

There is a very clear sharp line between them in that Rust has no undefined behavior outside of an unsafe block. This matters because the effects of undefined behavior -- particularly memory-corrupting undefined behavior -- are unbounded. Security issues caused by logic bugs are limited to only having local effects on the program -- a SQL injection bug can lead to the wrong SQL query being executed, a path traversal bug can lead to the wrong path being accessed, and a shell escaping bug can lead to execution of the wrong shell command. These are severe problems that can lead to data exfiltration or remote code execution, but the relationship between bug and effect is straightforward: "I'm passing user input to a shell here; this could have catastrophic consequences so I'd better review this carefully." In contrast, undefined behavior can happen anywhere in your program, and it can do anything. That is a clear and measurable difference, and emperical evidence from the last 10 years clearly indicates both that it is nigh impossible to write large C/C++ applications without both spatial and temporal memory safety vulnerabilities, and that adopting Rust measurably decreases the incidence of such vulnerabilities.

Zig is certainly an improvement over C on this front, but it is still a UB-heavy language, and that's where the sharp line is. It's hard to make emperical comparisons because Zig is so young, but the comparision between Deno and Bun in the article is a reasonably strong demonstration that Zig has not achieved a comparable level of memory safety to Rust.

> Sure, but then you might as well also consider softer effects. For example, maybe a language that's easier to review because it's more explicit, or a language that's faster to compile and is easier to test wins.

> And I agree that we should consider all these, but then we start seeing why correctness is such a complicated topic, and we could speculate just as easily that it is Zig that "clearly" wins.

This doesn't really have anything to do with my point that the CVE list does not provide evidence to dismiss temporal memory safety as irrelevant; it's more of a general statement on writing correct software. But regardless, "Zig has nullability and bounds checks" and "Rust has an affine type system, statically-checked immutability and exclusivity, language-level resource management, and no undefined behavior" are not in the same league of program correctness. Rust wasn't designed after some ideal of memory safety at the expense of clarity and correctness: the lifetime and type system came from a goal of reducing logic bugs in complex concurrent programs, and the fact that it is powerful enough to achieve memory safety without garbage collection was a happy accident. The emperical results that Rust programs have fewer memory-safety vulnerabilities demonstrate that Rust's static approach to software correctness is successful, and not just for the specific problem of memory safety, because Rust's tooling for achieving program correctness is generalizable to arbitrary application invariants. This lines up with my own experience; software I write using Rust is far easier to get right, more reliable, and easier to successfully maintain and refactor than anything else I've done.

Certainly Zig has its strong points as well -- explicitness can reduce complexity and compile times, but it also has downsides of pushing complexity into application code (thus making it harder to review and introducing more opportunities to create mistakes). A proper comparison of the two approaches is worthwhile, but just as "Rust is more memory safe" is an overly reductive generalization of the langauges' approach to software correctness, "there's a rather small difference between Rust and Zig" simply isn't true.

> There is a very clear sharp line between them in that Rust has no undefined behavior outside of an unsafe block.

Yes, but that's an intrinsic language feature whose value needs to be justified somehow. If you justify it by saying it prevents dangerous vulnerabilities, we're back to my point.

> This matters because the effects of undefined behavior -- particularly memory-corrupting undefined behavior -- are unbounded...

While I appreciate this explanation (and I've seen it many times), I hope you understand that it's more speculative and subjective than an empirical finding. At the end of the day, to measure the danger of a problem, we need to see what the actual vulnerabilities/exploits are and how common they are. Clearly, not all undefined behaviours are equally exploitable. That's why we see differences in weakness severity among different kinds of UB.

> emperical evidence from the last 10 years clearly indicates both that it is nigh impossible to write large C/C++ applications without both spatial and temporal memory safety vulnerabilities

Right, and that same empirical evidence shows that spatial violations are the more dangerous ones, and that's exactly why Zig prevents them.

> but the comparision between Deno and Bun in the article

What article? It seems that Bun has had one CVE. If you're talking about undefined behaviour you're, again, making an unjustified extrapolation from it to security. Some undefined behaviour leads to easily exploitable vulnerabilities, some does not.

It's true that in the presence of UB, the compiler could hypothetically do anything, but to get a dangerous vulnerability it has to actually do something that's exploitable, and when we look at vulnerabilities caused by UB, we see that some kinds are more dangerous than others because of that.

BTW, this touches on something that is sometimes misunderstood about UB. UB is defined with respect to a language specification, i.e. in the presence of UB, the language specification does not assign a program a meaning, but the compiler certainly does, because machine code has no UB. The program with the UB needs to compile to an exploitable binary for a vulnerability to exist.

> and that's where the sharp line is

I agree it's a sharp intrinsic line, just as saying that Zig avoids macros is a sharp line, but the impact of that line is anything but sharp. To draw the practical conclusion from it you don't follow the findings, but ignore them!

> This doesn't really have anything to do with my point that the CVE list does not provide evidence to dismiss temporal memory safety as irrelevant

I wasn't dismissing it as irrelevant. I was saying that if Rust's value is in eliminating dangerous vulnerabilities, then Zig has that value, too, and the difference between them isn't large on that particular front.

> are not in the same league of program correctness

Software correctness is something I've been dealing with and writing about for many years, especially formal verification (https://pron.github.io), and I can tell you that you're downright wrong on that. That "more sound guarantees is always the most effective form of improving correctness" is something we know (at least since the nineties) not to be generally true, which is also why the field is looking more and more into unsound methods. For example, Rust and Zig are more likely to effectively write correct programs than ATS, even though ATS is "in a different league" from both of them when it comes to sound guarantees.

We know that sound guarantees can help, but that their cost matters a lot. We also know that reviews and tests and dynamic verification are very effective, sometimes more than sound guarantees.

Rust proponents are free to speculate that, ultimately, Rust's approach leads to more correctness than Zig, and they can base that belief on some findings, and Zig proponents can do exactly the same in the opposite direction, also based on other findings, but both are speculations. People are free to choose which they are more inclined to believe, but the question isn't settled.

> This lines up with my own experience; software I write using Rust is far easier to get right, more reliable, and easier to successfully maintain and refactor than anything else I've done.

I'm not doubting that that's your experience. I'm saying we have no evidence that it's universal, likely to be universal etc. (and I know of some opposite experiences with Rust). Sometimes certain languages just click with certain people, but we can't extrapolate without more observation.

> "there's a rather small difference between Rust and Zig" simply isn't true.

The correctness difference between the two is unknown. It could be small or large and in either direction. The intrinsic differences are, of course, known, but don't really help reach an objective preference. My point was only that if we judge Rust by the vulnerabilities it soundly eliminates, then Zig is not far on that particular metric, and it's certainly much closer to Rust than to C.

Maybe we should be using ATS. Or more likely, maybe we should be using some novel language that doesn't exist yet that brings the benefits of ATS to a language with good tooling and good DX that you can use to build practical system software with - that is, a Rust for ATS instead of C/C++. I think we should be designing programming languages that help eliminate as many classes of bug as possible, and Rust is not the culmination of the line here.

One of the lessons of the past 50 years in software correctness is that sound guarantees are not always the most effective path to correctness. The problem is that proving something correct takes a lot of effort (and there are fundamental computational complexity reasons for that), while unsound methods are significantly cheaper and surprisingly effective in practice. A famous 1996 paper by Tony Hoare [1] expresses amazement at how software had become so reliable without proofs, something that in the 1970s was thought impossible. Since then, the field has moved to enthusiastically adopt more unsound methods.

And remember that a software system, unlike an abstract algorithm, is a physical system that cannot be proven correct, since the behaviour of the physical hardware cannot be proven. We're always dealing in probabilities, and so the question is: how do we get the most value (in terms of reducing the probability of costly bugs) for a unit of effort.

Since the 1970s, the size of software that can be proven correct in practice using deductive methods has only fallen compared to the average size of a program (i.e. the size of acceptably-reliable software we write has grown much more rapidly than the size of software we can prove correct using deductive methods). The largest programs ever proven correct using deductive methods are on the order of 10KLOC.

So the field of software correctness has long ago abandoned the position (held by some in the 70s) that proof is always the most effective way toward correctness.

[1]: https://gwern.net/doc/math/1996-hoare.pdf

Bit of a tangent, but what is your perspective on formal verification in hard realtime systems? Is the cost justified because the system tends to be simpler and doesn't need to evolve through time, or some other reason? Or do you see formal verification with hard realtime systems as unnecessary effort?

I think formal verification can and should be used everywhere it is helpful, which certainly includes hard realtime systems but isn't limited to them (it's helpful in quite a few areas). But formal verification is by no means the same as deductive theorem proving. Especially for hard realtime systems, which tend to be simple as you say, model-checking has been the formal method of choice for decades.

Even for large, non-critical software, there are useful formal verification methods that aren't end-to-end, i.e. they can cover the design but not the code, and have proven very useful in finding bugs. I for one, am a big fan of TLA+. TLA+ has both an interactive theorem prover and a model checker (or a couple). Most importantly, it allows describing the system at an arbitrary level of detail, which means you can use it at different levels as appropriate. For some things, it can and should, say, describe hardware in full detail; for others, it can be used to describe and verify a very abstract algorithm, well above the code level.

The problem with deductive theorem proving is that it tends to have a low ROI, and there are often more effective methods. It should be used when other methods don't work well.

> Zig must also be good for eliminating the same ones.

But Zig does not eliminate them, but rather it might catch them at runtime. The difference here is that Rust promises that it will detect them at compile time, long before I ship my code.

> The property of memory safety is itself not binary in both languages

In this case it is: either you catch the issue at compile time, or you don't. This is the same as type safety: just because Python can detect type errors at runtime it does not mean that it's as "type safe" as, for ex. Haskell. This might be due to imprecise usage of terms but that's just the way it's discussed in the craft.

> But Zig does not eliminate them, but rather it might catch them at runtime

So does Rust. That's what we mean by eliminating them. Zig/Rust will panic, but you won't get a vulnerability.

> The difference here is that Rust promises that it will detect them at compile time, long before I ship my code.

No, Rust doesn't guarantee catching spatial-memory-safety violations at compile-time. Neither do Java, Python, or JS/TS, by the way.

I think a lot of the absolutists around memory safety are just using it as a bludgeon to evangelize their preferred programming language. Otherwise, the position of "memory safety is the most important thing, but we should ignore other languages that provide stronger guarantees" is absurd.

There's no magic wand that can make memory safety unimportant, even doing other important things.

I agree! Memory safety is important because it's the cause of some of the top vulnerabilities, which is precisely why it's great that both Rust's level of memory safety (which is not absolute) and Zig's level of memory safety (also not absolute), both eliminate those top weaknesses!

But once you've taken care of the top weaknesses, it becomes harder to justify putting more effort into eliminating some weaknesses that could go into reducing more common/dangerous ones. If vulnerabilities are the justification, it definitely makes more sense to reduce, say, #4 on the list than #7.