Fuzz rising | Hacker News Reader

Fuzz rising

cloudatomiclab.com

64 pointsmasklinn79 comments

It's worth mentioning that there are sound static analysis tools that guarantee no undefined behavior in C programs (e.g. https://trust-in-soft.com/). Much cheaper and more realistic than a rewrite. Tenable or not, much of the software we rely on will remain written in C in the next few decades.

> Much cheaper and more realistic than a rewrite.

I really have to dispute this, these tools are not simple to use. You'll probably have a far easier time rewriting the code piecemeal, i.e. function-by-function (which will involve a lot of unsafety initially, since you're not using the idiomatic global patterns of a memory-safe language!) and then refactoring into something idiomatic for the language you're using. Projects like Firefox are basically taking a similar approach, only "rewriting" small portions of the codebase that can be deployed immediately once rewritten.

Where similar tools might be useful is for preventing logic-related issues that are not encompassed under simple memory safety. Unfortunately it's still not possible to endow, e.g. Rust code with automatically-checkable proofs, showing that the logic in the code preserves some appropriate conditions/invariants; while this is feasible, e.g. in Agda or Idris. Hopefully by the time these concerns become pressing, practical memory-safe languages will also offer this.

When you want to verify functional properties these tools are not easy to use. For safety, they are easier than a rewrite and largely automatic. Large, sensitive codebases have been verified (for undefined behavior, not functional correctness) relatively quickly. In general, they cover more ground significantly more quickly and cheaply than a rewrite.

> are sound static analysis tools that guarantee no undefined behavior in C programs

Proof left as an exercise for the reader, maybe? Seriously, UB is a very, very pernicious problem and without a formal specification of C/C++ and a machine-checked proof that this analysis is sound and complete, I am not going to believe it. The academic research community still regards this as an unsolved problem.

(It's a shame, too. With UB, we did this to ourselves.)

You could also say that you don't trust the compilers of safe languages, or their unsafe bits, to be correct. The guarantee here is about as strong. But the situation is not binary, and what we want is the cheapest way to at drastically reduce UB, even if you don't believe the guarantee rises to the level of being absolute.

The burden of proof is quite literally on those making claims, not on me to "not trust" them. There are certified compilers with machine-checked proofs of correctness (e.g. CompCert). That is a whole other level assurance than prose proofs and hand-wavy informal reasoning. It is probably true that most "safe" languages have some holes in them; e.g. Java does.

Fine, but the same burden exists for both sound static analysis tools and safe languages. There's no reason to doubt the claims of one yet accept the claims of the other.

As automated theorem proving and machine-checked proofs are making rapid progress, the burden is increasing. In the beginning, not much was required in the way of proof about a language's type system. Then came systematization and de facto standardization of programming language semantics, along with a standard way of proving things about type systems (typically, small-step semantics, progress and preservation lemmas). Soon, at least in academic papers, it became the norm and almost required to provide formal proofs of type safety. Those languages and systems that lacked proofs often had holes in them. Nowadays, like I alluded to in my original comment, the burden of proof (quite literally) is being ratcheted up even higher, with machine-checked proofs becoming mainstream in academic circles. So in that context, your original comment that these systems are "guaranteed" to be sound is not yet at the level of proof that I would accept. I have reasonable doubts, because dealing with UB in C/C++ is still an open problem.

That said, I think there's a large amount of software that's possible to rewrite in reasonable time frames if the authors commit to it.

And software with many individual components can be gradually rewritten. Firefox is starting to use components written in Rust like Stylo and WebRender. Mozilla has an Oxidation project for bringing more Rust-based components to Firefox:

https://wiki.mozilla.org/Oxidation

Sure, but sound static analysis can also be adopted gradually, and being quicker and cheaper, requires a smaller commitment from the authors.

For sure it will increase the quality and predictability of C codebases, and I would definitely recommend static analysis tools in any language. But it's a stopgap measure, it does not solve the root cause. It has to be made impossible to make mistakes, which is what newer languages are trying to enforce.

True, but if we believe the problem is serious and urgent, then a relatively cheap and quick stopgap is exactly what we need.

Except the first hurdle is to win the religous war against such tools, lint exists since 1979 after all.

According to Herb Sutter's question to the audience a couple of years ago at CppCon, JetBrains and ISO C++ surveys, developers and corporations that actually care about static analysis tooling are the exception and not the norm.

I totally agree, but safe languages have to climb a hill at least as high as well, and it might even be the same hill. I just think it's worth mentioning that if this kind of safety is important to you, then there are alternatives that are cheaper than a rewrite. Getting people to 1. care about the problem and 2. accept that there are tools that can help them -- be they safe languages or sound static analysis -- is, indeed, a hurdle.

> Cut down your dependencies on Linux distributions.

I think the author has this backwards. Distributions don't play language favourites[1], and distributions like Debian celebrate the packaging of alternatives to give users choice. The author is right in observing that most C code is shipped by distributions and not elsewhere for packaging reasons, but then later inverts the causal relation.

If good replacements for components [that are currently] written in C appear, then distributions will package them, and the author's problem will go away.

[1] Except that distributions dislike toolchains that embed libraries into builds instead of dynamically linking them, since then they can't issue a security update for a library just by updating that library; a world rebuild is required. This isn't a language or safety issue though, and could easily be changed without changing the language. For example, gcc's golang implementation supports shared libraries now.

Probably someone can post a link to this, as it doesn't seem to exist anymore.

The GNU project used to have a manifesto page from the early days stating that C should be the preferred language to deliver GNU software, except in cases where it did not made sense like libraries for other languages, followed with a list of endorsed alternative languages.

Perhaps that's because it made sense back then, but makes less sense now? :)

It only made sense in the context of cloning UNIX software and its relation to C.

Regarding safety it never made sense. Morris worm is now around 30 years old and lint was designed in 1979, with the expectation that every C developer would actually use it.

> Regarding safety it never made sense.

What suitable and sufficiently performant languages existed back then?

First of all, contrary to urban myths, C arrived 10 years later to the scene of systems programming languages. It wasn't the genesis of them.

There were already computers being developed in high level languages since 1961, most well known languages are ESPOL, NEWP, Algol subsets, PL/I and its variants (PL/S, PL/M, PL/S, PL.8,...), Mesa, Modula-2. And for those with deep pockets on the early 80's, Ada.

As for performance, C was a lousy language regarding optimising compilers, it was only due to UB and lots of hard work that it actually arrived where it is today.

"Oh, it was quite a while ago. I kind of stopped when C came out. That was a big blow. We were making so much good progress on optimizations and transformations. We were getting rid of just one nice problem after another. When C came out, at one of the SIGPLAN compiler conferences, there was a debate between Steve Johnson from Bell Labs, who was supporting C, and one of our people, Bill Harrison, who was working on a project that I had at that time supporting automatic optimization...The nubbin of the debate was Steve's defense of not having to build optimizers anymore because the programmer would take care of it. That it was really a programmer's issue.... Seibel: Do you think C is a reasonable language if they had restricted its use to operating-system kernels? Allen: Oh, yeah. That would have been fine. And, in fact, you need to have something like that, something where experts can really fine-tune without big bottlenecks because those are key problems to solve. By 1960, we had a long list of amazing languages: Lisp, APL, Fortran, COBOL, Algol 60. These are higher-level than C. We have seriously regressed, since C developed. C has destroyed our ability to advance the state of the art in automatic optimization, automatic parallelization, automatic mapping of a high-level language to the machine. This is one of the reasons compilers are ... basically not taught much anymore in the colleges and universities."

-- Fran Allen interview, Excerpted from: Peter Seibel. Coders at Work: Reflections on the Craft of Programming

Interesting fact regarding security, MIT continued with Multics, even after Bell Labs dropping out.

Guess what, DoD later assigned a better security level than UNIX, thanks to PL/I being the systems language.

https://multicians.org/b2.html

The title of this submission ("Fuzzing makes memory-unsafe languages untenable") makes no sense[1] and is not found anywhere in the linked article.

[1] perhaps the intended meaning of the title was "fuzzing shows that memory-unsafe languages are untenable", but that's certainly not the meaning of the current title

It's the tagline the author used on Twitter: https://twitter.com/justincormack/status/1153060402495991808 and seemed clearer than the actual title.

> perhaps the intended meaning of the title was "fuzzing shows that memory-unsafe languages are untenable", but that's certainly not the meaning of the current title

No, fuzzing makes these languages untenable because it provides a tool for automating memory unsafety issues. Without mature fuzzing tools, most of these issues can remain unfound, but fuzzing surfaces them — and their potential for exploitation — rather easily.

It's a bit of a "security by obscurity" thing, but I think there's a point to this view: fuzzing takes the existing crack / fault of memory unsafety[0] and blasts it open so wide you can get a truck through.

> fuzzing makes these languages untenable because it provides a tool for automating memory unsafety issues. Sans mature fuzzing tools, most of these issues can remain unfound,

<Pointy-haired Boss>Fuzzing is now forbidden in our offices. Next problem?</phb>

Jokes aside, Pointy-haired Boss ask why should we care about security issues from the business standpoint. Do we know any company went out of business due the security breach?

Yes, last month for example: https://healthitsecurity.com/news/amca-files-chapter-11-afte...

Not yet, but fines due to GDPR violations might have that effect in the near future. That only requires that a security hole leads to a massive data breach.

Ah, I see. It would help if the article made that argument a bit more explicitly, as I totally missed that interpretation the first time through.

It makes sense to me. The article is all about how a significant percentage of all vulnerabilities are memory-related. With the rise of fuzzing, finding these vulnerabilities has become far easier to do and to automate (you can run fuzzers 24/7 and come up with plenty of attack vectors given enough time). So his final conclusion is "start to explore and invest in ways to replace every legacy C dependency you are currently using. Write a deprecation roadmap. Cut down your dependencies on Linux distributions." And from that logical chain to the conclusion follows the title.

It could be seen as a form of editorializing though, so it could be argued that it should be changed to the original just based on that.

> It could be seen as a form of editorializing though, so it could be argued that it should be changed to the original just based on that.

FWIW it's the title / tagline the author themselves used when they posted the article on twitter, and I thought it was much clearer than the "official" title of "Fuzz rising".

https://twitter.com/justincormack/status/1153060402495991808...

Indeed. I prefer this title over "Fuzz rising" as well.

Fuzzing is a tool that can be used to find vulnerabilities. This can happen by good-intended actors (which is what there is the most data about, as referenced in the post). But it can (and certainly is) also used bad actors. And there is always a bunch of code out there that goes unpatched for extended amounts of time. So even with code that is actively fuzzed and fixed, there are usually vulnerability windows for deployed software. And with memory unsafe languages, we tend to continue to produce such vulnerabilities.

> Linux distros are the de facto package manager for C code, and C++ to a lesser extent

It's not a conclusion I'd end up with, but after a moment of thought, I agree with it.

Is it just me or this pie chart is extraordinarily terrible because the legend is not in order and there's just too many similar colors? And I'm not even colorblind.

A stacked barplot would have been better. And then annotations or another stacked barplot next to it to illustrate the groups the author wanted to talk about. Piecharts are generally best avoided...

The legend seems to be in order for me, the top-to-bottom legend matches the clockwise progression of the slices (from the 12:00 position).

The Color similarity is problematic indeed though.

It’s terrible because there are just too many colors. There is no focus so the graph explains nothing.

This would be easily fixable.

1. Sort quantities in both the list and the chart by descending proportion. 2. Assign colors systematically, e.g a spectrum from red to purple, or shades of primary colors (bonus if you pick better colors for certain types of color blindness).

> Look to smaller more nimble Linux distributions that start shipping memory safe code

Are any distributions moving in this direction yet? Seems to me like C/C++ applications are so foundational to Linux that this would be a huge undertaking. Are there even any memory safe window managers? Hell, even then you're reliant on X or Wayland. PulseAudio/Alsa are also hard to replace.

edit: Quick google search only gives me https://github.com/mesalock-linux/mesalock-distro which is trying to replace common tools with Rust/Go counterparts. I'm not sure how active it is. The main repo only has one main contributor.

> Are there even any memory safe window managers?

I assume "non-toy" is included in there, but there's xmonad at least.

5+ years of using xmonad and never had a crash that I can remember.

Do other window managers crash?

Yes, often. I can't count the number of times KWin has crashed on me. You might not notice, because it automatically restarts -- usually.

Sawfish, which used to be GNOME's one for a couple of years.

https://sawfish.tuxfamily.org/

As for userspace in Go, check https://github.com/gokrazy.

Google has a couple of talks with it.

So what are the alternatives for system programming?

Rust, Golang? Any other true contenders?

I feel like Rust is the most obvious one. I've seen a slew of common Linux tools being rewritten in Rust recently (https://github.com/rust-unofficial/awesome-rust#system-tools).

It really depends what you mean by "system programming".

If you mean things like foundational libraries & network stacks, then go is not a contender[0]. If you mean system daemons and the like, then languages like ocaml, D, … should also work (and that's assuming you want / need the performance of a native binary, if you don't then the world's your oyster).

[0] to my understanding — and I may be completely mistaken here — memory-safe ADA with dynamic allocation and without GC is pretty much an active research field so fails either "memory safe" or "suitable for going fast"

D with its conservative GC may run into random issues on 32 bit platforms. I have seen this effect in some rather trivial programs that should have used only a few MB of memory, but ended up eating up their whole address space and crashing. Not all D programs are affected all the time, mind you. But it's a bummer if it happens.

Having made this experience, I doubt that conservative GC can ever be declared safe, unlike completely precise GC.

D 2.085 added support for precise garbage collector.

It is still not enabled by default, though. I would suspect that you need to audit your code for pointers stored in integers amd other shenanigans before enabling it in production.

> then go is not a contender[0] > [0] ... memory-safe ADA with dynamic allocation and without GC is pretty much an active research field so fails either "memory safe" or "suitable for going fast"

Why would D with a conservative collector be "more efficient"? How does being conservative (with stack roots, presumably) fundamentally make a GC better?

Why would OCaml be better than Go? OCaml is garbage-collected, no opting out.

> Why would D with a conservative collector be "more efficient"? How does being conservative (with stack roots, presumably) fundamentally make a GC better?

It wouldn't? The footnote was for Ada's applicability in the context of "foundational libraries & network stacks" (as it's often advertised as a very safe yet low-level language), because in my understanding (and again I could be wrong here) it's either GC'd and memory-safe or neither, so same as e.g. D, making it unsuitable for that layer.

> Why would OCaml be better than Go? OCaml is garbage-collected, no opting out.

It wouldn't either? The "also" is the second clause indicates that Go would be suitable for "systems daemons and the like", and so would pretty much any other memory-safe language, possibly restricted to the more efficient ones (non-interpreted / JITed) depending on the specific use-case.

Real Time Java on bare metal, Oberon on bare metal (Astrobe), System C#, Modula-3,...

I think the idea was: if by systems programming we mean OS-level or similar, then Go is not a contender, only Rust (from the initial list). However, if by systems programming we mean daemons or other user-space programs, then the list can be extended, from Go and Rust to Go, Rust, D, OCaml etc.

Real Time Java on bare metal, Oberon on bare metal (Astrobe), System C#, Modula-3,...

Guess in what language Fuchsia IO volume management and TCP/IP stack is written on.

I'm pretty fond of Java (OpenJDK) since it also offers portability. Yes Java is much slower than Rust, but it's fine when high performance isn't necessary.

Neither Java nor Go for system programming. Java and its license is also controlled by one of the most evil companies, google at least pretend not to be evil...

Real Time Java proves otherwise. And then there is this pseudo-copy of Java running on phones and IoT as well, both support writing user space drivers in Java.

Fuchsia's IO volumne management and TCP/IP stack is written in Go.

ChromeOS hypervisor (gVisor) and Android GPU debugger (GAPI) are written in Go.

>Java and its license is also controlled by one of the most evil companies

Does that have any affect on OpenJDK?

OpenJDK is GPL, you can take it and do whatever you want in spite the fact that that evil company does like 90% of the work and no one else has stepped up to pick Sun's remains, regardless how Oracle has managed the assets afterwards.

OpenJDK wasn't yet a thing when the other data collection evil company decided they didn't want to pay for licenses like everyone else.

https://en.wikipedia.org/wiki/List_of_Java_virtual_machines#...

None of the companies described there had any issue either with Sun nor with Oracle.

Nor the ones repacking Oracle's work (some of them do contribute a bit to OpenJDK as well to be fair, although usually not JEPs) with their own additional features, https://adoptopenjdk.net/sponsors.html

"James Gosling Triangulation's Interview on Google vs. Sun"

https://www.youtube.com/watch?v=ZYw3X4RZv6Y&feature=youtu.be...

Meanwhile the other vendors listed above do care about supporting the JVM ecosystem at large, so no worries, OpenJDK is alright.

And if Oracle does get bored of it, it would be interesting to see if any of them actually cares a bit more than they did with Sun.

only Rust at this time.

Zig to some degree, but that is still manual memory management. But it do remove some of the undefined behavior.

ATS, Ada

ATS looks interesting. It continues to boggle my mind that Ada/SPARK isn't given more attention in conversations like this. It seems to do everything Rust promises while being simpler (SPARK anyway) and more powerful, and with 40 years of history and experience behind it.

My understanding is that Ada has memory-safety issues (or leaks like a sieve) if dynamic allocations are used and GC is not.

Ada doesn't use a garbage collector, so it appears that understanding isn't right. I think it had one as part of the standard in the 80s, but maybe such problems made it go away.

Nim for sure. It's faster and can generate shared libraries.

Besides Rust and Go, Swift, .NET Native, Ada, D for example.

Maybe this will fuel some much needed advances in static analysis, although the most popular memory-unsafe languages are also very hard to model formally.

I'm getting really tired of these clickbait headlines. Nowhere in this article does it assert the title

Although it uses different words (as noted in an other comment, the title here is based on the line the author used on Twitter), but the conclusion is the same sentiment worded differently:

> Memory unsafe languages are not going to get better, or safe. It is time to move on.

and this is based on an entire article making the case that fuzzing is what reveals the vulnerabilities, and thus both shows and makes memory-unsafe languages untenable: fuzzing by white hats reveals how full of holes these programs are, but fuzzing by black hats shows them what has exploit potential in a much more significant and systematic manner than manual exploit research.

I just read it again, and this comes off as another indirect push for Mirage/Unikernels. I hope it isn't and I'm just being cynical.

(author here, ex-C programmer) its not. Its just against C, and the dependence we have built on it, and the huge number of issues we are seeing now. I don't care what memory safe language you use. A safe userspace for Linux is achievable without as much change as unikernels.

Maybe it would help to write those guidelines here instead of complaining all the time:

https://news.ycombinator.com/submit

It is not like everyone is aware of some magical comment describing the HN submission rules.

> Most of the C and C++ code that causes the majority of open source CVEs is shipped in Linux distributions.

On what is usually a very strict patch review process, with kernel sanitizers and yet....

AFAIK most of the CVEs are not for the kernel itself (and the article doesn't say anything about replacing or rewriting the kernel itself), it's for everything that's built upon it. That's also a point Brian Cantrill makes in one of his talks:

> the safety argument just doesn't carry as much weight for kernel developers, not because the safety argument isn't really, really important. It's just because it's safe, because when it's not safe, it blows up, and everyone gets really upset. We figure out why. We fix it. And we develop a lot of great tooling to not have these problems.

Google has a talk where they mention 68% of them are.

That is why they started the Kernel Self Preservation Project and have been sponsoring cleaning the kernel from C bad practices like VLAs.

Interesting, do you have the link?

Yep, here is the playlist.

https://www.youtube.com/playlist?list=PLbzoR-pLrL6rOT6m50HdJ...

Check all the talks from Google.

Reply on news.ycombinator.com