Hacker News Reader

Comment by kouteiheika | Hacker News Reader

It's 2023, surely this is not yet another bug related to memory unsafety that could be avoided if we'd stop writing critical code that deals with extremely complex untrusted input (media codecs) in memory unsafe languages?

Yep, of course it is: https://github.com/webmproject/libwebp/commit/902bc919033134...

I guess libwebp could be excused as it was started when there were no alternatives, but even for new projects today we're still committing the same mistake[1][2][3].

[1] -- https://code.videolan.org/videolan/dav1d

[2] -- https://github.com/AOMediaCodec/libavif

[3] -- https://github.com/AOMediaCodec/libiamf

Yep. Keep writing these in C; surely nothing will go wrong.

View on HN

We all know that’s The Price Of Freedom. As they say, “give me exploitable trivial memory safety bugs or give me authoritarian dystopia” (https://news.ycombinator.com/item?id=27638940).

Dystopia? Given a choice, you'd rather trust a compiler than a human given the billions already lost to buffer overflows, read after free, and other memory related bugs, surely?

We don't have a very good mainstream [1] solution for critical and performance-hungry applications. I do think that they are limited in number (many applications are not that performance-hungry) and Rust is already good enough, but those applications need the proof for correctness so that more dangerous code---say, what would need `unsafe` in Rust---can be safely added. This particular bug could have been reasoned once you know where to look at, after all.

[1] In my definition, this translates to "as popular and user-friendly as sanitizers".

> those applications need the proof for correctness so that more dangerous code---say, what would need `unsafe` in Rust---can be safely added

There are actually already tools built for this very purpose in Rust (see Kani [1] for instance).

Formal verification has a serious scaling problem, so forming programs in such a way that there are a few performance-critical areas that use unsafe routines seems like the best route. I feel like Rust leans into this paradigm with `unsafe` blocks.

[1] - https://github.com/model-checking/kani

Yeah, I'm aware of Kani, but it is not optimal for many reasons, including but not limited: hard to install in some cases, loop unrolling is mandatory and does not scale in general (this is a main issue with bounded model checkers, not just Kani), and hard to give hints for efficient verification and/or harness generation. And Kani makes use of CMBC, which is a model checker for C/C++, so Rust is not even special in this regard.

It should be possible with Ada/SPARK. It's focused on low level systems programming and compilable programs are free from run-time errors. In addition, the correctness of the program or key properties can be proven at compile-time.

> "as popular and user-friendly as sanitizers" solution for critical and performance-hungry applications.

Google just did enable Memory Tagging Extension (ARM's hardware accelerated sanitizers) on the Pixels last year. May be sanitizing ISAs offer hope?

MTE is good to have, but it only reduces the probability of successful attacks (by a factor of 16). If the process is allowed to segfault multiple times attackers have a good chance to work around MTE.

That's a general limitation with tagged pointers, I belive? Have you come across demonstrations of such exploits?

Do you have a source for that? AFAIK Pixel 7 uses an ARM v8.2 CPU, which doesn't support MTE. The upcoming Pixel 8 might support it, but Google hasn't announced anything yet.

I'm going by the bugreports I see for a FOSS app I develop: https://github.com/celzero/rethink-app/issues/962

See: https://source.android.com/docs/security/test/memory-safety/...

Why aren't people using safe C compilers or libraries and stuff like that? Do they affect performance that much? If yes then what about libraries written in C so that they can be used in other languages (meaning performance is not the number one concern)?

This is a great question.

The answer is that it's literally impossible to write a "safe C compiler" since the language is inherently memory unsafe.

There are various static analysis tools that can try to simulate C programs and try to automatically discover memory management bugs, but due to fundamental limitations of computation they can never catch all possible faults.

How difficult is it to make a compiler extension that remembers buffers' size and checks if we're overflowing at each access? It could be used at least just in debug versions of critical software.

It doesn't sound impossible to me but I know nothing about compiler development :)

Hard. Apple actually has a RFC for this where functions taking buffer-like parameters are adjusted to take an additional length parameter and then the compiler edits the code to plumb lengths through all of these things to insert a bounds check at use. This can work in many cases, but not all.

Rolling out this sort of change across a large codebase is hard as shit. While it sounds like it is mostly transparent, as soon as you run into a sufficiently large codebase all sorts of things start blowing up that you need to fix by hand before such a feature can be rolled out.

You can also do this with pointer tagging and some other techniques, but without hardware support this is amazingly slow. You can see just how much slower an asan build is, for example.

Apple is basically catching up with the Windows XP SP2 effort, which lead to the introduction of SAL annotations on Windows, and yes it was the reason for its delay.

I think the short answer is "trivial in some cases, impossible in others". It's almost certainly possible that your compiler could inspect every allocation and tag each pointer with it internally. The problem comes with everything else - once you add loops and conditionals the length of that pointer can be all over the place. You'd basically need a symbolic executor tracking every pointer.

There are some big issues with this:

1. It's slow. Symbolic execution involves the interpretation of your program.

2. It would be imperfect and you'd likely have false positives.

3. It would likely be incomplete - for example, how would you handle the situation of only having a header?

So it's a good idea but it's very hard to make practically useful.

The easy way to do it involves changing the ABI of pointers so that they are now (address, bounds) pairs instead of just addresses. However, an awful lot of C code assumes that a pointer is just an address, and changing the ABI in this way will break the vast majority of non-trivial programs. (Witness the difficulty CHERI has in getting major software to work with it.)

You can, it's called valgrind (or more accurately, memcheck). And people don't use valgrind because it is slooooooow. Dynamic checking is useful, but not an ultimate way to go.

> remembers buffers' size

Where?

Once you have a bare pointer, you've lost track of what the original definition might have been, so you (the compiler / runtime / programmer) have no way of knowing that you've exceeded the size.

That's not true, it is merely true on most ABIs. The only case where C really erases this information is casting to uintptr_t and back.

Unless it is clearly specified on the ISO C standard, it is true in practice, and something that is impossible to rely on.

gcc also has some builtins to check pointer sizes when the compiler is able to figure it out.

https://gcc.gnu.org/onlinedocs/gcc-4.1.2/gcc/Object-Size-Che...

Which is why I harp on the idea that the real problem is the gold bricks on WG14 who are intentionally blocking improvements to make C safer.

Also point out that if you can implement C on 16bit 0x86's segmented architecture you can certainly implement C with phat pointers too.

It's trivial but Big Tech is in bed with Big Hacker

Or it's hard like everyone keeps saying.

I'm going with the second option

...What is a "safe C compiler"?

Check on the fiction section sir. Here is the CS one.

Although I think all the C compilers are safe ish lately. I haven't seen exploits that target defects in output. Usually the error is ID10T located in the prekeyboard device.

...a textbook oxymoron?

Not necessarily a "safe compiler" but maybe safe library for containers and things like that. It seems to me that most if not all major C projects just run sanitizers and static analysers.

rustc /s

All decent C compilers have compilation options so that at run-time any undefined actions, including integer overflow and out-of-bounds accesses, will be trapped.

The only problem is that these options are not the default and most C developers do not use them, especially for release versions.

I always use them, including for releases. In the relatively rare cases when this has a performance impact, I disable the sanitize options only for the functions where this matters and only after an analysis that guarantees that events like overflows or out-of-bounds accesses cannot happen.

Despite the hype, by default Rust is not safer than C compiled with the right options, because the default for Rust releases is also to omit many run-time checks.

Only when Rust will change the default to keep all run-time checks also in release builds, it will be able to claim that by default it is safer than C.

For now, when safety is desired, both C and Rust must be compiled with non-default options.

> Only when Rust will change the default to keep all run-time checks also in release builds, it will be able to claim that by default it is safer than C.

Which checks are you thinking of? The only thing that comes to mind is that integer overflow wraps instead of panics, but given that bounds are checked, it is still going to be a panic or logic bug rather than a buffer overflow.

It sounds like you're referring to sanitizers.

1. Notably, some sanitizers are not intended for production use. I think this has changed a bit for asan but at one point it made vulns easier to exploit. These aren't mitigations.

2. They're extremely expensive. You need tons of bookkeeping for pointers for them to work. If you're willing to take that hit I don't really understand why you're using C, just use a GC'd language, which is probably going to be faster at that point.

> Only when Rust will change the default to keep all run-time checks also in release builds, it will be able to claim that by default it is safer than C.

The only thing Rust turns off at release is that unsigned integer overflows panic in debug but wrap on release. That wrap can not lead to memory unsafety.

FWIW it is not recommended to use asan+co for release builds. These are designed as debugging tools, if you use them in production builds they may actually open up new bugs. See also: https://www.openwall.com/lists/oss-security/2016/02/17/9

I don't think anyone has built anything practically usable that is meant for production, though it wouldn't be impossible to do so.

It's more or less okay to use UBSan in production though, and that can be good.

But sometimes DoS is considered an exploit, and in that case you don't want to make things easier to crash.

> the default for Rust releases is also to omit many run-time checks.

...because the type system and borrow checker satisfies them at compile-time?

The only checks that are omitted at runtime are:

- checks that are exhaustively proven to be unnecessary by LLVM - checks that can never be triggered in the absence of UB

You shouldn't be triggering UB checks at runtime. If you rely on these checks, you're relying on UB itself, when all UB should be provably impossible.

Do you release with `-fsanitize=address` or what?

You can't really retrofit safety to C. The best that can be achieved is sel4, which while it is written in C has a separate proof of its correctness: https://github.com/seL4/l4v

The proof is much, much more work than the microkernel itself. A proof for something as large as webP might take decades.

> A proof for something as large as webP might take decades.

Assuming that it is even provable in the first place.

Stagefright was the point when I started to tell people to use a safe programming language.

When they ask "Which one?" I am answering "I don't care, as long as it's a programming language that uses a VM".

Every time I hear the discussions about how fast and perfect C is, people seem to miss the point that new programming languages try to avoid complexity, because they were using C/C++ themselves in the past and they fell on their noses more than once.

It's not about what is faster. It is about how often you make mistakes, statistically. And chances are you make dozens of mistakes per day in C that you will never be aware of, whereas in other memory constrained languages they have hundreds of mechanisms implemented to catch those common mistakes.

You should allow Rust as well, at least.

Yeah ofc. My view on programming languages is that both Rust and golang try to use VM methodologies wherever possible when it comes to memory safety and ownership. They just try to do as much as possible ahead of compile time rather than execution time.

We still don't have an alternative. No, you can't use Arc<HashMap<Whatever>> in the middle of a decoder routine. Some software to actually has to perform well.

In this particular case Rust could have helped, because the table construction can use a normal Rust while the tight decoder loop can remain `unsafe` which can be justified more easily. But I demand more than the human-readable justification.

What Rust brings to the table is static enforcement of its ownership rules, that's it.

How does Rust deal with buffer overflows? Bounds checking. What an innovative solution, congratulations to the Rust people for this groundbreaking innovation. And they keep acting like they've fucking discovered hot water.

Rust is novel because it bundles lots of existing ideas into one usable package. Even the borrow checking is not new, it's adapted and improved from research languages like Cyclone.

No one is saying that bounds checking is novel. It's really C and C++ that are novel, in that they are the only languages where bounds checking is not the default.

But if you're having the "memory safe replacement for C/C++" conversation it shouldn't surprise you that people bring up Rust.

Agreed, the interesting part isn't that hot water exists, it's that given the widespread existence of hot water so many people still insist on exclusively showering with cold water.

I'm not even saying that bounds checking should be used everywhere, just that it really does seem like unsafe shouldn't be the default for so many projects.

Time to repost my favorite C.A.R Hoare quote from 1980, as C was gaining traction.

"Many years later we asked our customers whether they wished us to provide an option to switch off these checks in the interests of efficiency on production runs. Unanimously, they urged us not to--they already knew how frequently subscript errors occur on production runs where failure to detect them could be disastrous. I note with fear and horror that even in 1980, language designers and users have not learned this lesson. In any respectable branch of engineering, failure to observe such elementary precautions would have long been against the law."

I'd say the analogy goes the other way around.

Showering with cold water is obviously safer (no chance of accidentally scalding yourself). But most people prefer showering with hot water because it's the way they've always done it, they're more comfortable with it, and while they could get burned by it, they view the risk of significant damage to be relatively low (if you discover the water is too hot, fix it quickly and you'll probably be fine).

Yea but did anyone else market hot water first? I think not!

Thankfully you can write C-like Rust code if you wanted, just safe. Nobody says that you have to use a "Arc<HashMap<Whatever>>" for this task. Indeed, people have written competitive and safe decoders for various image and video formats in Rust already.

Even if bounds checking made the decoder 10x slower, would that even matter, outside of low end mobile devices? How many milliseconds are spent decoding images in your average website anyway?

If you think Rust is entirely memory-safe, I've got a bridge in Brooklyn to sell you.

Man can make it, man can break it.

That’s like saying we shouldn’t wear seatbelts or have airbags and antilock brakes because you can still die in a crash.

If anything, that’s still underselling it: there are entire categories of bug which are widespread even in well-tested, heavily-fuzzed C codebases like Chrome but would not be likely or often possible in Rust.

Does it have to be? Riding a bicycle with a helmet isn’t entirely safe, but I still prefer to wear one.

Reply on news.ycombinator.com