Google assigns a CVE for libwebp and gives it a 10.0 score
stackdiary.com
stackdiary.com
https://github.com/webmproject/libwebp/commit/902bc919033134...
This is also a great reminder that fuzzing isn't a solution to memory unsafe languages and libraries. If anything the massive amount of bugs found via fuzzing should scare us as it is likely only scratching the surface of the vulnerabilities that still lie in the code, a couple too many branches away from being likely to be found by fuzzing.
Yup. For example, the Linux code for its relatively new[1] io_uring subsystem was so memory-exploit-ridden that Google disabled it for apps on Android, and entirely on ChromeOS, and their servers[2]. It is insane that this is how easy it has been for a person to break into Linux.
[1] released with kernel version 5.1 which came out in May 2019
[2] in June 2023: https://en.m.wikipedia.org/wiki/Io_uring#Security
Security exploits from out of bounds access should not be happening today, bounds checking is a solved problem, and has been solved for decades.
So this shows how the majority of WG14 members see security in C.
I would take a large—say, 2-5x—performance hit just to escape these kinds of vulnerabilities.
(Discussed in https://news.ycombinator.com/item?id=37600852)
By adding length counts and explicit bounds checks locally you provide local reasoning to back up this non-local reasoning which is much *easier* to make and maintain correct. I think that would result in code that is less likely to have similar bugs in the future.
If this is a common function in the webp library, this function is probably called something like a trillion times a day.
WebP gets a slight pass because the code is so old; WebP is at least partly based on code from On2 that extends back at least until the early '00s. Rewriting working code is always going to be lower priority than writing new code (and it's not always a safety win either; being deployed for 10 years gets you an awful lot of real-world testing!).
But look at implementations for newer image formats and you see more of the same; C and C++ with the token java implementation that won't get used outside of the JVM.
Maybe WEBP should have been written in Assembly, all the speed and I bet quite safe. /s
https://llvm.org/devmtg/2023-05/slides/TechnicalTalks-May11/...
I don't think there is any evidence to support that bounds checking causes any significant increase in power consumption, and measurements currently available such as shown above show minimal runtime performance impact.
Bounds checking is the perfect situation for branch predictors, 99.9% of the time the branch is predicted correctly, and when it's not it doesn't matter because you're probably aborting the program!
There's also the point that you can opt out of bounds checking as-needed on hot loops if you really need to.
There is no technological reason to not use bounds checking as the default mode of operation today.
My favourite quote, C.A.R Hoare's "The 1980 ACM Turing Award Lecture"
"A consequence of this principle is that every occurrence of every subscript of every subscripted variable was on every occasion checked at run time against both the upper and the lower declared bounds of the array. Many years later we asked our customers whether they wished us to provide an option to switch off these checks in the interests of efficiency on production runs. Unanimously, they urged us not to--they already knew how frequently subscript errors occur on production runs where failure to detect them could be disastrous. I note with fear and horror that even in 1980 language designers and users have not learned this lesson. In any respectable branch of engineering, failure to observe such elementary precautions would have long been against the law."
-- https://www.labouseur.com/projects/codeReckon/papers/The-Emp...
I also wouldn't be surprised if patching, rebuilding and distributing this fixed version of libwebp to all of these devices would be comparable to the extra cost of decoding.
Your intuition about the cost of a patch vs the extra instructions in the hot path to check bounds everywhere is also very much wrong: the cost of a patch (which is also amortized among many fixes) is not even close. The hot path adds up when you call it a lot.
That's the whole point of webp, though.
We just keep track of the table size and make it bigger if necessary.
We additionally have a mode where we just calculate the necessary size, without writing any data structures.
So we have kinda added double safety against this particular bug.
Rust code needs proper fuzzing too. It takes a lot of effort to ensure everything is covered and stays covered as the code is developed. Crashing libraries or applications can be a denial of service. Sure, it's lower impact than an RCE due to a buffer overflow, but it is still a security issue.
It is clear that even if we stood up the best bug finding systems the world has ever seen that critical software will still be a disaster.
It seems like it ought to be really good at this and I am suspicious that people in the know are afraid to talk about it publicly because it's too good at it and once people start weaponizing LLMs for this purpose we just won't be able to use memory unsafe code anymore.
Trivial vulnerabilities are easily discoverable yes -- but, they are also trivially discoverable by standard automation available today. I've found GPT-4 to be shockingly bad at vulnerability analysis for all except the most popular vulnerability classes. My speculation is that there just isn't enough literature on these vulnerability classes for it to have practical mastery of them.
Complex vulnerabilities are the emergent phenomena of multiple events across a codebase and it's dependencies, involving control flow, data flow, while missing type information and other runtime data. Even Anthropic's 100K context windows won't nearly fit it all, and if you stuff all the code into embeddings, the ability to reason across all this space will be poor.
You can train a model to ask very pointed questions about particular snippets, but wholesale LLM-based analysis to find vulnerabilities seems like it'll be extremely slow, expensive and inaccurate.
Inference is so slow, and almost everything about fuzzers are meant to be super fast. Maybe there's a late stage part in crash validation/analysis where you can use it but my bias is that we're just not there yet.
I do not know of LLMs that have been specifically trained on, say, the testing corpus of some "lint" programs and against known vulns. As you point out, it wouldn't be possible as a user of an LLM AI to do the equivalent by showing it some vulns, while it would be perfectly reasonable to get an LLM to write, for example, business case studies by showing it examples.
Interesting quote from Ben Hawkes (former Project Zero manager) in the article. I regularly compile Signal-Android from source and happened to notice they vendored libwebp a few days ago:
https://github.com/signalapp/Signal-Android/commit/a7d9fd19d...
*For the small fraction of Android phones that are new enough to get updates
For me, personally, it's a race to see if Google can get this patched for my Pixel 5 before security updates stop in October.
I actually looked into it, my previous phone was released in June 2020.
The last security update for the phone came out in January 2022.
I'm not sure whether they're just updating things infrequently or whether that model is abandoned altogether, but neither would speak highly of the Android support landscape for non-flagship phones.
This ends up being kind of silly: if I want to have a mostly secure backup device (for 2FA and my bank/eID apps), then I probably need two comparatively recent devices, since a 5 year old budget phone would no longer quite do as a temporary daily driver, in case I'd lose my main phone.
I can say that their rugged designs served me well, but some of the newer phones apparently have this weird component for wireless earphones and I don't really want that, so my current and future purchases are from different manufacturers. Was actually pretty close to stock Android, though!
Looks like this is it [1] and it looks like LineageOS 17 is supported even back to Pixel 3a [2].
[1]: https://review.lineageos.org/c/LineageOS/android/+/366611
[2] https://forum.xda-developers.com/t/lineageos-17-for-pixel-3a...
Pixel 5 should still qualifiy for it.
The security bulletin doesn't reference this CVE specifically but does mention a critical vulnerability that could lead to RCE.
https://support.google.com/product-documentation/answer/1141...
Good luck setting things up in such a way that you only get security patches and not a whole bunch of 'improvements' to go with them.
The reference implementation is C++, and it’s nearly guaranteed to have equally worrisome bugs in it — every image library has seen those over the years.
We live in 2023. We can deal with slightly worse compression until someone rewrites it in a sane language.
not just sanity of implementation but also reliability and compatibility
{h264,jpg,zip} for life!
Like, Google Meet insists on using VP8/9. Why, cause it's "free?" The strain on my laptop and extra energy usage for it to CPU-en/decode video ain't free. Zoom just uses h.264 instead of being annoying about it.
Look at how much fun we can just have with the stack!
std::string_view foo(std::string_view s) {
return s;
}
auto s = foo("temporary"); // kaboom
Modern C++ does not force you to initialize everything before it is read. Modern C++ happily lets you ignore bounds checks with vector operator[] or by using c style arrays or by doing pointer arithmetic. Modern C++ happily lets you overflow integers or silently truncate when widths change. And on and on and on. Turning every "new" into "make_shared" is nowhere close to enough to make C++ safe in the face of bugs.That sandbox is pretty robust - the difficulty of finding an exploit somewhere in a browser renderer is much lower than the difficulty of finding a way out of the sandbox the renderer runs in.
Also, if you want to get security updates without the e-waste, you could just install LineageOS on it.
The image decoders are in android.graphics (https://developer.android.com/reference/android/graphics/pac... ) & that API subset is not listed in the Android Mainline system modules that are catalogued here: https://source.android.com/docs/core/ota/modular-system
It looks as if they planned to include image decoders, but that was dropped sometime during the Android 11 development cycle, unless I’ve missed something (which is certainly possible).
Edit: the final dev release announcement for Android 11 keeps the same paragraph about the native image decoder. So maybe it is there & I’m just not finding it? https://android-developers.googleblog.com/2020/09/android11-...
You don't have to use Rust but you **can't** use C.
There's no reason to be finding these bugs in 2023; period, we can do better and we know how to do better, there's just no reason apart from legacy code (and even then) that you should be using memory unsafe languages in production.And as an aside, this is also why people don’t take security seriously. Not every bounds overflow will result in my database being breached and/or my computer being pwned. Instead of freaking out every time a bounds overflow or some sort of memory error is found, it may be useful to actually find out what the impact is. Is that code sandboxed? Is it local, or does it have access to the web? Does it request elevated permissions at any point? Does it even need elevated permissions?
There are many factors that can reduce or increase the risk factor of a bug. Security researchers need to start actually explaining the risk instead of blindly proclaiming everything as a “critical” vulnerability.
And everything I’m saying here has no bearing on this specific vulnerability. It’s in response to the general claim that every memory bug is critical.
Or they can, I don't know, at very least use C++ with std::array and std::span instead of raw C arrays, with the compiler flags to do bounds checking.
Is anyone working on a Rust version?
And yet I am surprised they chose C as a replacement... hence my question above!
(Edit: I should mention that this is one reason why I really like wuffs, mentioned elsewhere in this thread; it's a safer language that compiles down to C, so you can have mostly the best of both worlds)
This is pretty bad, there will be many vulnerable applications and someone just posted how to exploit it. I just reproduced it on a VM:
SUMMARY: AddressSanitizer: heap-buffer-overflow (/home/<name>/webp_test/examples/dwebp+0xb24e9) in BuildHuffmanTable Shadow bytes around the buggy address: ...
Also, here's the webp it generated as base64, but chrome doesn't crash or anything. Other apps may handle it differently, but many just say invalid format.
UklGRukAAABXRUJQVlA4TN0AAAAvAAAAAPAAWgAAsKwlnZsEAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAALRt27Zt27Zt27Zt27Zt2/b92fUAWgAAsLTknJskSQAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAALw/23oALQAAWFpyzk2SJAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAN6f bT2AFgAALC055yZJEgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAO/Pth7A/xsAQoutz/f3 fwAwMzNvVVXt7p5zLw==
In this context, “memory-safe” means it does bounds checking on an array when you try to access an element. But the webp code does bounds checks up-front so that array accesses can be non-checked, to help performance. (If they didn’t want this performance, they could have easily used a std::vector and used bounds-checked access.) The vulnerability happened because this up-front bounds checking logic was incorrect.
If we were to hypothesize a counterfactual world where webp was written in rust, presumably the devs would have wanted a similar optimization where they did the bounds checking up-front still, and they would have put the actual array access in an unsafe block to have the same perf optimization. The same bug would have thus happened.
The lesson here is that bounds checking on every element access is probably a worthwhile overhead, no matter what the language is. C++ has safe ways to do this (well, safe-ish… safe for the purposes of this bug at least) with std::vector, and maybe they should just switch to that and eat the performance overhead.
This is a big presumption. Yes, it could happen. In practice, doing this isn't even the first tool you'd reach for in this circumstance; the compiler can and will eliminate duplicate bounds checks, so if you've hoisted it early, you shouldn't be using unchecked accesses, even if you care about performance, until you've demonstrated why the compiler isn't okay with removing them. The extra ceremony ("unsafe { foo.get_unchecked(n) {" vs "foo[n]") makes this even simpler to catch in code review.
Right, and in said code review, the webp author could have easily said “yup, we want unsafe here because we already checked up front that the buffer shall not exceed k elements”. Sure it’s easier to see that unchecked access is happening, but when the whole point of large sections of the huffman table code in webp is to make this very thing work, it wouldn’t cause any additional scrutiny in a code review. In other words, it would already be super clear to the reviewer that bounds checking is being disabled for performance sake, seeing the word `unsafe` as ceremony isn’t really adding any information here.
It’s possible webp could have been implemented in rust with a naive approach to early bounds checking and relying on the optimizer to elide it, but having looked at the code, with all the buffer sizes they’re passing around, it looks unlikely that equivalent rust code would have been able to auto-optimize it. I don’t think it’s unlikely at all that, in this parallel universe, they would have found the bounds checking to be a decent enough overhead that they would have reached for unchecked access, and would have passed code review the same way the current code did.
> it wouldn’t cause any additional scrutiny in a code review.
I would hope that it would at least cause a "demonstrate that this is actually necessary" review. People do do this! and sometimes, you do have to use unchecked. A famous example of this is litearally huffman coding, by Dropbox, back in 2016. They ended up providing both bounds checked and unchecked options as a flag, because they actually did measure and demonstrate that things were causing performance drops. I am curious if they'd still have the same issues today, given that a lot of time has passed, and also there were some other factors that would cause me to wonder a bit, but haven't followed up on myself. Regardless, this scenario will end up happening for good reasons at some point, the goal is to get them to happen only when they need to, and not before.
That was for Dropbox specific use case, and 11% seems too much in any case.
For example, what improvements has had rustc in the meantime to better optimize bounds checking, it is still the same, it is less, does it actually matter on the acceptance criteria regarding the definition of done on the story.
You don't have to hypothesize. It's here:
https://github.com/image-rs/image/blob/master/src/codecs/web...
and it doesn't use unsafe indexing.
Meanwhile the real code that exists doesn’t work like that. It’s not just a matter of personal style. Rust has constructs that help avoid bounds checks. It has design patterns that move unsafe code into smaller, easier to verify helpers. It has a culture of caring about safety – people don’t rewrite projects in Rust to just give up on its biggest selling point.
It is not a stretch whatsoever to assume that a direct rewrite of the webp library into rust would have uncovered the same perf findings that Dropbox did, and decided that the pattern of “up front bounds checks, disabled bounds checks at element access” is a reasonable perf enhancement, especially for a Huffman table, where you’re continually accessing the compression table over and over when expanding it.
Edit: the more I think about it, you’re probably right. I had mistakenly thought I was reading C++ code when I was looking at the original patch to the vulnerability. The fact that it’s actually written in C, erodes my argument quite a bit… my logic in my head went something like, “if they wanted bounds checking they would have used std::vector”, but now I realize that’s impossible since it’s not C++. I’ll concede that there’s no real reason to assume that they would have skipped bounds checking if written in a safe language.
https://security.googleblog.com/2022/12/memory-safe-language... shows improvements at scale.
It looks like dropbox experimented with disabling bounds checks in their huffman coding impl, and found that using the unsafe pattern increased throughput from 224 MB/s to 249 MB/s (11%-ish faster.) We don’t even need to hypothesize about whether webp would have elminated bounds checking, we can see that other companies arrived at the same conclusion: Disabling it can be worth it if you’re quite sure you’ve gotten the up-front checking right. We can imagine that if Dropbox went to prod with the unchecked huffman implementation (never mind that that article isn’t about webp in particular), we could imagine they could easily have the same bug. And I don’t think a naive code review saying “unsafe is bad” would have stopped them from doing it: they clearly did the work to show why it’s worth it.
It's already common for PC apps to split potentially unsafe rendering into subprocesses, like in Chrome. If you don't want to pay the full IPC toll, there's shared memory. In theory should be about the same speed as inlined unsafe code, right? What if Rust's "unsafe" blocks could do this for you?
Rust has a pattern of isolating unsafety into small components behind a safe interface, so that the component can be understood and tested in isolation. For example, if you need some adventurous pointer arithmetic, you write an Iterator for it, rather than do it in the middle of a complex algorithm. This way the complicated logic can be in safe code.
It's sort of like Lego, where you build from safe higher-level blocks, but you can design custom blocks if you need.
I can't find the actual code causing the libwebp vulnerability, so idk if mixed safe/unsafe Rust code would've been any better here. Maybe what we really need is an "unsafe-jail" block in Rust that uses a child process limited to a piece of shared mem, and you put big pieces in there to avoid overhead. Like, libwebp can screw up all it wants, just don't touch the rest of my app.
C's type system is not nearly expressive enough for this. You can barely declare a pointer non-null with extensions, but you can't express ownership. You can't force callers of your function to check for error before using the returned value. You can't force correct lifecycle of objects (e.g. Rust can have methods that can be called once, and no more, and then statically forbid further uses of the object. Great for cleanup without double-free.)
C doesn't have ability to turn off Undefined Behavior for a section of code. You can't just forbid dangling pointers.
For a foolproof API the best you can do is use handles and opaque objects, and a ton of run-time defenses. But in Rust that is unnecessary. Use of the API can be guaranteed safe, and a lot of that is guaranteed at compile time, with zero run-time overhead.
For example, a mutable slice in Rust (a buffer) has a guarantee that the pointer is not null, is a valid allocation for the length of the slice, is aligned, points to initialized memory, is not aliased with any other pointer, and will not be freed for as long as you use it. And the compiler enforces that the safe Rust code can't break these guarantees, even if it's awful code written by the most incompetent amateur while drunk. And at run time this is still just a pointer and a length.
In C you don't get this compartmentalization and low- or zero-overhead layering. Instead of unsafe + actually enforced safe, you have unsafe + more unsafe.
These decent C programmers are like True Scotsmen. When top software companies keep getting pwned, even in their most security-sensitive projects, it's because they hire crap programmers.
Even basic boring C can be exploitable. Android was hit by an integer overflow in `malloc(items * size)` (stagefright). Mozilla's NSS had vulnerability due to a wrong buffer size, which fuzzing did not catch (BigSig).
Am I missing something else, like shared mem being slower? Maybe the inability to share CPU caches?
Scheduling, I dunno. Would imagine it's not bad as long as you don't spawn a ton of these.
Android 12 was the first version to support Rust code, and came out in 2021 [0, link talks about the first year of integration].
On the iOS side (which also was affected by this), Swift 1.0 came out in ~2014.
As far as I can tell, Chrome doesn't yet support a memory safe language, but do have a bunch of other safety things built in (see MiraclePtr, sandboxing, etc). Since both WebP and Chrome are from Google, this would stop a possible transition.
WebP was announced in 2010, and had its first stable release in 2018 [1].
[0]: https://security.googleblog.com/2022/12/memory-safe-language...
In addition to the safety features you mentioned, Chrome supports Wuffs, a memory safe programming language that supports runtime correctness checks, designed for writing parsers for untrusted files. I don’t think it existed at the start of the webp project either, but that’s what I would expect the webp parser to be written in, over Rust or a garbage collected language.
That said, not switching over to a memory safe language, in my opinion, is letting perfect be the enemy of the good. Folks will still be able to write footguns, but better language choices will prevent bugs in the all the non-crazy optimized parts.
Many anti-bounds checker advocates, usually miss the first part.
Do bounds checks even matter, even for very tight loops? This article [1] seems to suggest that removing the checks can give up to 10-15% speed increase under some circumstances. Worth it? I'd say no.
[1] https://dropbox.tech/infrastructure/lossless-compression-wit...
None of those are suitable options for an image decoding library on the range of WebP supported platforms.
What makes them unsuitabe is lack of widespread compiler support across those platforms, if we ignore how long Ada has been available in GCC.
And Modula-2 is now in GCC as well.
Well Rust also has unsafe, better not use it.
What makes these unusable for this task?
No. That is true for the list upthread (well, if you consider runtimes measuring a couple of MB "large", it's reasonable but quite arguable). It doesn't have much correlation with any feature.
"Oops"
That is, to abstract, our security issue exists because:
A) There is complex compression/decompression software/code;
B) To implement this compression/decompression -- there are one or more lookup tables in effect;
C) The software implementing those lookup tables and the decompression side of things were never properly fuzzed, bound-checked, and/or mathematically proven not to create out-of-bounds errors, that is, for every potential use of the lookup table to be guaranteed as correct mathematically given any possibility, any combination of data in an input stream.
Also -- Didn't stuff like this already happen in GIF and PNG formats? Weren't there past security vulnerabilities for those formats?
Isn't this just (I dare to say) -- computer history repeating itself ?
Point: Software Engineering Discipline:
If you as a Software Engineer implement a decompressor for whatever format (or hell, more broadly and generically something that uses lookup tables on incoming streams of data) -- then the "burden of proof" is on you, to prove (one way or another) that it does not have vulnerabilities.
Fuzzing can help, mathematics and inductive/deductive logic can help, testing can help, and paranoid coding (always bounds-checking array accesses, etc.) can help, running in virtual machines and other types of limited environments and sandboxes can help. Running as interpreted code (albeit slower) could help. Deferring decompression to other local network attached resources running in limited execution environments could help.
In short... a monumental challenge... summed up as:
"Prove that all code which uses lookup tables can not generate hidden/unwanted states."
https://cve.mitre.org/cgi-bin/cvename.cgi?name=can-2004-0566
Uncompressed bitmaps with absolutely NO extraneous unnecessary "handling code" (which would have existed in the case of the Windows BMP display code, if a security flaw was found in them...).
But... all in all, doesn't surprise me...
It may turn out to be... but I think a more broader generalization of the language /compiler aspect of things may be:
Not so much to "use Rust", so much as "NOT to use C".
That is -- any library performing decompression of any sort (regardless of whether that decompression is related to visual images or not), if it uses C and lookup tables and implements decompression -- should at least be considered a potential source of future problems, and should (ideally) be migrated to a more memory-safe, bounds-checked language -- in the future.
Rust -- may or may not turn out to be this language...
Negatives for Rust -- large size and complexity of Rust's compiler source code.
Positives for Rust -- Rust's treatment of memory and bounds-checking.
Anyway, some thoughts on the language/compiler aspect of this...
If you're willing to count libjpeg-turbo, there's also CVE-2020-17541[1].
For h.264, if you're willing to count Firefox or gstreamer, we're talking CVE-2022-3266 or CVE-2021-3185 (the latter is even critical)
1) Kindly define "serious"...
2) Kindly define "major"...
Also... why can't you ask the question you pose:
"When was the last time there was a serious vulnerability in major JPEG libs?"
...with the words "serious" and "major" removed, i.e.:
"When was the last time there was a vulnerability in JPEG libs?"
?
As that might be a far better question if the generation of insight is desired...
?