OpenBSD on Zenbleed | Hacker News Reader

OpenBSD on Zenbleed

marc.info

71 pointsstargrave60 comments

OpenBSD only implemented loading AMD firmware two days after AMD published updated microcode to fix Zenbleed. Which makes me believe they were not among the "major kernels", vendors or other entities that got a heads up of this vulnerability which happened over two month prior.

Whether they were last to be in the know or not, i applaud them for being one of the first to have patches out for their latest two stable releases (7.2 and 7.3).

Don't know if it was still the case, but OpenBSD would not get early vuln info because they wouldn't sit on embargoes and would patch right away.

It's possible they knew just enough to know that they needed to implement firmware loading, without knowing the full details of the vulnerability.

This is untrue. OpenBSD pushes to release as early as possible, but if they're on an embargo, they've respected it.

Eh, it's more or less true. OpenBSD violated the KRACK embargo in 2018. They decided to publish for the benefit of their users, and fuck everyone else.

Incorrect. They had explicit permission from the researcher involved to commit when they did. Here's the full discussion, if you want to read it yourself.

https://marc.info/?l=openbsd-tech&m=152909822107104&w=2

More directly, from the KRACK FAQ: As a compromise, I allowed them to silently patch the vulnerability.

https://www.krackattacks.com/#openbsd

It's also worth noting that Microsoft violated the embargo as well: On this topic, it is also worthwhile to mention that Microsoft pushed their fixes on patch Tuesday on 10 October 2016 [1]. That's before the agreed disclosure deadline, albeit quite close in time.

Quite rightly, nobody is suggesting that nearly a decade later, we should be keeping Microsoft off responsible disclosures as a consequence.

I mean, I just disagree with that interpretation of 2018 events. The embargo deadline was extended, OpenBSD violated that. They refused to play ball with the coordinated extension. The researcher is orthogonal to that. Their actions in 2018 are certainly not universally celebrated as a good way to participate in an embargo.

You don't get to extend an embargo unilaterally. Now that there's a new uniform 90 days the conversation looks different.

They asked for permission from the coordinator and got it. It's that simple. There's nothing open to interpretation here.

The next time you coordinate a disclosure, you can run it differently.

Why do people blame OpenBSD but not the coordinator is beyond me.

This subject of this article is OpenBSD; it's reasonable to be discussing OpenBSD. I'm not saying that guy is blameless.

Technically correct I suppose - https://isopenbsdsecu.re/mitigations/embargoes_handling/

But they do have a relatively difficult history with embargoes. This isn't criticising them BTW - although I don't use OpenBSD any more I still have a soft spot for them and respect for everything they've achieved.

Rather difficult, in the sense that people continue spreading falsehoods about their relationship with embargoes, which makes it difficult to participate in responsible disclosure.

See this thread for examples.

There is further information from de Raadt on impact and mitigations [1]. Hearing that the microcode fixes from AMD does not cover all CPUs that are likely to be vulnerable is not great. Reading “[W]e are setting DE_CFG bit 9 on all the models that we think have the bug” is comforting and exactly what I would expect from the OpenBSD developers, as it follows what happened back around Heartbleed and it is one of the workarounds mentioned by the Zenbleed security researchers [2].

[1]: https://marc.info/?l=openbsd-misc&m=169025404406996&w=2

[2]: https://lock.cmpxchg8b.com/zenbleed.html#solution

Worth re-posting Theo's 2007 note about CPU security bugs again:

https://marc.info/?l=openbsd-misc&m=118296441702631&w=2

My hunch is that as they suspected these types of issues is what guided them away from things like AVX and other optimizations.

OpenBSD was also one of the first operating systems to disable hyper-threading by default due to all the related security issues with this technology. Yet another case of security over speed.

Hyper-threading, at least in the beginning, had limited if any performance improvement when enabled. Windows (XP, 7), at least, was faster at work with hyper-threading disabled.

I'm still waiting for benchmarks which show more than a 5% increase in performance with hyper-threading enabled.

If you haven't seen a >5% performance improvement for any workload with HT/SMT enabled, you're just willfully blinding yourself to benchmark data. It can be closer to 80% than 5%, at least for some reasonable workloads.

They also have a limited number of developers and no interest in performance.

“[N]o interest in performance” is probably a bit unfair, no? It makes it sound like any patch that brought solely performance would be rejected. But it is certainly true that they have much less manpower than the “competition” and that performance takes a backseat relative to other priorities.

No interest in performance means that, no? If they did what you said and rejected performance patches they’d be actively against performance.

No, it just means it’s not the top priority.

If the performance patches would make the code less secure, they would not implement it.

No, it's fair -- they're actively disinterested in patches that improve performance.

> On Linux, glibc has AVX-based optimizations for simple functions (string and memory copies) which will store secrets into the register file which can be extracted trivially, so the impact on glibc-based systems is HUGE.

Interesting. I would have expected it to be some amount of worse performance not using AVX. Though perhaps the past throttling effects from AVX-512 and friends made it so it was too complex to manage which instructions to select in BSD.

OpenBSD is not a fast system.

Nor does it needs to. Safety is better than performance anyway

"Safety is better than performance anyway"

... unless you need performance. What a vacuous statement!

Rust would love a talk with you...

I am a Rust guy, sorry.

Then you know Performance and Safety are not mutually exclusive.

They still are mutually exclusive. For example, you don't have built-in bitfields in Rust. You have fat objects that requires heap memory to work properly. Still, there are many kinds of metrics for performance. While the execution performance of Rust is fantastic that it reaches the figures on the ballpark with C and C++, and sometimes even better due to fundamental stuff like borrow rules and strong alias rules and immutability propagation, the developmental performance of Rust is abysmal at best :[ Bunch of concepts to understand, generic and trait bounds, async and Poll, crate orphan rules, ownership and lifetime and although my prior C++17 knowledge did help, but consider in the picture for a complete newbie, they are no easy feat to challenge.

The compilation performance of Rust is also unsatisfying too with right even though it got the package manager right. I remember sitting 15 minutes to build my first ESP32 RISC-V firmware in Rust and still have to wait 2 to 3 minutes for consecutive debug builds. All of that on a Ryzen 3700X, maybe machine issue idk.

It's trade-off. I have backtesting cluster. No inbound except SSH in-office. Run with `mitigations=off` on Linux. So far all my apes gone nowhere.

OpenBSD isn't materially more secure than Linux, these days.

They've shipped several RCEs over the years, in software included in base, but because the software is not enabled by default they don't update their security slogan.

It gives the impression of an OS that is free of vulns which is not true. The "default install" is not useful for much as IIRC it only has ssh enabled which means their claim of only two remote holes in $time_period is really just a claim against their TCP/IP stack and OpenSSH, not the entire OS.

It's deliberately misleading.

I'll take performance over safety any day of the week.

configure comes to mind as a great example of this.

It used to be very slow ( or for some people fast enough ). But I wonder if that is still the case in 2023. This reminded me nagging phoronix to do another BSD reviews.

Depends on the architecture. On Sparcstation 20 circa 2000 OpenBSD was way faster than linux.

Full pipe string copies are not a frequent task for any OS.

String comparisons are, though.

Large memory copies, however, are quite frequent.

For what purpose? That seems pretty inefficient.

Yes, it does come with worse performance (for those functions).

>"OpenBSD does not use the AVX instructions to the same extent that Linux and Microsoft do"

While I love OpenBSD and what they do ... I have to admit, I get frustrated because many times OpenBSD is immune to security vulnerability simply because they don't implement modern tech advancements like AVX.

Not being as vulnerable doesn't make OpenBSD more "secure", it just makes them behind the times - like riding a horse & buggy in a world that's quickly evolving to electric vehicles.

More like a walkable city in a world that is quickly devolving into noisy, polluted, car-dependent cities. You may find that over a decade or a few hundred years negative externalities of a technical decision outweigh the benefits. When AVX dies a painful death in the next decade, projects like OpenBSD won't feel as much pain from it.

Having such lousy implementation bugs show that at least AMD doesn't care about it anymore, I'd wager there is something better coming round in the near future.

I think this is a bit short sighted. In the end, x86 will be replaced entirely, should openbsd just ditch it altogether?

If you think AVX will die “a painful death in the next decade” you’re not really in a good position to call AMD’s implementation lousy.

You might need a box to be as safe as possible, but performance is secondary. Following your car example, some people prefer a safer car over a faster one.

If you chose Open BSD you make that explicit choice. Their promise is to avoid technologies unless they are rock solid, even at the cost of performance or convenience (like the forking to LibreSSL).

> While working on our fixes, I ran the test programs for quite a while and I never saw anything resembling a 'text' string. However when I ran a browser I saw streams of what was probably graphics-related fragments flowing past. The base system clearly uses AVX very rarely by itself.

So it seems that your statement is only true for the base system.

Those days the web browser is the main path for RCEs on a user's desktop. For example firefox on a linux system uses around 25 external libraries, if not more.

Or that just mean the rest of the industry is careless about their security and unrespectful of their users and prefer racing the marketing and performance arm race.

You could also read it as "Other operating systems are way to quick to adopt new features without fully considering the security ramification".

Software running on OpenBSD can clearly use AVX, but the kernel and core OS doesn't yet, which security wise seems to be the smart move.

Look at SMT, that's still disabled on OpenBSD I believe, which indicate that Intel and AMD have yet to devise a way of implementing it safely. You can enable it, if you need the speed more than security, but I doubt that many does.

>"Software running on OpenBSD can clearly use AVX, but the kernel and core OS doesn't yet, which security wise seems to be the smart move."

That's not accurate. OpenBSD does use AVX, it's just very minimal.

What's the "smart move" then here? That OpenBSD just never got around to implementing AVX more thoroughly?

If you're implying that OpenBSD had always considered AVX a security issue, then why are they still having to patch for it (and why did they adopt some use of it)?

Skimming through various OpenBSD mail list, I can't find any past threads discussing their concerns about AVX prior to ZenBleed.

https://www.openbsd.org/mail.html

> That OpenBSD just never got around to implementing AVX more thoroughly?

Realistically that probably the right answer, they didn't have the developers nor did they priorities going in and just retro-actively fitting AVX in everywhere where it could potentially help with speed. The "smart choice" deliberately or not, is to not just jump onto everything new but adopt new features at a slower pace.

My reaction was towards against stuff like this:

> I get frustrated because many times OpenBSD is immune to security vulnerability simply because they don't implement modern tech advancements like AVX.

I don't get that. Sure part of it is might not getting around to implementing it, but there's also an implicit choice in not just going in and adopting new features everywhere just because you might need it. OpenBSD developers doesn't seem to have viewed AVX as being something that needed to be prioritized. Otherwise it would have been in more places.

But you're right, they do use AVX, that's clearly a mistake on my part.

FWIW, my original post was mainly fueled by the fact that AVX was announced in 2008 and in both Intel & AMD chips by 2011.

So it’s not like AVX is a “new” development. It’s 12-years old.

(Not directing these comments at you. Just trying to expanded upon what drove my original post)

https://en.m.wikipedia.org/wiki/Advanced_Vector_Extensions

That's honestly also a little older than I expected. It is a little weird I'll admit. It would be interesting to know if the slow adoption in OpenBSD is deliberate or done on purpose.

My understanding was that Zenbleed code runs in userspace therefore it doesn't matter if kernel/libraries use AVX optimizations or not. That jab against Linux sounds like blame shifting and moving discussion away from the fact that OpenBSD did not offer microcode update at all.

I don't agree, I think it's a fair assessment of the impact.

For data to be exfiltrated through this exploit, the sensitive data must first be loaded into an AVX register. On GNU/Linux this happens all the time, because strcmp et al use AVX. On OpenBSD this is not the case, so only applications that themselves are loading sensitive data into AVX registers would be vulnerable.

Anyway, OpenBSD users have the new microcode now, while most Linux users don't (I checked Arch, Fedora and Ubuntu, none of them has updated their packaging yet).

Zenbleed's impact is due to userspace (including library) use of AVX registers. Glibc is part of userspace. OpenBSD libc is not glibc. It's a valid difference.

I don't think this is an example of foresight / good design choice, though. They just got lucky that no one cared enough to do performance optimization on their libc string functions.

"OpenBSD does not use the AVX instructions to the same extent that Linux and Microsoft do, so this is not as important.

On Linux, glibc has AVX-based optimizations for simple functions (string and memory copies) which will store secrets into the register file which can be extracted trivially, so the impact on glibc-based systems is HUGE."

Perhaps I missed something, but it appears musl does not use AVX instructions much if at all.

> it appears musl does not use AVX instructions much if at all.

Right, or at least, not explicitly (a sufficiently smart compiler might be able to auto-vectorize it). The author is (reasonably) suggesting that most Linux systems use glibc, rather than musl.

Here's musl:

strlen: https://git.musl-libc.org/cgit/musl/tree/src/string/strlen.c (64-bit words at a time)

memcpy (generic): https://git.musl-libc.org/cgit/musl/tree/src/string/memcpy.c (32-bit copies, unrolled to 4 at a time for a suitably long copy)

(Throwing the generic memcpy into godbolt -O3 shows Clang does choose to implement this in part with XMM and YMM registers: https://godbolt.org/z/dh6hqG77j as does GCC: https://godbolt.org/z/aKraqPaPs )

memcpy (x86_64 asm): https://git.musl-libc.org/cgit/musl/tree/src/string/x86_64/m... (rep movsq -- 64-bit words at a time, implemented by the CPU microcode)

Remember that the first disk device in the hardware tree (sd0 or wd0) might not be the disk you're actually booting from. Take a peek in your dmesg output first before installing new bootblocks to be sure you'll be getting the new microcode loader.

Reply on news.ycombinator.com