Hacker News Reader

Comment by 2bluesc | Hacker News Reader

Interesting walk through in the linked video as he tried to troubleshoot this:

https://www.youtube.com/live/1UnoBfw6soI

Pretty shocking to see (extremely unlikely) non-malicious code work / not-work depending on a security mitigation setting.

Curious to see where this goes as to whether it's a kernel bug and nobody is paying attention to `mitigations=off` now or the unlikely outcome that it's an actual hardware bug where mitigations work around it and nobody has noticed.

Seems he had overclocked and maybe disabling the migation has revealed an instability in his setup that's otherwise stable (or at best marginal).

View on HN

He says that he tried stock voltage for everything, and even undervolted/underclocked CPU/memory and still had the bug.

mitigations=off does a lot of things, it's a bundle of options

Perhaps the next step is trying to figure out which mitigation exactly causes it to fail. Then kernel peeps should have a fighting chance of tracking this down

In the video comments it looks like he's done some of that and it's one of the spectre v2 mitigations. Makes me wonder if there's now less QA happening on the oem side without mitigations=on.

SMT off fixing it is a good clue. It heavily points to the problem being within a core rather than between execution cores. That's still a broad area though. Within a core some state is inadvertantly shared between the shared threads running on it but there's a lot of possibilities of what that could be.

Some state being in the CPU or kernel? I guess if you have per core state in the kernel that's only accessed by that core, then you're safe to modify it without locking, but this isn't the case for data shared between hyperthreads of a core?

Within the CPU core (terminology here gets awkward). Spectre in its simplest to exploit form used the fact that the branch prediction based state wasn't completely cleared between running different threads on the CPU and led to a really easy timing attack. Time your own code to see which way the branch of the previous code likely went. There's a lot of CVE's around this but that's the simplest case.

The mitigations work by making sure to clear everything they can when the CPU switches contexts. This includes clearing loaded local CPU cache, clearing the branch prediction table, clearing all speculative execution entries. The mitigations likely avoid a crash by clearing something that should be cleared between context switches in all cases. As in the mitigations likely accidentally fix this. Which is a good clue. It'd be nice if the mitigations were more fine grained than all or nothing so we could turn on the mitigations one by one but the microcode is not open so we can't do this :(.

Is it likely the issue is within the Linux kernel itself or within the microcode for the CPU?

I'm going to say microcode until someone reproduces on a completely different architecture :).

A lot of these mitigations are joint efforts between the kernel and CPU, and the mitigations kernel parameter isn't a binary on/off, you can configure individual mitigations. It's not a single flag that does signals the microcode to do spooky proprietary things.

Mitigations can have very large effects on system stability, I ran into this with my old Haswell system. Mitigations on allowed, I believe, 100Mhz higher at lower voltage - but it may have been 200Mhz. These settings were tested over many months, completely stable, and mitigations off with the same setting wouldn't even allow booting. Huge effect relative to anything else, basically like adding an extra 0.1V vcore.

What does he say at 8:17?[1] Arsink Overism?

[1]: https://www.youtube.com/live/1UnoBfw6soI?si=R8nJ1FxdE4zBuO0i...

"rsync algorithm"

Reply on news.ycombinator.com