Comment by rincebrain | Hacker News Reader

Comment by rincebrain | Hacker News Reader

Parent Full thread

The simplest way to do this, what I believe memtest86 and friends do, is to write a fixed pattern over a region of memory and then read it back later and see if it changed; then you write patterns that require flipping the bits that you wrote before, and so on.

Things like [1] will also tell you that something corrupted your memory, and if you see a nontrivial (e.g. lots of bits high and low) magic number that has only a single bit wrong, it's probably not a random overwrite - see the examples in [2].

There's also a fun prior example of experiments in this at [3], when someone camped on single-bit differences of a bunch of popular domains and examined how often people hit them.

edit: Finally, digging through the Mozilla source, I would imagine [4] is what they're using as a tester when it crashes.

[1] - https://github.com/mozilla-firefox/firefox/commit/917c4a6bfa...

[2] - https://bugzilla.mozilla.org/show_bug.cgi?id=1762568

[3] - https://media.defcon.org/DEF%20CON%2019/DEF%20CON%2019%20pre...

[4] - https://github.com/mozilla-firefox/firefox/blob/main/toolkit...

That would tell you if there's a bitflip in your test, but not if there's a bitflip in normal program code causing a crash, no? IIUC GP's questions was how do they actually tell after a crash that that crash was caused by a bitflip.

The example I gave in there is of adding sentinel values in your data, so you can check the constants in your data structures later and go "oh, this is overwritten with garbage" versus "oh, this is one or two bits off". I would imagine plumbing things like that through most common structures is what was done there, though I haven't done the archaeology to find out, because Firefox is an enormous codebase to try and find one person's commits from several years ago in.

But it would be also possible that sentinel value used for comparison changed because of bitflip, not data structure used by program.

Detecting that is the point, yes.

This doesn't always protect against out-of-bounds writes. Although if these sentinel values are in read only memory mappings it probably gets pretty close. (Especially if you consider kernel memory corruption a "bitflip".)

Well, yes, the author described it as "a heuristic", which checking sentinel values would seem to match.

[4] looks like it's only a runner for the actual testing, which is a separate crate: https://github.com/mozilla/memtest

(see: https://github.com/mozilla-firefox/firefox/blob/main/toolkit..., which points to a specific commit in that repo - turns out to be tip of main)

Yes.

I pointed to the runner because otherwise it seemed likely I'd get a comment insisting there's no proof it's used in Firefox.

Reply on news.ycombinator.com