Comment by dark-star | Hacker News Reader

Comment by dark-star | Hacker News Reader

Parent Full thread

"backing" is a Virtual-Memory related term. With virtual memory, you can have memory areas (called "pages") that are not really there but only present in the metadata (the "book-keeping", so to say). The first time, someone actually tries to access this page, that access is interrupted ("faulted") and the kernel gets a say in what should be done to that piece of memory (i.e. load it from disk somewhere, reserve actual physical memory and fill it with zeroes, or crash the process).

The "backing" refers to the physical memory that might or might not be present for every "virtual" piece of memory that your program has allocated.

"Freshly faulted" means that a page of (virtual) memory has just received a "backing" by the process above and is, thus, very fresh in physical memory (even though the virtual memory might have been allocated much earlier)

"~10 instructions later" refers to the assembly- (machine-) code, which, contrary to a high-level language like C, usually has long(ish) sequences of rather simple "instructions". 10 instructions is a rather short interval in assembly code.

As for the "localize", the term used is actually "localizes to" which I read as "turns out to be located in" (probably just a bad English translation by the original author)

While this whole summary reads a bit weird, I don't think it is necessarily the result of an LLM, it's probably just that someone who is rather inexperienced at writing up technical summaries did it.

So the theory is that the page is faulted in, then somehow evicted within 10 instructions, then re-faulted in somewhere else, resulting in writes not making it to the page? That would need a context switch and another page fault to happen in short succession. But the context switch to evict the page back out would have necessitated pending writes to have finished. Nothing actually makes sense with that explanation.

Sounded like a race condition causing a correct new mapping in the TLB to be cleared away by mistake, reverting back to the zero page mapping it was before.

Except incorrectly flushing the TLB would merely be a performance bug.

The way I read it was that it wrote to a page and then did a read on that page within 10 instructions and that was not enough time for the memory system to have gone through the process of creating the page backed with real memory and that the timing bug widened in Linux 7.0 such that more things like this exposed the bug. Or there was flapping in the TLB for some reason, or the locking wasn’t correct, or whatever else that the kernel devs will figure out was the cause.

I’m guessing it wrote to a page and read back zero page as the memory was in the mist of being allocated.

But the write should have triggered a page fault immediately in that case, and should have been blocked on the page being actually allocated before resuming.

yes, i went back and re-read the report. an immediate read 1 instruction later works, but 10 does not. the page marked dirty but pointing to the zero page. some context flapping i assume

I understand all the words in it and follow the argument, but it's still bullshit in the technical sense of being written by someone who wants to sound authoritative but doesn't ultimately care if they are right or wrong.

The write-up should have been about five paragraphs: (1) "The script at https://... generates a tree with X million files taking 20 GB total. Running ripgrep 1.2.34, compiled with musl, on my Threadripper 9876 as 'rg ...' crashes about a fourth of the time. With glibc, it does not crash." (2) "This appears to be a bug introduced by commit abcdef1234 in some_vm_call() that allows a race between [...] such that user space can see a corrupt whatever." (3) "The sequence of operations that causes this is: (show two or more kernel threads with lines interleaved to explain the bug)". (4) and (5) as needed to elaborate on those three core points.

But the actual post is long on (fluent) speculation and short on reference to actual code. That's a large part of why it's offensive slop.

Thanks for the explanation. I should have looked up each term a bit more.

So "backing" is the underlying physical memory the page maps to.

So if I were to make sense of that piece of slop, is this what it would be?

"In one thread, a page fault happens during a store operation. The physical memory address is the proper address, which is immediately (around 10 assembly instructions later) replaced by the 0x00000000 memory address by another thread due to race condition, resulting in a crash when the original thread tries to read that page again."

The first thread is the user program, the other thread is the kernel doing things on another core.

Reading from the zero page is legal, that doesn’t cause it to crash. The crash is because of a logic “bug” in the program where the zero it reads back causes some other issue in the program.

I say “bug” because it’s clearly impossible without the kernel messing up. It just stored a non-zero value there.

Nope. It's more complicated. The store operation does not need to trigger the fault, it can happen for other reasons.

What happens here is a bug in munmap() that happens in another thread. It corrupts the TLB, for a brief instant replacing the correct page.

So that the virtual memory page that the crashed thread tried to read its RAM, it gets replaced by the special zero-filled physical page. The kernel uses this zero page as an optimization when it needs to create a region of data filled with zeroes, so it can just map one physical page over the entire range.

It's a really low-level bug that just requires a lot of specialized knowledge just to explain. A better write-up is certainly possible but needs to have a lot of explanations to make it understandable.

Reply on news.ycombinator.com