brucedawson

It would be easy to have an "auto" clobber list option. The inline assembler would note which registers were touched (mostly trivial, a few instructions have implicit destinations) and would add them all to the clobber list. In about 99% of cases this would be sufficient. I am not aware of any code that conditionally uses a register _and_ conditionally preserves it.

So, 1% of assembly code would use the manual clobber list, but the other 99% would be guaranteed (barring bugs in the compiler) to not have this bug. It seems like the right tradeoff.

Or, instead of an "auto" clobber list the compiler could have a warning if a register is used without being in the clobber list. The programmer could silence that warning in the rare cases where they need to optimize register preservation, and the bugs would be greatly reduced.

Both bugs were programming errors in assembly language files. One was inline assembly that was missing entries from a clobber list, the other was an assembly function that lacked invocations of the macros that were supposed to be used to preserve/restore the registers. There was no compiler bug.

The compiler is making assumptions (which it is supposed to make) but nobody is enforcing the assumptions. The only player who could reasonable enforce the assumptions would be the compiler, in a special checking mode. I am not aware of a compiler that does this. Pity.

In fact the fix to the WebRTC bug was to adjust the clobber list, thus telling the compiler to save the registers.

Why the compiler didn't notice that registers were being used that weren't on the clobber list is unclear to me. Your suggestion seems totally reasonable.

https://chromium-review.googlesource.com/c/libyuv/libyuv/+/3...

You could write an arbitrarily complex analyzer to try to find violations but I suspect that the halting problem means that you can never be sure you've found all errors. I think that either crude heuristics (found two bugs!) or UBSan style instrumentation (finds all bugs in code executed under test) is the best set of solutions.

I was wondering if anybody was going to point that out. It did occur to me that the CHECK was not technically valid due to that exact concern, but given that we control the compiler and the C++ library implementation and given that it's just debugging code (albeit debugging code that we ship to users) I'm fine with it.

In other words, I guess you shouldn't oughta do that generally, but I was fine with it being used there, and it did its job.

IIRC we do block DLLs that aren't signed by either Google or Microsoft in some of our processes. In other processes we can't because third-party DLLs are needed for shell extensions (utility processes) or accessibility (browser process).

And, as you say, code injection is possible without loading a DLL, and does seem to happen.

In this case I don't understand how the state leaked from the file-system filter driver to our process, as it seems to have done. It's a mystery.

I have plans to do the address space iteration you speak of in our crash reporter.

The webrtc fix was thematically similar in that the programmer declared what registers were trashed and then the compiler knows which registers need to be saved. I'm not sure why the compiler doesn't notice when registers are used without being declared as being trashed - I'm really not an expert at _writing_ assembly language.

Exactly. My understanding of the conventions and macros in those source files is that you declare what registers you will be trashing, and then the registers are saved/restored as required by that platform. On Linux it would be a NOP, and on Windows it saves and restores XMM6 and XMM7 (XMM0-XMM5 are volatile).

That would be possible, but we'd have to install the software, then guess which binary was the culprit, and then have some way of finding the function boundaries. My crude analysis technique required on having symbols for chrome.dll to indicate where functions started, so I'd have to have switched tools to something else that could find those.

Correctness bugs are just one type of bug. Performance bugs are another type.

The designation of what should be counted as a bug is less clear for performance bugs but if some operation is slower than it could/should be, and if this slowdown is significant enough to matter, and if fixing the slowdown is quite practical then, yeah, I would definitely call it a bug. I have filed many bug reports for performance issues over the years, and many of them have been addressed.

I develop Windows client software and I need to debug and profile that software, so I need to work on Windows. And, the development tools on Windows are excellent so I'm happy to do that. So, I want bugs like this fixed.

Also, Linux necessarily has a different set of quirks/bugs/issues so "move off this platform because of this issue" is often not worthwhile

Trivia: the Xbox 360 CPU design used 10 FO4, meaning that it could get less done per clock cycle than the Pentium 4. In order words, the Xbox 360 CPU was trying to beat the Pentium 4 at the clock-speed game (fewer stages per clock allows for higher frequency) but the power wall intervened and said "Nope!"

Sorry, I've never seen a non-NDA document with full details. Some information was released but it was a bit spotty.

The material was all proprietary/NDA and I don't think it's ever been shared publicly, sorry.

The main point I wanted to make with how I became the CPU expert was not actually the "reading multiple times" in order to deeply understand it (although that was important) but rather the fact that I chose to become the CPU expert rather than having it assigned to me.

It was one of many times that I became the acknowledged expert of some area, and perhaps therefore in charge of that area, just by working hard in that area until I was "obviously" the expert.

2004 was a bit early for a Netflix binge. Plus, the power was out, so there goes that plan.

The documentation that I read showed the pipelines in great deal, but they were badly presented such that the flows of data and time were obscured. My main contribution was to straighten out and simplify the diagrams, and then animate them. Well, and then explain them. It was definitely a CPU that required that level of detail.

Reading from L2 requires sending down a request and then waiting for the response. This makes me realize that the actual speed is 5.5 mm in 0.625 ns because it's a two cycle delay in each direction. I'll update the article.

But, this request/response means that a ring bus seems unlikely because a unidirectional ring bus should make the sum of the request/response times identical for all CPUs, if I'm understanding correctly.

Having more logical/architectural registers is great except for a few costs:

1) More bits to encode register numbers in instructions. Doubling the number of logical registers costs another two or three bits depending on how many registers are referenced in an instruction

2) Logical registers have to be saved on context switches

3) Logical registers have to be saved around function calls. Either the caller or the callee has to save (or not use) registers, and most functions are both callers and callees. That is, if you are not a leaf-node function then every register you use you have to first save to the stack, or else assume that the functions you call will trash it. Thus, more registers have diminishing returns.

4) No matter how many logical registers you have you _always_ want to have an order of magnitude more physical registers, because otherwise you can't implement OOO or speculative execution.

Point #4 is probably the most critical because I think what people are really asking is why are there more physical than logical registers, and OOO/speculative execution is the answer.

If x87 and MMX registers don't support renaming then that means that they can't support OOO and speculative execution of these instructions. This is possible but seems unlikely to me.

That is, even though not a lot of x87/MMX code is executed these days (and even less is written) I would be surprised if the OOO/speculative-execution of the processor gets halted whenever these instructions are encountered.

Note that x87 instructions still get used in most 32-bit programs because the x87 registers are the defined way for functions to return floating-point results.

Which is to say, citation needed.

Register renaming is implemented in hardware. Because it is used on every instruction it is on the critical path and is probably hand-optimized. Here is some more reading on this topic:

https://en.wikipedia.org/wiki/Register_renaming

Yep, exactly. The rule of thumb is that whenever a register is written to a new register mapping is created. For instruction #1 eax might be assigned to physical register 103. It maintains that identity for instruction #2. For instruction #3 eax can be assigned to, let's say, physical register 67. Then for instruction #4 it could be assigned physical register 92. The only tricky thing is that instruction #3 also reads from eax, which normally would cause a dependency, but because XOR doesn't depend on the register contents this clever optimization applies.

Here's another variant:

> 1: add eax, 1 > 2: mov ebx, eax > 3: mov eax, 42 > 4: add eax, ecx

Here instruction 3 writes the constant 42 to the register. This might make it clearer why instruction #3 can run in parallel with or before instruction #1.

Note that the results must be _retired_ in order, but execution can happen out of order.

Adding more logical registers is compatibility breaking. And, since you have to encode the register specifier in the instruction it means larger instructions (hence the compatibility breaking) which makes reading and decoding instructions slower.

And, regardless of how many logical registers you have you need to have more physical registers. These are needed for out-of-order (OOO) execution and speculative execution. An OOO super-scalar speculative CPU can have hundreds of instructions in flight and these instructions all need physical registers to work on. If you don't have excess physical registers you can't do OOO or speculative execution.

Ouch. Zombie handles are so much worse. Now we're talking serious memory, and since it isn't attributed to the process that leaks the handles it is easy to not realize what is happening.

More details: as I explain in the first link in the blog post, if a process leaks process handles then roughly 64 KB of memory will be used for each leaked handle (each zombie process) but that memory will not be attributed to any process. If you have the Handles column open you will see a large count in one process, but that process will not have a large memory footprint. If you kill that process you will reclaim the memory from all of the zombies.

Good times.

That would _probably_ be safe, but you have to be sure that the xdcbt instructions in the "special" function are far enough into the function that speculative execution can never reach there. Pipelines are way deeper than most people realize so this might require a lot of instructions before the first xdcbt.

And then, for maximum performance you want prefetch instructions to be as early as possible. So, you immediately have a contradiction.

And, assuming that you resolve this there is still the risk that a mispredicted indirect branch could end up triggering an xdcbt. So, you end up with no guarantees anyway.

If Intel ships a CPU with a bug in it then that is an expensive mistake. If they produce a bunch of CPUs with a bug (escape to silicon) that can also be an expensive mistake.

That said: 1) I deal with CPU bugs pretty regularly on Chrome. Some old CPUs behave unreliably with certain sequences of instructions and we get bursts of crashes on these CPUs with some Chrome versions. 2) Intel regularly "fixes" CPU bugs with microcode updates. 3) The Spectre family of vulnerabilities are arguably CPU bugs. 4) The Pentium fdiv bug was definitely a CPU bug.

So, CPU bugs escape to silicon all the time. Way more than I would have guessed just a few years ago. Our industry is built on a pile of wobbly sand.

Making the instruction not speculatable would indeed be a hardware change, which there was not time for. So that was not an option.

And, let's say they did that. All other loads/prefetches are done in the early stages of the pipeline, when execution is speculative. I think they would need new logic at a later stage of the pipeline just for this instruction, in order to initiate a "late prefetch". That is potentially a lot of extra transistors and wires. And, at that point you have a prefetch instruction that doesn't start prefetching until potentially dozens (or more) cycles later. At that point using xdcbt instead of dcbt may just make your code run slower.

What about, then, an xdcbt which is seen in a context where it is known early on that it will definitely be executed - a context where it is not speculative. Well, there really is no such context. Practically speaking there are so many branches that when an instruction is decoded there is almost always a conditional branch in front of it in the pipeline. And, architecturally speaking, any earlier instruction could trigger an exception which would stop execution flow from reaching the xdcbt. Pipelines are really really deep.

TL;DR - On heavily pipelined CPUs (even in-order ones) you don't know for sure that an instruction is "real" until it is time to commit its results, and that is way too late for a "prefetch"

The question of how to properly implement an instruction like xdcbt is interesting. Undoing the damage would be both tricky and expensive. Only doing the L2-skipping when the instruction is executed (as opposed to speculatively executed) would probably be way too late. It seems that such an instruction is probably not practical to implement correctly.

I never asked any of the IBM CPU designers this question (it was too late to make changes so it wasn't relevant) and now I regret that.

Good point. I don't think anybody gave much thought to the title after I came up with it - I suspect it is easy to gloss over during the edit passes.

I chose the title so long ago that I can't really remember what I was thinking.

If the slowdown is sufficiently bad and it's happening on Windows then you can always record an ETW trace (https://support.google.com/chrome/a/answer/9025467) and share it with me. My DMs are open: https://twitter.com/brucedawson0xb I've resolved a few Chrome performance issues due to strangers on the internet sharing ETW traces with me, and analyzing those traces is actually my day job.

Watch for nested loops. That's pretty much it. Well that and be aware that if you have two nested loops then one of them had better be looping through log(n) the elements of the other or less.

The most common quadratic algorithms have two loops that both loop 'n' times, or the inner loop runs between 0 and n times. Both are quadratic - the second case is twice as fast, but still quadratic. In some cases the inner loop is not on 'n' at all but is on something that is typically proportional to 'n'. That's just as bad. That is, O(n*n/100) is still O(n^2).