My, what strange NOPs you have!

Modern compilers may actually do this as well, though for slightly different reasons -- GCC (at the right optimization level) likes to put branch targets and functions at aligned addresses, so you often end up with little pads of NOP instructions scattered throughout your binaries (run 'objdump -d' on something built with 'gcc -O2' to see firsthand, if you're interested).

I actually have a mostly-written LD_PRELOAD library lying around that exploits this for purposes more like the ones you describe though -- hot-patching code in memory at program load to dispatch system calls via little dynamically-generated trampolines so you can insert calls to arbitrary tracing functions. Perhaps I'll polish it up a bit and toss it on github...

Nowadays MSVC supports the /hotpatch command line option, which inserts fillers like "mov edx, edx" or "lea edi, [edi]" at the start of each function for the same purpose.

Out of interest, why does this need a NOP? Why not replace an existing instruction with a branch, bounce down to the end of the code segment, put a longer sequence of code there, put the instruction that you replaced there, and then jump back?

Is it that the patch could be conditional and not incur as much of a performance penalty if the condition isn't met and the patch doesn't run? Would this have made a significant difference to performance?

I discovered Raymond Chen's blog some weeks ago and I started reading all the posts that I could, it is fascinating and very well written.

All the problems that Windows developers had to face for compatibility almost make me sympathize with them. Some context is given in Joel Spolsky's "How Microsoft Lost the API War" (http://www.joelonsoftware.com/articles/APIWar.html), where he talks about "The Raymond Chen Camp".

On PIC microcontrollers, it's common to use a "goto $+1" instruction as a two-cycle NOP to save an instruction word (every word counts when you only have 256 words of ROM and 32 bytes of RAM). It just branches to the next instruction.

See http://www.piclist.com/techref/piclist/codegen/delay.htm

I remember that from the 6811s we programmed in school. BRN also takes more cycles to complete than a standard NOP (or 2 NOPs) since it performs some branching calculations before it ends up doing nothing.

If it was the right thing to do, then it was the right thing to do despite any previous work. See: http://en.wikipedia.org/wiki/Sunk_cost

Circuit-level (layout) problems I'd guess make up a decent portion. I heard (from a guy who worked on it) about a bug in a prototype version of a processor from a major company that didn't cause any correctness problems but was a major performance problem: one of its four cache ways simply didn't have its power rails connected.

Some CPU verification code I wrote on an internship a couple years ago discovered a few bugs in a certain fairly widely-used processor, though I'm pretty sure they were all logic-level problems (i.e. RTL bugs, not circuit level ones)...

- The L1 D-cache tracked clean/dirty status at half-cache-line granularity, and if you did a store (with just the right timing) to one half of a cache line you had just explicitly cleaned with a cache-clean instruction, the dirty bit wouldn't get set on that half line, so as soon as the cache got flushed the data written by the store was lost.

- The prefetcher would shut down sometimes as a power saving technique, but if you laid out the right sequence of cache operations and branches in the last 32 bytes of a 4KB page, sometimes concurrent TLB misses would cause it to not get re-enabled, meaning the processor would lock up, stop fetching instructions and just sit there dead in the water until an interrupt came in (assuming interrupts were enabled).

There were a couple more, but I thought those were the more interesting ones. Granted, these weren't bugs that were likely to be encountered in normal usage for various reasons (in addition to being extremely difficult to reproduce sometimes -- i.e. on one in particular you could run the exact same sequence of instructions from system power-on and sometimes it happened, sometimes it didn't), but bugs nonetheless.

I didn’t work on the 386, and bugs can be caused by anything, but based on bugs that I’ve seen on processors that I’ve worked on, my guess would be that there was some forwarding logic designed to speed up consecutive 16-bit operations and consecutive 32-bit operations, along with some logic to detect when to apply the forwarding logic.

If the detection logic is wrong, you could easily end up forwarding 16 good bits + 16 bits of random garbage into one input of a 32-bit operation. That would explain Raymond’s "if all the stars line up exactly right" line, since the hole in the forwarding logic must have been really small (or it would have been caught in testing).

If it's repeatable, then it's a bug in the logic or microcode of a chip. Some of these could be really extreme corner cases, especially when people are doing things with mixing 16-bit and 32-bit instructions.

It's amazing how much hardware out there is buggy and fixed by operating systems and drivers.

One could make a case against using 0x00 as NOP to prevent a rogue program from NOPping its way across zero-filled pages and into other parts of memory.

Remember that the bits that make up the instructions are not actually numbers - they are switches. i.e. each bit in the number turns on a section of the cpu, then that section of the cpu checks the next bit, and turn on the next part of the cpu, following it down till you get to the proper component that carries out the instruction.

So by having a pattern to your instructions you can optimize the layout of the CPU. XCHG AX,AX does exactly that, with no visible effect, but the action is carried out.

That was then.

These days CPUs compile machine language into microcode, so you can pick any number for the opcode. The microcode is still switches based on the bits, but you don't see that.