Why It’s So Hard to Create New Processors

These days I design Verilog for FPGAs without a verification department. I've learned to code defensively, to use what I know is robust without creating complex corner cases. This often involves clear handshaking and dataflow. Anyway, it's more fun than fixing bugs deep in the weeds of a complex system.

    https://alastairreid.github.io/

Brings me back to 1995 when I was working on the C-Cube CL4010 MPEG-2 encoder processor. A fully custom 32-bit (with some 36-bit registers) RISC architecture.

http://www.w6rz.net/cl4010.png

Are we sure about that? They start out talking about how hard it is to write correct state machines in Verilog, and they follow that with a lot of skepticism about RISC-V processor designs. But most RISC-V designs use Chisel, not Verilog. Chisel is much higher level than Verilog and using it should also make it quite a bit easier to prevent many design-level errors.

Doing a complete verification of a CPU in a strict sense is impossible.

These days we are relying more and more on computational proofs for math and science research.

Therefore - what can we say about the solidity of our scientific knowledge going forward?

Metaphorically, it's like the rush to build auto-autos (self-driving cars): the problem is too hard. If they had started by trying to make a self-driving golf cart Elaine Herzberg might still be alive.

If "verifying hardware is a much more complex task than designing it" (and I don't doubt it) then that is the limiting factor. (Or should be IMO. The liberties taken by software cowboys are bad enough w/o the hardware getting all squirrelly too.)

The goal of UVM is to try to catch all the edge cases in simulation before you fab. The basic idea is to make a model of your design and compare it to your implementation by making a set of random inputs and checking the outputs.

You'll first have to become 'fluent' in EE, but for a physicist, it's just spending the time and getting used to things. Not terrible, long, but straightforward.

As towards what the article is talking about, you need to be trained in it. Honestly, you have to apprentice with the Greybeards (they are mostly men, but not always). There are other ways, like reading through Intel docs or the manuals for ICs or digging through forum posts from 2003. But those guys in the basement with funny newspaper clippings from the 80s or old xkcd printouts are a much better return on your time. They have tons of knowledge about specific chips and machines, stuff that is nearly impossible to recite unless prompted. You just got to spend long lunches blabbering with them, despite their strange political and societal views. Just listen to them, then write down every little thing they said. They are gold in terms of hardware.

Networking for example is the same. If you want to test high scale network equipment OR virtualized network functions, you will buy hundreds of thousands of dollars worth of closed-source testing hardware, software and/or professional services from one of a few big vendors. You will not let anything about your algorithms and designs slip to the outside world, and neither will your test vendors.

Edit: the same is true of most of the software world in general. Sure, you have Microsoft and Google and many others collaborating on Linux, or releasing Kubernetes, VS Code, Go and so on. But the core IP that is key to their business? That is staying in-house, fiercely guarded, developped and tested by an army of engineers.

The main difference is that there are far fewer well-defined software classes that can be tested generally, so it doesn't make too much sense to look for a 'software testing' industry, like you can for hardware. There are some tool vendors, but they offer far fewer guarantees, since it's hard to imagine a product that could find a large proportion of the bugs in both the Haskell compiler and World of Warcraft.

Using the raspberry pi example from another thread: broadcom designed the BCM2835 SoC, which included an ARM1176 core. Broadcom probably didn't do a ton of verification for the ARM1176 core itself, since ARM already verified it.

If by dependent types you mean theorem provers, then that is used, but rarely -- hand-verification doesn't scale to modern processors, usually you model check against some temporal logic formulas that the processor meets its specification. If OTOH you mean using HDLs (= hardware description languages) that use dependent types, then mostly not. Arm's ASL (= Architecture Specification Language) has a tiny bit of dependency build in to reason about length of bit vectors.

And you have to check/debug those 10 million compiler passes at various stages, and each design change may require developing a new debugger or disassembler from scratch to plug into the compiler at each stage of compilation.

What I'm saying is that CPU designs aren't programs, because you can generally trust the compiler to be infallible (and compiler bugs are there, but they're rare). In a CPU process you have to consider the physical impact of the design on manufacturing, what yields you get, how the product is binned, and so on. There are feedback loops between the packaging, testing, and design teams to alter the silicon before production ramps up to go to market. There are tons of moving parts to the actual design process itself, let alone what is being designed.

While you could never release a CPU that didn’t pass the tests you describe, they don’t even begin to exercise all the corner cases for a chip. Multiplying two specific numbers together, while the instruction crosses two memory pages, when an interrupt arrives? How do you even test for that kind of thing?

Once you do all that, you can try carrying out your program as described above. But it guaranteed that the first time you try it, it won't work, because your design will have bugs. Then what?

You need the ability to debug. This means you need to have probes, you need to be able to extract the data, and you need very high bandwidth. You need to have testbenches that are partly in software and partly on the FPGA hardware. Again, that's what the EDA industry will sell you: the hardware and software to do it, as well as the expert consultants to walk you through the process.

And your device needs to interact with the environment. Some of the hardest verification problems have to do with the timing of interrupts; if one comes when the processor is just at the right point, and that case isn't handled in the design, it could lock up. THose cases have to be covered.

Now, for your example of a small core, perhaps it's small enough that you could get it to synthesize and fit into one large FPGA chip and avoid some of these issues. Good luck doing something that can boot Android in that size.

> Start with the simplest possible in-order core so that you get it working very easily, and then evolve to your desired end-state with a series of small commits

You can't just trivially evolve a simple design into something more complex, much in the same way when Linux does a new major release they haven't started with some stripped down basic *nix and worked their way up from there.

If you can actually test on an FPGA (which is usually not possible, and if it is, it's not representative of the actual silicon/analog system you're building anyways), you can get 1T instructions in roughly ~6 hours at 50 MHz. But most hardware emulations are ~1 MHz, so now you're looking at weeks to hit 1T instructions (SPECint alone is 20T).

And what happens when you hit a bug, 2 weeks in? It may not be because you actually wrote new, buggy code, but because a new, higher performance branch predictor uncovered existing bugs. But you'll never know, because the FPGA historically gives you terrible visibility.

But in simulation (where testing is actually done), you're looking at ~1 Hz for a cpu core. Ouch. Obviously a better approach is required (unit-tests against models, formal, etc.), since at the level of detail you can't test much of anything.

> 4. Correctly run several software unit tests that you write to exercise instructions that may not be produced by LLVM/GCC

and

> 5. Correctly run tests you write to exercise specific processor/cache states

These two alone seem like they could be really quite complicated.

Also, "run the world" is pretty slow when you have to do it in simulation (or emulation if you wait to have a netlist to find out how broken it is).

I suspect the coverage from your list is substantially lower than you might expect. Would this have caught F00F? FDIV? AMD Phenom's TLB bug?

People are already doing every step you mentioned but there are three problems:

- Processors nowadays are so advanced and complex that you can't simply approach them as if it was one single block. You need to divide the processor into smaller blocks, develop those smaller blocks and put them together by the end of the development. Like in any engineering problem. The main problem is that it takes a lot of time for all sub blocks to be mature and stable enough for top level integration.

- Once you have all your blocks ready, you can start integration and bring up the system with FPGAs. But now you face the problem that FPGAs are really slow and are not really usable as a normal system. You can run some preliminary tests, the short ones but it would take months to properly execute a normal benchmark.

- You could create and tape out test chips but then you would also need to create all the infrastructure needed for the processor to work like the memory system, memory RAM, communication buses, firmware and etc.

Getting from an initial logic design to a manufactured chip is a big adventure. This requires a lot of layers of software, a lot of it highly specified. And yes, you can buy very good simulators for chips, just search for Cadence Palladium for example. They are huge monsters.