QEMU Internals

I built about a third of a NES emulator. The nesdev wiki is mostly decent, although there's a fair number of things where it seems like the first people to figure things out got stuff kind of backwards, and if you flip it, it's a lot easier, that's the sort of fighting the specifications I think you're talking about.

All that said, emulating the CPU was pretty fun. There's a CPU test rom out there you can run with tracing and compare to the published results. I also got the background tiling from the PPU done, but the foreground processing has a lot of steps, so I indefinitely paused for now. Also, I had amazingly poor performance, so I wasn't super motivated to continue.

The 2600 has a very similar cpu, but the very limited Stella output chip means most games are very timing dependent, which means you have to be super accurate, which adds difficulty. I think you should try to be cycle accurate anyway, but it's easy to mess that up, and having some freedom would be nice.

A subset of CP/M calls is a pretty simple "rest of the system" to implement on top of an 8080/Z80 CPU emulation. (It's a bit of a cheat - like qemu's "Linux user mode emulation" or early version of DOSBox, because you restrict software to interacting with a high-level software interface, there are no lower-level details to aim for fidelity with)

Generally "systems-y software" is my background. I joined Google for a semi-experimental operating system project and from there it was a small jump to virtualization when I decided I was interested in doing something else. I'm definitely on the platform side, but been doing some interesting stuff recently :)

Yea, it's streaming so there are some debugging sessions in there, but the information is gold.

Well, Toyota had the sticking gas pedal issue 10 years ago: they did not implement a brake override when the gas pedal was stuck. This was a recommended feature by European manufacturers when they introduced the electronic throttle, apparently Toyota didn't get the memo.

Although I find the GM ignition key issue way worse than Toyota which was an oversight.

Indeed, a Toyota with a critical fatality-inducing safety defect every 200,000 trips would be rightfully viewed as a deathtrap. Given that the average trip is probably somewhere around ~30 miles that would be a fatality per 6M miles versus the standard of ~60M miles in the US, or about 10x more dangerous. However, when comparing a car versus airplanes, given that they both fulfill the niche of transportation and are to some degree substitutable, a more reasonable analysis would be fatalities/person-hour or fatalities/person-mile. For fatalities/person-hour the average flight is something like ~2 hours. In the same amount of time 200,000 cars for 2 hours at an average of 40 mph would be ~16M miles, so the 737 MAX is ~4x more dangerous on a person-hour basis than cars. If we go by distance the average flight is ~500 miles, so the 737 MAX had a fatality per 100M person-miles or is ~1.6x safer than driving. That is just how high our standards are with planes that a plane that is viewed as an absolute death machine that is totally unfit for use is safer than its primary alternative for an equivalent distance. A plane that is 100x worse than any other commercial plane is still better than the non-plane alternative on a per-distance basis.

Obviously, this does not excuse their actions as they still made a system at least 100x more dangerous than the standard, but it should give perspective on the difficulty of the problems actually being solved. It is not a bunch of amateurs or below-average engineers who need to adopt basic practices. It is a bunch of highly-skilled professionals developing systems with a level of reliability far beyond what most software developers even think is possible. Even the abysmal processes of the 737 MAX that are far below the standard in the airplane industry would, relative to most software, be very good. It is just that the problems they need to solve are very, very, very hard and very good does not cut it when lives, not data, are at stake.

One failure per 3.8M hours would be once per 433 CPU-years, so they probably actually do have somewhere between 10-100x that failure rate for their CPUs given that expected CPU lifetime is probably around 20-30 years. Even using a much more reasonable 2 hours per flight that is still ~45 CPU-years so still within the likely range of expected CPU errors. Also that is a comparison against a system so dangerous that it is unfit for use instead of the actual standard which is once per 50,000,000 flights or ~250x better.

Even ignoring that, I am discussing the uptime of a system using AWS which only guarantees 99.99% uptime for AWS service in any given AWS region and only a 10% refund (which is less than their profit margin) as long as they keep your system up more than 99% of the time. Downtime for a system due to AWS downtime in a region constitutes a critical failure of AWS to deliver expected service. That their lack of service does not result in deaths unlike an airplane is immaterial to a reliability analysis, it only tells us if their critical failures matter and what level of reliability we should require/demand when making reliability-cost tradeoffs. In other words, the probability and costs of failure are not actually related. It is just that costly failures result in more effort being spent on developing mitigations. In the case of airplanes, critical failure in the form of a crash is very costly, so they take great pains to minimize the whole-system risk of that failure mode.