It improved from then. We evolved unit tests and full system tests. A release would have to undergo a full simulation to show it was not only problem free but that it still had profitable results that look correct. We had test/debug shims for glibc functions that may cause syscalls, such as time or direct/indirect locale (such as from a printf), that would error on use in code to prevent sneaky costly timing things jumping in.
We did have one rather insidious error where the risk was too strong and preventing some orders going out. It took us a few months to track it down as it was making the system feel clumsy but it was still working. We assumed the market had changed a bit. Cost us a few million and it was a mistake from our best software dev. It didn't change the fact he was our best software dev (he's at Google now). Lesson learnt was to check for the positives and not just the errors.
We introduced a number of risk check evolutions. A separately developed shim was the final check on orders to make sure that regardless of risk the order made sense, e.g. not a zero price. This code was structured to be as independent as possible from code in the main system. That saved our butt a few times. Adding timing throttles was also important to prevent the system reacting in a way that would send a silly number of orders per sec. We also evolved to giving the broker html web pages where they could view the risk in real time and control risk if necessary. This often included integration with broker risk systems and taking their risk files and integrating the constraints to our engines.
At a later firm, we had automated test systems that also ran performance tests with hardware like that which Metamako now provides. The unit tests on code check-in would run not only the unit tests but there was a suite of performance tests where each would reconfigure the network and run things with external performance measurement on the network. This would allow us to track performance bumps of tens of nanoseconds to specific code deltas. Very useful indeed. A slightly customised version of graphite allowed us to see the chart of the performance of all components and tests over time.
Further to this, we evolved to a specific kernel, OS, and bios settings being replicated so that we could reproduce exactly a production system and vice versa. Tuned BIOS and Linux kernels became important.
The risk controls, unit and systems tests are probably the most important things an HFT does. YOLO is very true.