So many years ago I was involved in an issue with a BTS (a cellular radio base station) for a telecoms equipment manufacurer. The software was written in C, and had a message parsing architecture between processes, with fixed length queues, which discarded messages when they overloaded.

There were various problems (the subject of a number of war-stories, which I won't repeat now) - but basically anything that wasn't using a properly designed state machine could die horribly when messages started getting discarded.

So we did some work around the fundamental design of the queue handling mechanism. We came up with a software modification which would improve the efficiency of the queues handling, which would stop the queues overloading (which we saw happening occasionally at peak loads on the biggest BTSs).

We could measure various aspects of the queue handling performance, and use queuing theory to prove that the performance would be better at very high loads.

We had a test rig and we could test the relative performance of the new software vs the old software at normal loads.

What we didn't have, was a test rig which could run at the highest loads, so we couldn't test the behaviour of the new algorithms at very high loads.

So the improvements were binned. The existing software was known to fail (badly) at extreme loads. The new software could be tested at reasonable loads, and mathematically be shown to improve matters at very high loads, but because it couldn't be tested, the process said that it must not be deployed!