I had a memorable experience of that, actually - for a long time we had a Trello card titled “faster to operate on the bits instead”, and the team looked at that card fondly during development. After we started getting traction, it was finally time to optimize! We conscientiously wrote tests to ensure correctness, and then began an incredibly enjoyable informal hackathon. We blazed through every step, replacing the logic piece by piece with bit operations. This step was 2x faster, that step was 5x faster, and so on. A few steps were slower, which was a bit weird, but we were having so much fun shouting out our gains to each other across the room, racing to improve. By the afternoon we had optimized each step and gotten an order of magnitude improvement.
“Great stuff, team.”
“I thought it would have been faster.”
“Yeah, same.”
We did some evening profiling and noticed a lot of time being spent in a dozen or so functions with names like “toRepr”, “reconstitute”, stuff like that. Actually it was like most of the time - what!? And then we saw, at each step we converted to raw bits, performed the equivalent logic with fast bit operations, then converted back to the representation for handoff to the next step. Wait, why we are converting to and from bits like thirty times each run? That can’t be right. Realization strikes - right before most handoffs there was a commented-out line that ran out tests on the representation to ensure it was still correct.
Someone pointed out we wrote that test before we had done any investigation of performance (I remember it almost word for word actually, “When we wrote the test, all we knew was it would be faster in bits, but not how. And the test doesn’t even look at bits!”). So we ripped out all the conversions and stayed in bits from step to step. Without changing the bit logic it was now almost three orders of magnitude performance gain, basically running in milliseconds instead of seconds. Two OOM that we had earned, but left on the table!
Tests introduced arbitrary checkpoints and imposed irrelevant constraints on the problem, without us noticing until afterwards. We caught the problem pretty easily that time, but I do wonder how many other times we (or others) didn’t because it wasn’t as obvious. The service that accepted the output maybe should have been accepting raw bits instead of the representation. The service that gave us the input definitely should have been staying in the bits.
So, do test your optimizations for correctness, but be really careful. Tests can easily and subtly constrain your optimization space in major and arbitrary ways!