I struggle with this a lot with Marginalia's index. Where I identify a hot method in a prod profiler run, try to replicate it on a test machine where I can never get the same cache characteristics because everything in this space is like an onion of caching layers that you affect the real performance of the system. I may get it to run significantly faster, but that only sometimes makes the production profiler sample move its needle.
e.g. I've recently been experimenting with using a cursed hybrid model in Marginalia's index, where based on a mincore probe, I switch between mmap and io_uring for reading a cluster of pseudoadjacent data. There are real tanglible benefits both in the test machine and in prod with this, but the numbers do not agree at all about how the needle moves :P