This is true, when you keep optimizing, you soon face death from a thousand paper cuts. But often, it's enough to find that bottleneck and make it a few times faster.
The sad fact is that a manager won't approve you working on something that'll save 1% CPU. But once the tall and medium tent poles have been knocked down, that's all there is left. There are hundreds of them, and they double or triple your response time and/or CPU load.
I've had much, much better outcomes by rejecting trying to achieve an N% speedup across the entire app, and instead picking one subject area of the code and finding 20% there. You deep dive into that section, fulling absorbing how it works and why it works, and you fix every problem you see that registers above the noise floor in your perf tool. Some second and third tier performance problems complement each other, and you can avoid one entirely by altering the other. The risk of the 1% changes can be amortized over both the effort you expended learning this code, and the testing time required to validate 3 large changes scattered across the codebase versus 8 changes in the same workflow. Much simpler to explain, much easier to verify.
Big wins feel good now but the company comes to expect them. In the place where I used this best, I delivered 20% performance improvements per release for something like 8 releases in a row, before I ran out of areas I hadn't touched before. Often I'd find a perf issue in how the current section of code talks to another, and that would inform what section of code I worked on next, while the problem domain was still fresh in my brain.