OP talks about performance optimization, not about "infrastructure work" in general.

The issue with performance optimization is that fast enough systems are ok -- nobody needs the fastest system possible. So its not much of an optimization, really.

If you are spending time scaling stuff that is fast enough, you are not spending your resources well. Naturally, you will not be appreciated. On the other hand, if you resolve a critical scalability issue, you will certainly be recognized as a hero.