When I was at Google there was a "rules of thumb" page that described how much of your time X performance was worth. I always looked at that before micro-optimizing, but also always came out ahead. I remember a coworker and I redesigning one of our subsystems late on some Friday evening; the rules of thumb said that the performance improvement we predicted, at our scale, was worth a month of SWE time. We did it in 2 hours. So we came out ahead, and our system worked better. (I joked that my colleague and I would be taking 2 extra weeks of vacation.)
TL;DR performance matters everywhere. The one user using an interactive tool on their workstation will appreciate their day not being wasted by random pauses. The person out and about on their phone will appreciate the additional battery life. And your company's bean counters will be quite happy to hear that your cloud bill or data center expense forecast for next year is down. Finally, it's fun! Truly a win/win. Make it fast!