Almost every developer can fit a bit of discretionary work into their
existing stories. You can polish up the performance
when you first code it.
Almost every employer also has a bunch of things they'd like you to do outside of your stories - mentoring junior developers, recruiting, planning, team building, professional development.
Some employers have SOC2 compliance policies that all changes should be traceable to a documented requirement, so fixes after the change is complete can be a bit more bureaucratic. If the project is behind schedule, the bosses might want you to work on something else instead.
Many developers are given pretty high-spec machines, so they can compile quickly and run weighty IDEs and so on - meaning an issue that's painful for users with 2GB of RAM and 2 CPU cores might be barely noticeable to users with 32GB of RAM and 8 cores.
And of course, some workplace cultures would say if you've 'got your work done' you can do discretionary performance work or you can leave early - and who'd choose performance work over spending time with their loved ones?