I die a little inside every time I lose an argument over whether or not we should spend time optimizing our software (I lose that argument every time I have it).
Most places are so far behind on priority tasks, that to slow down and discuss/debate the priority of tasks is likely to just cause more issues.
It isn't that you are even wrong. You almost certainly aren't. But nobody will wait for things to get better. If you stall out on features while prioritizing, you will lose folks.
Unfortunately, I don't have a legit answer/solution. My view is it is a lot like keeping a clean kitchen. That is table stakes and assumed. Such that folks in a kitchen have to create a somewhat self cleaning flow of things. Similarly, you have to constantly optimize and clean up the code as you go. Double down on what you have, and constantly work to improve it. Not to replace it.
It doesn't mean "don't optimize" at all. It means: optimize those parts that actually make the application slow. I can tell from experience that intuition has repeatedly failed me about which parts those are. Measure, don't guess, before you optimize.