Well it depends what you are doing. If the thing you are optimizing is the thing blocking everything else routinely, spending a lot of resources on optimization is a no-brainer. If the thing you are optimizing takes long, but that does not matter so much, it is not worth optimizing.
When you are creating VFX scenes for a still image you don't really care a lot about the render times as long as it is done the next day or so. If you are doing it for animation, you care a lot about render times, because anything times a thousand will be quite a duration.
The work of a programmer is always a factor in a multiplication. If the small change you make is used by thousand people a day, using it 100 times a day each second you shave off will save a collective time of ~28 hours. And this is only time. You could also think about electricity, about frustration, etc.
A programmers work is always multiplied into the world. It is a huge responsibility - and I think we all should act more like it.