I think it is really hard to know when you should optimize and when not to. Most of the time a prefer to design my code in such a way that is not fully optimized but it can be with a few tweaks. There are many places that might not be needed and it has no sense in optimizing.