Casey does say that almost always you don't need to "optimize", but rather just need to think about the execution of your code on hardware. This video isn't quite as nuanced, but he does talk about this in other ones.