Except if you need a certain level of performance. It would be silly to write a million lines of code to find out that you could never reach the level of performance that was actually required.
But in general you are right.
I'd like to add that perhaps the most difficult aspect of software engineering is a situation in which specs change all the time. This happens a lot during prototyping, and afterwards, when a prototype has been promoted to a product.