Well the salient word in the phrase is "premature". As opposed to optimization after careful observation and measurement, which everybody agrees can and should be done.
If you want performance, you have to design for performance, not just build the first thing that comes to mind and try to optimize it later.