If you need to maintain a certain level of performance, and the system has early indications that you won't be able to reach that by continuing with the current implementation, then that's hardly "premature" optimization.

When people use that phrase, they're usually talking about the nit-picking, obfuscation-inducing stuff that'll get you a 3% increase in speed, that you'd generally do at the end of the development cycle.