Actually you would worry more about incorrectly implemented padding or a side channel attack or something similarly stupid destroying your cryptosystem. Sad, but true.
You've got execution time (latency), sounds, radio frequencies, electromagnetic fields, etc. just to cite the 'accidental' leak channels that act as a side-effect of the intended infrastructure.
we're not secure.