I think you mean computational efficiency will go _up_ in the future. To your last point: Jevons paradox might apply.
if computational efficiency goes up (thanks for the correction), and CPU inference becomes viable for most practical applications, GPUs (or accelerators) themselves may be unnecessary for most practical functions