As for new algorithms, surely this would be more of a logarithmic increase for any specific problem as it gets harder and harder to improve on existing knowledge. Has, say, sorting become exponentially faster since Tony Hoare invented quicksort in 1959? My guess is no.
They're like a CPU: highly programmable, flexible, multi-purpose. This means that when any single "part" of the GPT inference code is running, it's probably using only 10-40% of the die area while the rest is not used and idle.
Imagine a future where a nearly-optimal architecture for LLMs has been figured out and stops changing every few months. At that point, it would be possible to make a dedicated chip that does "nothing else but that". It would be 100% utilised and faster because it would have more units dedicated to the specific task at hand. The data paths could also be optimised to minimise wasteful trips to-and-from external memory, etc...
The ideal "inference chip" would be something akin to an FPGA. Huge numbers of simple cells that can do just the dozen-or-so instructions required, with local memory for the weights and biases. Make these on a cheaper process, run them at 500 MHz for power efficiency, and tile them out by their hundreds in a hybrid memory+compute architecture.
It would be relatively easy to get a factor of 10x, and I suspect 100x is possible.