The issue is the emphasis on the importance of an inevitable threat. Organisations start to justify implementing QC-resistant algorithms on the basis that others are doing so, and in that way, they justify themselves that the threat is real. After that, we would end up with government agencies issuing a requirement that only QC-safe algorithms are acceptable for security in the process, killing useful cryptographic primitives like ElGamal cryptosystems, homomorphic encryption, blind signatures and a class of zero-knowledge proofs.
At the moment, there are only three points from which one can extrapolate further advancements. The Shor algorithm requires exponential suppression of errors with the number of qubits. This is why, although we have hundred qubit QCs, the most significant factorisation is being done at most with five qubits. Threfore, if we would have “exponential progress” in advancing QC, the advancement of factoring integers would be linear. Faster progress requires miracles. QC error correction would be no panacea either, as it would require repeated application and would require an unreasonable number of qubits as well as would increase runtime.
The miracle as well be that we figure out a way to factorize numbers on classical computers in polynomial time. Or as well break lattice based crypto on classical computers.