The US federal government once expended 10 percent of the US's electric energy supply in the Manhattan project for getting weapons grade nuclear material. This gives a rough ballpark for the amount of energy they are willing to invest into major strategic advancements.

However, if you apply Landauer's principle, current factoring algorithms would require enough energy to boil all oceans on the earth, that's a lot even compared to the US's energy supply.

So algorithmic improvements are the real danger basically. Even if we discovered a decryption method now, and immediately everyone stopped using RSA, there would still be an immense impact because all the past encrypted traffic that someone might have stored somewhere suddenly becomes decryptable. And usually, traffic from 20 years ago is still relevant today.