If I were you, I wouldn't worry about the "P = NP" case. It is much more likely than an application-specific attack is found.
Although I think quantum encryption will become common place before someone builds quantum computer with enough qubits to factor large numbers.
Certainly there are many systems that will be vulnerable to quantum computers, but very common systems like TLS can be upgraded to ECC without much trouble, and other systems will have to be replaced. It seems unlikely to me that the second quantum computer capable of factoring multiples of large primes will be sold before the majority of vulnerable systems are upgraded or replaced.