By the way, I took a graduate course in quantum computation more than 20 years ago. People were equally gung-ho about quantum computation then. Not much has changed in the meantime. Quantum computation remains equally "just around the corner" now as it was then.
QCs also useful for quantum simulations.
That entirely depends on how big N is.
There's also the point that we only know about quantum algorithms that have already been discovered despite the almost complete absence of working quantum computers.
I don't think too many classical algorithms were discovered before we had access to classical computing either. (I could be wrong).
Quantum computers are different, because we have reason to think that they can scale exponentially faster than "follow these steps on pen and paper, but more faster", unlike classical computers.
This very website is using RSA for TLS authentication. There are lots of applications where the longer key generation time and longer keys of RSA are not an issue. RSA is pretty common. Which is a good thing. We don't want to put all our eggs in one basket.
The real applications are in simulating other quantum systems for drug discovery etc.