Article "How to factor 2048 bit RSA integers in 8 hours using 20 million noisy qubits" https://arxiv.org/abs/1905.09749 shows how to significantly reduce the cost of factoring integers and computing discrete logarithms. Implementation of the efficient device mentioned in the article requires 2.7 billion Toffoli gates for 2048 bit input. It could factor one key in 8 hours.
Microprocessors reached that high MOS transistor count less than 10 years ago. For example, 8-core Core i7 Haswell-E has 2.6 billion transistors (2014). If quantum computers follow Moore's law like conventional IC, it will take 40-years before they can factor 2048 bit RSA.
Note however that quantum computers can potentially benefit from existing lithography processes, so it may not take as long to scale up as we have already developed processes for very small highly integrated circuits.
The devise itself can be huge, mostly because it's cooled, but the actual quantum state inside is just tiny chip.
Yep, it's definitely a scaling thing. If it weren't then we could simply use Daniel Bernstein's (et al) proposal for Post Quantum RSA.
We have factored some small numbers on quantum computers but nothing a classical one couldn't do in a split second. https://en.wikipedia.org/wiki/Integer_factorization_records / https://arxiv.org/abs/1805.10478
Regarding the „except possibly contrived problems designed to be fast on quantum computers“ part: That’s their entire purpose. They cannot and will never be faster for all applications compared to a classical computer. They are designed to solve some very special problems efficiently, such as solving dlog and RSA using Shor‘s algorithm or database search using Grover.
Shor's or Grover's aren't contrived problems, they're real problems which is why they're interesting. And none of today's quantum computers can run these for non-trivial inputs.
The scalable way to factor is using Shor, but Shor only becomes practical once we have thousands of qubits, and it breaks RSA2048 with about 20 million qubits. (For clarification, a "qubit" here is a noisy qubit.)