We have factored some small numbers on quantum computers but nothing a classical one couldn't do in a split second. https://en.wikipedia.org/wiki/Integer_factorization_records / https://arxiv.org/abs/1805.10478
Regarding the „except possibly contrived problems designed to be fast on quantum computers“ part: That’s their entire purpose. They cannot and will never be faster for all applications compared to a classical computer. They are designed to solve some very special problems efficiently, such as solving dlog and RSA using Shor‘s algorithm or database search using Grover.
Shor's or Grover's aren't contrived problems, they're real problems which is why they're interesting. And none of today's quantum computers can run these for non-trivial inputs.
The scalable way to factor is using Shor, but Shor only becomes practical once we have thousands of qubits, and it breaks RSA2048 with about 20 million qubits. (For clarification, a "qubit" here is a noisy qubit.)
Yep, it's definitely a scaling thing. If it weren't then we could simply use Daniel Bernstein's (et al) proposal for Post Quantum RSA.