I think the main thing is: quantum computing doesn't really work right now, at all.
Imagine if you had crummy, unreliable transistors. You couldn't build any computing machine out of them.
Indeed, in the real world progress looked like:
* Useless devices (1947)
* Very limited devices (hearing aids)
* Hand-selected, lab devices with a few hundred transistors, computing things as stunts (1955)
* The IBM 1401-- practical transitorized computers (1959)-- because devices got reliable enough and ancillary technologies like packaging improved.
In other words, there was a pattern of many years of seemingly negligible progress and then a sudden step once the foundational component reached a critical point. I think that's the point of the person you're talking to about this.
And then just a couple of years later we had the reliability to move to integrated circuits for logic.
If you looked at the "transistorized factorization record" it would be static for several years, before making a couple steps of several orders of magnitude each.