Using the largest number factored as a benchmark for progress in quantum computing is like evaluating floor(f(0)) = 0 and floor(f(1)) = 0 and concluding f(x) = 0. You can't distinguish f(x) = 0 from f(x) = x/2 from f(x) = e^x/3 when your test is too coarse.
If you want to see the progress in quantum computing today, pay attention to component reliability. For example, in 2014, the best rep code run on a quantum computer had a 1% error rate per round [1]. In 2024, it was 0.00000001% and it had become possible to run full quantum codes with a 0.2% error rate per round [2]. If it takes another decade for that 0.2% to go down by a factor of a thousand, then you've got lots of time. Maybe you'll be dead before progress exceeds the coarseness of factoring. If it takes a year instead of decade (because the cost of error correction is frontloaded) then, well, pay appropriate attention to that.