The main benefit of building quantum computer is verification of correctness of quantum mechanics. There is a good reason to hope that quantum computers won't work allowing us to create better theory.
Shor's algorithm working for large numbers would rule out some of interpretations of quantum mechanics like objective collapse, something that have not happened yet.
In any case my argument was not about any particular interpretation of quantum mechanics, my argument is that building quantum computer allows us to test our understanding of physics in conditions where it have not been tested yet. And so even if that did not produce any additional benefits the testing itself would be worthwhile.