The debbie-downer perspective would be that for each additional qubit you add, you effectively double the cost of isolation from the environment, quantum error correction schemes, etc.
So, while compute power for quantum algorithms grows exponentially in N, so would the cost of operating the machine.
Do people who work in QC see this as a concern? Are there scientific arguments or engineering insights that lessen or obviate this concern?
For your enjoyment and amusement, here is a QC-related show-HN!
"An elementary proof of a key lemma in Shor's quantum factoring algorithm": http://gregfjohnson.com/qft.html