Because you have to build the quantum computer in a world that is overwhelmingly classical.

There are no qualifiers along the lines of "underlying phenomena". It's simply difficult to get a stable enough interface between the classical and the quantum, so you can control it, while at the same time isolating it enough that it doesn't decohere to classicality.

Who knows, maybe reliable scalable quantum computation truly isn't feasible for some reason, but if you study the physics, the fact that this is so hard is not really a surprise.