Without empirical evidence, I'd always be skeptical that we really knew what was going in inside some unobservable region of spacetime, even if the math all works out and agrees with what's observable. Theoretical physics divorced from the need for experimental results gives us endless string theories. It's the same arguing for some interpretation of QM. You might have the most convincing arguments for there being many worlds making up the wavefunction, but without some way to confirm that, it's metaphysics. And one mights well go all the way and embrace Tegmark's mathematical universes or some simulation argument. But that isn't science anymore.

I don't care how good the math looks. Reality isn't obligated to confirm to some human aesthetic about beauty or simplicity. You have to first make some metaphysical assumption that the universe has to be that way. And the only way we can really know is to have empirical evidence.