- Albert Einstein
Certainly internal self-consistency will take you a long way if you don't have experiments. Some people find beauty in this :-)
The other things you refer to are still Occam's razor: symmetry is handy because it eliminates symmetry-breaking entities even though we know they can happen in the standard model (Higgs) and "beauty" is really just another way of saying Occam's razor - you'd prefer your theory to not be full of dozens of free parameters because it starts to fit any possible outputs and be less predictive.
At all points the issue is that unless you've fully explored a simpler space with less entities, don't start adding them because you can always keep adding them to solve any problem but predict nothing (ala epicycles keeping geocentric solar models alive. You could probably run a space program assuming the Earth is the center of the universe, but it would be fiendishly difficult to model).
I think it's better to think of most real world models as being low dimensional-ish, where there is a decaying power law of eigenvalues, and most are quite small, though not zero. You can get quite far by looking at the largest modes and ignoring small ones, but you're not exact, so you're not seeing The Truth, or whatever. Forcing your self to use fewer parameters is a way of denoising, however, that is quite effective.
Because you can write a lot of mathematics with no practical applications for generations (then whoops: number theory and cryptography!)
Of course physicists sometimes do make wrong predictions and it can take some time to figure out the hypothesis is wrong. However the goal is always to make something they can test to prove the hypothesis holds, which string theory has so far failed to do.
The TLDR is that you can never expect the same level of certainty when you don't have direct experiments, but you can still rule out _some_ hypothesis, and see how far other hypothesis take you. This is called theoretical physics. Just because you can't make an experiment doesn't mean you can't do anything.