It wouldn't be so bad if the orbiter didn't have such big wings (due to a military requirement for a large cross-range flight ability that was never actually used). Big wings means a more difficult reentry scenario, which means exotic, and brittle, thermal protection systems on the wing leading edges, which means a much larger area of highly vulnerable thermal protection materials on the orbiter. And it wouldn't be so bad if the vehicle didn't use liquid Hydrogen, which is super-cryogenic, requires excessive thermal insulation, and is extremely prone to formation of ice. A lot of people think that we were unlucky with the loss of Challenger and especially with the loss of Columbia, but in truth it was the opposite, we had gotten extraordinarily lucky prior to that. In reality the chances were very high that we would have lost a vehicle to either of those failure modes very much sooner.

But yes, it's a very troublesome design for a lot of reasons.