Wouldn't that just make those few oracles an increasingly valuable target for manipulation/corruption?
Of course not, and when they screw up, we lean on a robust (yet imperfect) legal system to intervene.
One of the advertised features of smart contracts is a reduced dependency on trust, so if they don't actually have that advantage (at least in relation to real-life interactions), then it's hard to say they're any better than traditional legal contracts for those use cases.