> So there is literally only the goodwill of one single person
Well, not quite. Immunity from prosecution is not immunity from impeachment/removal (which is a political, not judicial, process).
Of course, that has its own problems, but just pointing out that in theory there are more checks than just the one you mention.