I think you're making quite the leap to state that average people could not have confidence in the system, people seem to have confidence in things that they can't prove all the time.
That's true. Fiat money is a good example. But in this case, losers have incentive to claim the system doesn't work. You want the average person to be able to refute that claim, or else suspicion will run rampant.