You could try the opposite: give entirely random people the ability to vouch.

I have no idea if this would work, but.. I've been doing lots of ML lately where it is often pretty hard to beat random, and throwing noise into a system make it much more robust.