You obviously still have your own ability to determine what fits your model of the universe and what doesn't. These are just groups of people more likely to have access to information that you might not. You don't have to trust anyone, but in a system where someone has to judge what is true, these groups are closer than the alternative, which would be people with no qualifications, no training, and no history of telling the truth.
If you have another alternative I'm all ears.