Trust is a necessity, not authority. Those with authority are often not trustworthy.
It's sort of like seeing a lot of arguments that rely on false or misleading evidence and deciding that "appeal to evidence" is a fallacy. The choice of authority or evidence is the issue, not the act of appealing.