If you haven't built trust with every person on your team, that doesn't make you a shitty manager. The more people you can work with and trust, the more effective you may be as a manager, but don't let a counterexample serve as proof positive that you suck. Trust is a two way street.
Sometimes people underperform for other reasons that have nothing to do with you. They may have a different perspective, or they may just be having a bad day. Or they may have family or health problems affecting them. Or, they may actually just no longer care or not be good at their job.