It has nothing to do with 'trusting them' and everything to do with the threat model because it significantly increases the attack surface area.
Just because I want to grant system access to a relatively simply USB driver doesn't mean I want to grant the same access to a 150MB UI app.