It's about common sense.
Police/fire/ambulances are there for emergencies, their drivers have better training (theoretically) in safe driving, and the vehicles bring attention to themselves.
Uncle Jim Bob trying to drive his Buick around is what's obviously prohibited as that's the vehicle/driver most likely to cause harm...