Your example of driving privilege is inapt. My hypothetical privilege to drive is not rejected based on my ability to take care of myself. It is rejected on my ability to take care of others around me.
Obviously there's a difference between defecating on the sidewalk and recklessly injuring someone with a car, so the question becomes how we define harm.