Maybe you do. I mostly spend every moment in terror when someone else is driving.
It doesn't mean they are bad drivers, but not being in control gives you a very different view of the situation
I guess since you don't know their "internal state" you don't know how recklessly they are driving actually