You still have to fill it up by touching and breathing in the fumes while pumping it into the tank, so not exactly, but you're right that it does affect others as well.
:)
Naively (without looking at data for concentrations in various environments), I could buy that people regularly handling / being around leaded aviation fuel are worse off than those breathing the (dilute) exhaust products.