It’s really unclear to you why a long term neurotoxin that leads to lifelong disability is worse than allergy producing pollen?
Chunks of lead just aren't that bad. https://doi.org/10.2146/ajhp060175 lists common sources of lead poisoning, and none of them are lead weights.
Common sources involve things like airborne lead powder (from grinding or smelting), lead compounds dissolved in food & water, paint, and lead in soil.