Worse in what way? Lead is a long term neurotoxin, but pollen triggers the immune system and makes you miserable. Both are bad but in different ways.
Chunks of lead just aren't that bad. https://doi.org/10.2146/ajhp060175 lists common sources of lead poisoning, and none of them are lead weights.
Common sources involve things like airborne lead powder (from grinding or smelting), lead compounds dissolved in food & water, paint, and lead in soil.