Leaded gasoline is an issue because it's a very bioavailable form of lead. It contains tetraeythyl lead, which is extremely volatile and likes to absorb into your cells.

Solid lead, on the other hand, is much more inert. The main danger of working with elemental lead is breathing in dust if you're grinding it.