The problem with DDT was that while it didn't cause significant problems for people (at the concentrations used), it caused huge problems for birds (thinned shells). At the same time it wiped out the mosquito carrying malaria along the gulf coast. The ecologic side effects were terrible, but in many cases human risk may well have been reduced.
Some asbestos is extremely dangerous, some is not (it depends on their chemical composition and microstructure)... using the same name for all of them is not helpful to safety, nor is declaring all of them dangerous.
There is Plutonium in the Pacific ocean. Should we guard the ocean, ban swimming in the ocean, ban fish caught in the ocean? Why, when sun exposure is much more likely to kill you with carcinoma than ocean plutonium?
I definitely am still going to go outside during the day even though the dangeous sun is irradiating me. I'm going to eat toast with dangerous acrylamide.
...And I'm not taking off the dangerous lead weights ballencing my tire rims on the drive to work, because I want to live.