You're mixing up a few things here. Every drug has a list of side effects. They are known, that's what your doctor tells you, and then you make an informed decision whether you want to take the drug or not.
It's not Medical Doctors that decided to withdraw approval and then removed it from the market. The approval was all fine. What was not fine was the public perception. Because the anti-vax campaign was very effective. Too many people started to think it's too fishy, so out of caution didn't want the shot anymore, so it became unprofitable to keep it on the market, so the manufacturer pulled it. (A bit surprising they didn't sell the IP or license it cheaply to somebody more adventurous..)
That's how your life can get impacted if the anti-vax lobby gets too strong, and that's what scares me. They can mess up their own health as much as they like, but once my life gets worse because of it, I have a problem.