The problem here is how tobacco smokers are affecting people around them. Passive smoke consumption is very dangerous, and the smokers tend to ingore any regulations for smoking. Take public bus stops for example, if someone starts smoking while waiting for the bus you usually don't want to start a scene by asking them to stop, and you wouldn't call the police on them, because it's really stupid to bother the cops for such a tiny thing they wouldn't even catch in time.
The workplace can be really affected too. Offices may be free of smoke now, but have you ever been a construciton worker? And hell, even living next to a smoker is really uncomfortable, if you live in an apartment building it's always a nuisance when they smoke and it gets sucked into your place if you have a window open. I just really don't want smokers anywhere near me, but the reality is - they are all over the place, and when you try to limit the places they can partake in their habit or complain they always act like they're the victim and their rights are getting violated.