It's pretty simple: we stigmatize people whose choices place undue burden on others' ability to live safely.
Driving drunk?
Driving 90/55 on a crowded road?
Indiscriminately discharging a firearm into the air?
Exposing people to typhi salmonella after being established as a carrier?
Exposing people to HIV (and other STDs) knowingly?
Choosing to be unvaccinated against covid?
We seem to have a pretty consistent pattern here, and considering that remaining unvaccinated increases spread and increases everyone's risk of death (especially those who can't receive the vaccine), stigmatization of those who choose not to vaccinate without sound medical grounds fits this easy binary filter.