Sure but those other folks can continue to isolate themselves to a greater degree, in line with their own risk assessment. I am not sure I understand the case to curtail one person's rights for another's benefit.
Thats pretty much the entirety of laws that involve safety. My right to drive 150 mph is curtailed for the benefit of people I might kill.
The point of having society, and of having governments, is that cooperation allows us all to lead longer, healthier, more fulfilling lives. You can't just discard the necessity of cooperation in specific instances where the benefit you personally receive is lower than the amount of effort you need to put in.