There was an interesting piece on NPR this morning on "vaccine passports". It brought up the history of the eradication of smallpox in the US up through the middle of the 20th century and a large part of it was making vaccination mandatory for things like attending school and checking this. They didn't formally have vaccine passports because the smallpox vaccination left a distinctive mark so all someone had to do to check was ask you to roll up a sleeve and they could take a look.
They did bring up a risk of vaccine passports that would not have been a risk in the mid 20th century: privacy. With the propensity nowadays to exploit for tracking everything that can be tracked and to tie together all the different databases that contained anything about us, it is quite possible that a vaccine passport system could turn into something more--showing you vaccine passport to be allowed into a store for example might end up being equivalent to showing you full driver's license, credit report, and everything else.
Any vaccine passport system should be designed so that all it reveals is that the bearer had been vaccinated, not who the bearer is. I can't offhand think of a way to do that other than having vaccine passports be active electronic devices which might get expensive.
You also can't implant a gun (an autonomous medical decision) and complain when it's still covered by gun laws.
So if this is "grim", it's been grim since before you were born. Guess that slippery slope takes a while?
In addition, although I don't know if the case would stand up today, the Supreme Court has held that states can require vaccinations generally. (A 1905 case involving Massachusetts and smallpox vaccination.)