Pretty much every position has its own sort of moral defence.
My issue is the conceptual mistake of ignoring the "unknown unknown" risks you impart on others when you accept one set of "unknown unknowns" yourself. It is a sort of hypocrisy that allows you to dodge the shame of a private decision.
In the end, when rejecting this vaccine, you have to fully own the "shameful aspect":
Taking the vaccine is an act many are taking on behalf of others, and in doing so, accept a private risk for a public good. We ought rightly praise this.
Rejecting the vaccine is a rejection of the priority of this public good which many ought regard as shameful.
Personally, were I so inclined to reject it, I'd simply say: "I do not care about society so much" -- and it would be true -- "shame me all you like, who cares". But the latent hypocrisy here and doublethink makes me believe many are unwilling to say this -- because they dont believe it -- and hence my issue.