If living a 'natural' life is higher on your value scale than preventing a possible infection, then you wouldn't want the vaccine even if no one else was getting it either.
Let's say your neighbors want to build a $1M lake. They want you to chip in $50k. You're going to get the 'benefit' of the lake one way or the other. But you don't even like lakes. In fact, you'd prefer they didn't build a lake, as lakes attract mosquitoes, and you are (perhaps irrationally) concerned about infections from mosquitoes. Would it be selfish for you to not chip in? Would it be rightful for them to take the $50k from you by force since you're getting the 'benefit' of the lake?
Edit: I'm not saying people should be forced into taking the shots or anything. Just that I'll laugh at people who use reasoning that doesn't begin to make sense to me.
So the argument has been lifted from the individual to society. The next logical step might be the species. There may be a strong argument against immunization for the sake of human kind. Are our immune systems still evolving? Will we need rely on technology to replace the improvements that were previously made by natural selection? If not, is the human race going to lose to a virus we can't defeat with medicine? Short term gains, and all that jazz.