It's certainly arguable that there is downside for some people. Some vaccine side effects are worse for some people who have already been infected. [1]

[1] https://www.kfyrtv.com/2021/03/15/health-experts-say-those-p...