All vaccines have a nonzero risk of adverse outcomes. That can be as simple as the risk of physical injury due to incorrect administration of the needle. That's weighed against the risk of nonvaccination.
In almost all cases, the balance is so extremely skewed that it's safe to issue general recommendations in favor of vaccination. For HPV, the issue is that there's literally zero benefit after exposure, and it's difficult/infeasible to test for exposure, but from epidemiological data we know that most people over the age of 25 have already been exposed.
> In my opinion men should also be vaccinated for it
The recommendation is for all genders, and has been for well over a decade.