That's the difference.
If we had a plague as bad as the Black Death, then yes, it would be deadly.
People will use the death numbers as counter-arguments, yet most deaths (94%-ish last I heard) are from people who had comorbidities, like obesity and such.
Until there is the same logic being applied to obese people as to the unvaccinated, as well as counting previous infections, especially for healthcare workers, I personally will not apply the same logic of smoking and drunk driving either.