The reality is that everyone is prepared to allow others to die for their own convenience. I intentionally phrased that harshly, but it is indisputably true. In the US, 70,000 people died from the influenza season of 2018-2019. We could have sharply reduced that through quarantine measures. But no one wants to do that.
Once you accept that everyone is prepared to allow others to die preventable deaths for their own convenience, then all you are talking about is a matter of degree. It can't be immoral to believe 1 death is okay, but 2 is wrong. That's just an opinion. The only difference between believing 70,000 is okay, and 700,000 is not is a gut feeling about how much preventable death is worth it to maintain our living standards.