So to summarize you: rights are not inalienable but dependent on if people get annoyed.
Do you see how philosophically weak that position is? Should Muslims be deported because the US government finds them annoying and people 'never chose to live next door to them'? Rights must be grounded in something other than 'popular opinion'.
I'd rather live in a world where I simply have to buy/rent in an HOA which everyone signs a contract stating that we will abide by XYZ rules rather than one where we have to rely on the threat of murderous government.
Government's proposition to you is: Let me kill whomever I want and I'll make your life marginally better. I find it ethically unconscionable to benefit so marginally at the expense of someone else's life.