It has become a pejorative precisely because it has been bandied about stupidly by people who aren't always arguing in good faith. At best, it's often invoked in an unsophisticated manner that assumes that individual freedom supersedes any other public good--i.e. during Covid.
In the American context, there's also a lot of hypocrisy on the right in terms of using that word, e.g. celebrating post 911 stuff like The Patriot Act and No Fly lists and Guantanamo, not caring about the conditions of immigration detention, but then bitching about having to wear cloth on one's face sometimes. Right-wingers literally cheered Joe Arpaio saying he runs concentration camps. So long as great atrocities befall minorities, they don't care. If they get minorly inconvenienced --> somehow it's a big deal.
I'd say more, but it's hard to discuss properly on phone. I would say there are reasonable libertarian concerns about this bill and Covid policies and a bunch of other stuff, but if you're wondering why so many of us reflexively distrust such arguments, you have to look at the cultural and political context in which we grew up. You're not seeing the full picture, to say the least.