Right, and there's truth in what you say - especially wrt extremism. Your argument brings to mind the argument that "capitalism isn't the problem,
crony capitalism is". Like, if only powerful people wouldn't lie, cheat, steal and write the rules to apply differently for them.
I will concede that, on a limited basis, government - specifically in the regulatory (and law enforcement) capacity - can do some good.
I'm more pessimistic than you though and believe that the vast majority of new laws are not written for my benefit, and most of the time we - the society as a whole - would be better off without more rules. It's not about my trusting my fellow citizen more than the ruling class, my position is about my distrust of and opposition to busybodies and cronies.
If you believe - for whatever reasons - that laws are not written nor enforced for your benefit, then maybe libertarianism isn't the worst thing. And that's all I was driving at.