> "bandit" AKA government.
No, sorry. If you're thinking using excessively coarse categories like "governments," you're going to get things pretty wrong in most cases.
If that's too hard, then try this: if you think the government-as-bandit is bad, it'll be a heck of a lot worse when you're dealing with bandits-as-bandits, who don't even theoretically have any accountability to or responsibility for you.
Individuals are extremely weak; groups are pretty much always stronger. If you don't want to bullied as a weak individual, you need to band together with others, but then you're subject to how the group governs itself.