Most people are good, trustworthy, and deserving of this power. A few people are not, but taking power away from regular people really won't do so much to help them. In fact, at the end of the day, all it does is force them to become like children, dependent on someone else -- or really, a special class of person, since people absolutely should work together -- for their own security.
"Society" is really nothing more than these people: those that choose "cooperate". It's not the same thing as the "political system" or even the "rule or law". Some people choose "defect", and view the cooperators as prey. Correct me if I misunderstand, but it seems to me, that out of fear of defectors, you want to take power from cooperators and give it to a sanctioned, "regulated" class of people. Presumably because you think you would be safer.
I think you can imagine why this would be ill-considered. If you are concerned about "equality", it's hard for me to imagine a more unequal state. We've basically arrived again at a medieval distinction between peasants, warriors, and priests. Who keeps your warriors in check? The unarmed peasants? The priests with their "regulations"? I'm guessing you are more of a priestly type. I would caution you, your magic is not as strong as you think it is, and it doesn't seem to work at all on the bandits.
It's a bad idea to dox gun owners, but I personally would be very pleased to learn that my neighbors are well armed. I already know a lot of them are. Perhaps you would not be. But if not, consider that you might need better neighbors or maybe a better relationship with them.