Since money is used as a medium of exchange, it's essentially a placeholder for your labor (when you receive it from your employer/customers as payment) or value provided to others. It is arguable whether society has as much right over an individual's labor as he/she does, I think they don't (while society may have a contribution, it might be significantly below 50%).
One last and minor thing we'll have to disagree on: I don't think government equals society. An individual can be a part of society without being part of government, and I also believe government sometimes represents its own interests at the expense of society.