my user's privileges contain my ssh keys, my passwords database, my personal files, ...
You can firewall by process UIDs/GIDs too, and I use that to allow the user access only to the internet and not to localhost or home network, or only to localhost, etc.