The restrictions are along the "meta" parts (voting to indicate the quality of an answer), I think it is entirely sensible to restrict these actions to users who have built up a currency of trust within the SO community.
That being said, I think you are onto to something around the user experience. Perhaps an option shouldn't be shown to someone if they cannot perform it.