The flagging system is almost there from that point of view, but it's still pretty hit and miss... I think if the flagging system was tweaked to give more weight to the flaggings of people on the leader board, the system would work extremely well.
Of course, this may be by choice - after all pg may not want to create an oligarchic moderator class on news.yc... up to him.
I'd see an oligarchy as more of a site like Reddit or News.YC, but rather than every person having a vote, a set group of people are able to. Like if only the leaderboard could perform certain tasks.
The problem to circumvent then would be if the leaderboard became overtly political, but I think that a) that isn't going to happen on a site like this, and b) it should be easy to make the system balanced if we have Dictator Paul Graham around to make sure that the people in such a system don't turn into absolute jackasses.
The problem lies when normal people stop thinking that they can submit and get content to the top, the submissions tend to become rote, and you lose community interest.
(As an aside some of the power users on Digg started taking bribes and were banned en-masse IIRC, but I don't see that happening here.)
It won't happen, but I think it would be an interesting experiment.
If I remember correctly, you're expected to lurk or use the newbie boards (http://www.dumbrella.com/bb/viewforum.php?f=2&sid=cb8a89...) for at least 6 months before posting on the party board.
Even then, if other members don't like what you post, you get harassed pretty quickly and often an admin will delete your thread, or lock it with an (often insulting) explanation of why it was locked. If someone continues to submit posts that members just don't like, that person will get banned pretty quickly. There's a certain culture there, and everyone, admins especially, will berate people who don't fit in.
That would work here, but it certainly works for them.
Also, on HN there's a distinct "we" feeling - my (limited) experience is that admin-dominated forums are more focused on class (ie. looking up, even sucking up, to mods).
I thought that's already the case, though. Can't you not vote stories up until you have a certain karma level?