Hacker News Reader

Comment by leroy_masochist | Hacker News Reader

This is good news. I used to work at Goldman IBD, and the rankings system was really dumb.

The problem was that you had to rate people on a 1-9 scale. As you might imagine, this rating was done via a web survey where you clicked a radio button under a corresponding integer, and under the numbers were verbal cues. The one under 9 was something like "One of the very best people I have ever worked with".

When you get your review, they give you your score for each assessed area, along with the mean score for your position/group (e.g., "associates in IBD").

I distinctly remember that most of the mean scores were between 8 and 9. One was as high as 8.7.

Clearly you had a lot of grade inflation that made the data points of dubious value. Whatever system they chose to replace it is probably better.

DISCLAIMER: I can't get over the paywall and some of the above points might be in the article.

View on HN

The way the system would get described to me by the people I know who used to work at Goldman was that the tech guys would all treat the system seriously and attempt to rate people while the traders would all rate everyone at 9 for everything, unless they hated someone or someone had done disasterously in that area, in which case they would be ranked at 7. The programmers would always be looking for excuses to get a trader to rate them.

It was always told as a parable about how bad at real life game theory the academic, logical, game theoretic minded programmers were vs the traders.

Worked there as a tech guy for 6 years. That is exactly how things worked. I very belatedly realised that the traders were right all along.

Are you saying you didn't lard your ratings with 9's as a techie?

I played the game very, very wrong :/

To poll meaningful opinion in a highly incentivized environment like that, ratings would have to be strictly relative. Not "is person p1 any good at quality q1, q2, q3?", but "quality q1, q2, q3, which one of those is p1's strongest, second strongest, third?". A decision maker could then use that data to extrapolate from qualities he can judge (even if it's as stupid as "attention to personal appearance") to those he can't. People who score badly on a visible quality they are obviously good at must be pretty stellar at the invisible ones, those who score much better than expected in a visible quality, well, there might be a reason for that.

Still, that system would be sabotaged by favor based raters. They would quickly rank qualities by perceived importance (I know I would) and blindly rate the most important category highest/lowest depending on popularity.

Enter sudoku rules: make it a square of persons/qualities and require both rows and columns to be a complete distribution. The challenge of creating a valid matrix should then take care of any remaining game-theoretic influence, leaving only honest opinion and a fair amount of noise (which is a much lesser evil than deliberate manipulation).

It doesn't say how good anyone is at game theory. It might be a proxy for how social they are. The outcome is based on how "selfish" people are, or how much retribution they expect in future iterations. I don't see how this could possibly be used to make a deduction about the understanding of game theory (or practice).

but this is _exactly_ what game theory is about, the parent probably hinted at a failed unterstanding of the theory as shown by your comment ;)

what i mean is, people may think they understand the concept of game theory but are often unable to link that to its practical consequences, as in this case.

I really enjoyed "Jane Austen, Game Theorist" which argues that Jane Austen systematically examined Game Theory years before it emerged.

The author also argues that Jane Austen has analysed in more depth game theoretic 'cluelessness' than has been done since, and interestingly, many of her 'clueless' characters show signs of being unusually interested in numbers, maths, card games, etc.

There was a user around here (michaelochurch?) who made exactly this point loudly and repeatedly. I wonder what happened to him...

That's actually surprising and I think it's a smart way for people to respond. Rather than trying to fight a losing battle and complaining about the problems of such a system, they seemed to realize the better course is to simply give everyone excellent ratings.

It also likely meant people were more productive as a result of not having to deal with getting bombed with a subjective score and other harsh annoyances, so in effect they were simply maximizing shareholder value.

On the other hand, Goldman likely does attract some of the best people, so everyone may just have been being honest.

Note that the "give everyone excellent ratings" strategy is explicitly codified in the standard documentation that is exhaustively and thoroughly studied by every analyst and intern before they ever come near a performance review.

http://www.mergersandinquisitions.com/investment-banking-rev...

Look up at the top of this comment page. Under the headline is a "web" link, which does a search of the article for you. In this case the very first one is to the WSJ article, and it's not paywalled. I'll also note that when I manually try the google search workaround it no longer works for me, but this did.

>I distinctly remember that most of the mean scores were between 8 and 9. One was as high as 8.7.

I was told by several people that the ratings are normalized to adjust for this.

Well yeah, the same way that at some schools you might be in the top 10% of your class with a 3.95 GPA and only in the top 25% with a 3.85.

They can calculate where you stand on the score, but when the question of whether you're in the "top bucket" comes down to whether one or two people out of the 6-7 who are rating you give you either an 8 or a 9 on a 9-scale, that's a pretty fucked up rankings system.

That sounds like a prisoner's dilemma where all of the prisoner's are at least somewhat academic and aware of the choice they are making.

Reply on news.ycombinator.com