Actually, they should be judged by what they can and do contribute to the team, whatever that is. Skills is a subset of that.
Ok, we should judge people by what they can and do contribute to the team. The billion-dollar question is, HOW? Humans are extremely skilled optimizers - give them an objective metric and they'll game it for maximum benefit; make it subjective and it can be better or worse - but in either case, you'll no longer have any agreement on "what they contribute to the team".
* Documentation - Can they write?
* Potential - Can they provide original solutions to problems? Do they need hand holding?
* Initiative - Do they need a signal from a starting gun or are they self-entertained with word products for the team?
These are all identifiable maxims.
What I _am_ saying is that it's an art not a science - very hard to teach, impossible to scale.
Take your example with "Documentation" - if you measure anything objective (words written, number of functions/features that have associated documentation) and tie it with the performance evaluation, those metrics will soon become meaningless.
In the example of documentation a better metric might be a composite value of
(i) how many wiki articles a developer has written, weighed 0.33 and
(ii) how useful on a numerical scale his/her peers rate the documentation produced, weighed 0.67.
This is just an example but it's a composite of perceptual and objective, with the perceptual being a lot more important (and to prevent gaming the system by writting lots of useless articles).
When I worked at Microsoft one of the teams I was adjacent to had a metric on number of new apps in the Windows Phone app store. So the teams went out and got a bunch of college students to build shitty apps in bootcamp style working groups. Suddenly the number of apps isn't good enough, so they added review metrics. Now those teams add a "let's all rate each other's apps" portion to the bootcamp taking you even further away from the results you're trying to obtain.
I hate doing performance reviews, and I especially didn't like when at a former employer I was asked to stack rank programmers. I feel like it's a task that's so difficult to get right, I can't even think of a wrong way to do it that's useful.
No, it certainly doesn't mean that "your performance was [7.23541 8.1241 34.412515 .52632 995.154]". It means "you did good/ you did very good/ you need improvement", with some details like "<these things> you really handled well and I appreciate it; maybe you can work a bit on <this area> though". I definitely agree one shouldn't stack-rank people - especially in a good team (it's entirely possible, desirable even, that everybody did a great job)
The second is that different types of people prefer different styles of working and sometimes forget there is another side (multiple "other sides", really). A team needs to be a mix of capabilities and personalities to be successful - a team of 100 identical individuals would likely fail, even if all them are Peter Norvig). So, you need people that are "senior" in the sense that "knows the business well enough to provide different perspectives" (and for those "shows initiative" is critical) but you also need specialists where "senior" means "knows technology X really really well". Say, a DBA - can keep your database up, can write efficient queries to retrieve information that you want; but doesn't know sh*t about what information would be interesting to retrieve. Or whether it's a good idea to keep some information X in the database, considering the various business, legal, social, cost perspectives.
> Do my managers know what they're doing?
Here's the thing: I believe nobody _really_ does. Sure, some know more than others, but in absolute terms, we're all basically guessing. That's why people insist on engineers that "show initiative" - not because they're always right, but in an environment where we don't _really_ know what we're doing, people yelling different perspectives are valuable.
However, as mentioned above - not all senior engineers want or are inclined to show this kind of initiative; and it's unfair to penalize those kinds of engineers, because we _need_ the different types of personalities.
[1] FWIW I believe the reality is , always, that you need to (A) work on a successful project; and (B) be generally liked by your colleagues and maybe managers. For very senior titles, also (C) have a large network of connections within the company - i.e. work on many things, or on one thing but a thing that is used by many teams/ really popular)
Objective metrics are good if you think you can measure accurately for all individuals at the macro level. You may not need that (I mean, really _need_ it for your business to function, unless you're in the metric-tracking business).
If your teams have distinct and articulate enough outputs, you can retribute each team according to its specific outputs. At the macro level, you don't need to have more details (but you still may ask).
Let the team manage its own distribution of the retribution - and so on (so you do need trained managers to do it appropriately at each level).