Dissing Track Records

I naively asked a lawyer what his win rate was when I was seven. He explained, fairly patiently, that that was an exceptionally poor metric to judge lawyers by. "Few disputes end up in court. Most are settled out of court, with trials meaning a negotiation broke down or one thought one had a slight edge. If they have a major edge, you beg for a settlement. If you do, typically their lawyer will.

If you never lose a case it's because you are being insufficiently aggressive on behalf of clients with strong-but-not-unbeatable claims and are disposing of them pre-trial to the detriment of the client. A lawyer who would do that is a poor lawyer regardless of the stats."

A very interesting point. Note the pattern in responses:

1. Lots of people criticizing the proposed metrics.

2. Hardly anyone proposing ways to improve them (e.g. augmenting them with information about the difficulty of the cases that were taken).

3. Hardly anyone pointing out that even worse criticisms can be made of the informal criteria customers currently use (the shape of someone's nose, where he went to school, who he knows, the amount of money that was spent on his office).

Most of the criticisms made in #1 are true as far as they go, so looking at any one of them in isolation wouldn't tell you anything much. It's the overall pattern that's informative.

While I'd be interested in the track record data, it is hard to properly metric these professions. How do you control for a priori risk of a patient (IE- one total knee case is a 70 year old smoker so ways 300 lbs and the other is a 28 year old athlete who fell skiing.) The ACA has resulted in the collection of this type of data and we'll probably see it released someday. We'll almost certainly see complication rates reduced and risky v patients refused certain types of care, but I do feel bad for the professions getting metriced like this. After all, I'd be pretty pissed if my value as a software developer was judge entirely by my bugs/KSLOC number.