The Worst-run Big City In The US
lesswrong.com
lesswrong.com
Look at what has been happening in the schools with standardize testing. Proponents think they are measuring outcomes (effectiveness in preparing children to be skilled & productive members of society) with standardized tests. They are actually measuring outputs (test scores) as a proxy for the outcomes.
The result is the system being measured will adapt. Schools "teach to the test" which improves the outputs but we don't know the effect on the desired outcomes. It may result in children being less prepared.
This is a well known effect in companies that measure individual performance in terms of outputs (widgets manufactured, # of upsells, etc). The employees will deliver the required metrics but it often comes at a cost to quality and customer satisfaction which is a negative outcome to the company.
Which is the main lesson in quantifying outcomes: Measuring the wrong thing can be worse than not measuring at all.
I'm calling bullshit on that one.
Yes, measuring the wrong thing can incentivize poor outcomes, but movement in the wrong direction is visible and can be corrected, whereas lethargic inaction just lazily drifts into a morass.
You have to measure outcome. Period. We can have great discussions around what kinds of measurements to have and how to apply them, but money should not go out the door without some indication that something is happening as a result of it -- even the wrong thing. In fact, big visible indicators of the wrong thing happening as a result of measurements is a terrific result. If we're arguing over measurement, the game is already lost.
Another way of stating this is that let's say I pick a measurement to implement. It's a terrible measurement, and it causes much pain and suffering. You come to me and say "This is a terrible measurement and causes much pain and suffering!" To which I respond "How do you know this?"
So you show me a study -- another form of measurement! Then we have a conversation about including parts of your study in my measurement, scrapping my measurement entirely, and so forth.
We cannot have meaningful conversations about management without measuring things. Once we start measuring things, the world is wide open about which things to measure. But if you convince me that the act of measuring itself is broken, we have no further basis for any conversation around performance at all. Management has no function. There is no quantifiable return for any possible amount of money I might invest. Very bad.
We don't need to wait around to pick the perfect measurement or do nothing at all. We need to pick measurements -- any measurements -- and rapidly iterate towards finding what works best for us.
If the testing system is broken, and I have no doubt that it is, it's broken based on bureaucrats creating a large, brittle, hard-to-change architecture of testing, instead of just trying to continuously evolve what works best. It has failed because we put the same idiots in charge of school testing that were creating the mess the tests were supposed to fix in the first place.
No, he shows you an anecdote or two. Which isn't worth very much, but it's more than what you had before. It's not like the original measurement had any studies validating it either. But it looks scientific: it has numbers!
I've heard some amazing stories of mismanagement and excessively liberal policies (e.g. Homeless people being paid stipends by the city) but there are plenty of well managed cities I would move out of in favor of San Francisco. (My favorite post in the linked thread was a comparison of SF to Saudi Arabia.)
http://www.sfweekly.com/2009-12-16/news/the-worst-run-big-ci...
I guess that's better than £1 billion on a not-very-good idea that fails for stupid reasons:
http://en.wikipedia.org/wiki/Edinburgh_Trams
£1 billion (and counting) to transport people along a pretty short route in a city that has excellent bus and taxi services is a wonder of our time - there are only half a million or so people in Edinburgh so that's £2000 per person for a service most of us will never use....