Decisions Without Data are Guesses
20bits.com
20bits.com
...however...
He misses the fact that it can be very hard to gather a sufficient sample of data for a complex set of variables. If you are designing a web site there are literally hundreds of choices to make. You can't reasonably expect to test all of your decisions before making them, or else you will be paralyzed by delay.
This leaves you with a second option of mocking up a couple of choices and see which ones are liked best, leaving you potentially with the best of a bad bunch.
Or, my preference, option three: Pick a couple of design champions who will tear any design to shreds and make you really justify your choices in terms of work flow and user interaction.
Generally you can assume that you aren't stupid, and you can figure out the right choices to make, however you won't always notice you need to do so unless you are being forced to justify your decisions by a grumpy critic.
Anyhow, yes, you can make arbitrarily complex measurements.
The point was more human than technical: some data is better than none, even if it's not fine-grained. It results in fewer pointless arguments and better decisions overall.
Not necessarily. If the features of your small data sample are unrepresentative or poorly analyzed, you run the risk of distorting your priors and making a poorer decision than that expected by chance alone.
We can chew the fat on this one for a very, very long time :-)
Still, I tried to keep my point constrained to those situations where you don't run into decisions not easily resolved in the manner I described.
Sure, if you're wading into a new market, old models or assumptions might be incorrect. Or if you need to make a decision now and don't have time to gather data your intuition and savvy are going to be instrumental.
But for most day-to-day, pedestrian decisions around software, web development, and business there's plenty of data that will tell you whether you should zig or zag.
Being able to make good decisions based on conflicting or incomplete information is critically important.
It was intended to be a small point. The psychology and effectiveness of decision-making strategies could fill up books (and they have).
The scenario I had in mind is one where you have some limited set of options, a consensus is necessary, and the overhead of measuring the effectiveness of the alternatives isn't too high.
In those situations you're going to argue until someone gives in, someone higher up tells you to knock it off and makes a decision for you, or you decide to see which alternative actually does best.
I understand that, actually, and left it out of the blog post only for brevity's sake.
When I make decisions about my websites and other projects I have in mind a picture of my audience and what they're doing. This sort of experience is the basis for most intuition. Some people are really, really good at it, too.
The right kind of experiences can be data, too, but there are two dangers: (1) not being wiling to change your mind and (2) acting off too little experience.
I just expect that any "expert" whose intuition I'm supposed to trust will be able to justify their opinion when push comes to shove.
Don't expect them to justify it solely on your terms, however. If you're breaking into a new market or operating under asymmetric information pressure with significant uncertainty -- or even just regular old data starvation -- appealing to similar but difficult-to-quantify means of supporting one's intuition may be crucial.
It would be better for all concerned if the person making a claim could always articulate their reasoning in toto, but under the pressures of time, you may occasionally need to rely upon a person's track record, and trust them. Because their track record under similar conditions is a prior -- that, too, is data.
Extraordinary claims require extraordinary evidence -- but people with an extraordinary track record can sometimes be trusted to make extraordinary judgments with ordinary data.