Wait, so just because there's fuzziness to the data you're going to instead defer to what? What's the proposed alternative/status quo that you're giving a data-less free pass?
Data grants authority to a decision that gut feel driven one doesn't. It is hard to argue against evidence as it should be, but that assume a certain level of quality in the evidence.
Second, if practice doesn't match the expected outcome, the first thing you will look at is what the team is doing wrong, not review the decision as not working.
That said, parent is far from unique in his skepticism, so I think the problem is more often reversed in the industry. Having some data, even flawed can help your company decide to try something new.
What is "bad data", and how is it worse than no data?
Have a really dumb example:
All of the developers I work with are short, therefore all developers must be short, and we could save time by filtering job applicants by height.
(Note: I work with only one developer. He's great!)
The alternative is to rely on rich experience and good taste. If you want to make it a bit more rigorous, you can approach this in terms of qualitative research—which makes sense for academic research, but isn't necessarily the best way to learn for yourself or to design tools.
Expert experience is far more effective at capturing complex, multi-dimensional phenomena than anything we can reduce to a small number of easily-gathered quantitative metrics.
At what point, and how do you determine that?
>The alternative is to rely on rich experience and good taste.
Everyone thinks they have good taste.
That is not true
In a different domain I play music in bands
I have played in a band where the band leader was so so at playing their instrument, but had oodles of taste, so much more than me
Not all of us, not many of us, are arrogant pric,s who think we know better or are better than we are
By looking at the research? By understanding what kind of studies and experiments are actually viable? By evaluating what we can measure effectively, what we can't, and how good (or bad!) our proxy metrics can be?
As a field, we really need to understand the inherent limits of quantitative methods.
Very few of these kinds of studies follow scientific method well enough to be vaguely useful let alone generally applicable. How many have you ever read of being successfully reproduced?
Of course personal experience is very valuable, precisely because it's a (limited form of) research.
I had an incredible moment in a job interview: for context, I’ve been programming for about 30 years and 10 of those years have involved a lot of JavaScript. I was given a timed debugging problem to test my skills: “Here is some JS code with failing tests. Go fix the bugs.” Before I even ran the tests I scrolled through the code to get a sense of it. My instincts tweaked on one section. “This code smells. I don’t trust it.” I said. On second glance - “Oh yeah this is totally buggy”. Sure enough, I was right. The interviewer was blown away.
There’s no way I could do that consciously. Science is great for a lot of things, but when we don’t have science, we could do a lot worse than trusting our intuition.
you can attempt to dismiss that as being a small sample value, but that's the reality.
The reason people want to be so data driven is because it takes the decision making out of their hands, which takes the responsibility for poor decisions out of their hands.
But when it's all said and done, some people are better than others, not all "small sample values" are equal.
Better at what though? Success doesn’t tell you why someone was successful.
The point of looking at larger sets of data is to learn something tangible beyond anecdote.
it's another version of cargo cult programming. The data suggests if a plane is present, food will come. We want food to come, lets build a plane, with no real understanding of where the food actually comes from.
> Better at what though?
whatever the fuck it is you're trying to evaluate. Are you evaluating musical aptitude? then the answer to your question is that some people are better at music. are you evaluating the ability to tear down an engine? then the answer to your question is that some people are better at tearing down an engine.
Linus Torvalds opinion on kernel development is not equivalent to that 16 year old web developer. Dismissing opinions as "small sample value" _completely_ misses the forest for the trees.
Likewise anecdotal success of an individual is also just correlation but with only a single data point.
I’d also agree that expert opinions are great but they’re also not infallible or necessarily generalisable. Nor is it necessarily easy to tell who is an expert and who is a charlatan. Which is why we should use them as the basis for investigating the larger picture.
Similarly both expert opinion and singular anecdotes can lead to cargo culting.
hmmmmmmmmmmm.
> Likewise anecdotal success of an individual is also just correlation but with only a single data point.
oh yeah, someone should tell Linus Torvalds that his expertise is just correlation. Same with Elon Musk, Bill Gates, et al.
"oh but I think expert opinions are great!"
doesn't actually change that stance, whether you think they're great or not is irrelevant to the claim.
Your problem is putting everything into data terms.
stop doing that and suddenly your problems with my responses go away because I stop calling it out as ridiculous.
I'm constantly amazed at how people are willing to throw away data if it's not perfect. All you need is a bit of signal, and it's, literally, better than nothing.