Hacker News Reader

Comment by Godel_unicode | Hacker News Reader

People get really mad when you call this p-hacking, btw. I used to try and make this point to ”analysts” that I worked with, that science involves coming up with a hypothesis and then looking at the data. The response I got often enough that I’m sure it’s being taught somewhere was that you “need to let the data tell it’s own story”.

Turns out lots of people have decided what the conclusion is and are taking that conclusion and the data, and then blindly munging until they get a path which connects them.

View on HN

I think a lot of times unless you are operating at hyperscale and bps matter, most business data analysis is more art than science. Ex, if you have < 200 sales in a month and you want to report this, it's totally normal to want to pull it into excel and fiddle with the numbers, make ad-hoc corrections, divide those 200 sales up in very subjective ways, etc. The smaller the company the more the reporting layer is only a weak approximation of reality, there isn't enough there to create a narrative around. "Hypothesis testing" such data is honestly a waste of time.

Of course you want to be intellectually honest about it, and I agree that cherry-picking the "right" data can be a big problem in the wrong kind of organization. I remember one time a Jr analyst I managed was asked by the CEO to create a certain chart. I was not looped in. The CEO then used that chart to convince himself and many others on the executive team to make a huge product change that ended up being a complete disaster.

As my professor said: All good statistics is done before you have looked at the data.

But where do you get your hypothesis from?

your hypothetical model of reality.

But that's a lot of work so businesses don't want to pay for it.

Whatever orifice you desire.

True, for clean data. You can’t clean data without looking at it (a lot), though.

If you have data that's so 'dirty' that you can't decide on the filtering rules in advance (or based on only historic data), then what you have is garbage, not data. Therefore, we could call the art of shaping this into meaningful stories garbage science.

Tell me in a comment you have never worked with business data in your life.

Business data is full of minor inconsistencies which are not obvious until you sit in front of it. Products are sold by different units. Reporting ranges and aggregates are slightly different. Subsidiaries use categories which are close but not exactly identical.

There is generally plenty of massaging to do before you can get the information you need.

> I’m sure it’s being taught somewhere was that you “need to let the data tell it’s own story”.

AWS is pushing this message on every NFL broadcast with their Next Gen Stats ads. "The data tells us all"

Reply on news.ycombinator.com