Hacker News Reader

Comment by truantbuick | Hacker News Reader

Beyond overt fraud, I feel like there's so much incentive for imprudent analysis.

Several times when I've worked on a project at work that involves data analysis, I get really impatient responses when I don't come to a firm conclusion. We're not even talking about highly charged subjects. I'm not aware if my stakeholders are biased toward one conclusion or another -- I think they're just very upset that my analysis contains uncertainty. They expect me to be able to massage any kind of data to derive clear and obvious facts.

Thankfully, I can refuse this without much consequence, but it's really opened my eyes to the potential pressures to corrupt the integrity of fact-finding in even mundane circumstances.

View on HN

I saw some truly horrific things done at the last place I worked, and the kind of environment fostered was such that I wouldn't even hold myself to any standards if push came to shove. Now I work at a different company on a team of one and could do whatever handwaving I want but due to our culture I'm allowed to maintain a high standard of analysis, which is my default as a scientific and mathematically minded researcher.

> They expect me to be able to massage any kind of data to derive clear and obvious facts.

Not to be rude, but if that's in your job description, then I can understand where they are coming from. As someone in IT, it's always my fault that the computer is not working even if the ten year old device simply broke. That doesn't mean they are trying to get me to do an imprudent analysis.

Sure, there is a difference between operations and trying to find correct answers to questions (i.e. science), but they do have money to make, and if they're not (even slightly) asking you to bias your work, I can understand if they put a bit of pressure on getting the results clear.

Edit: As u/jeremyjh points out, I completely misread this post. If it's about uncertainty due to not enough data, then please completely disregard what I said. (I can't delete the post anymore because there exists a reply to it.)

That isn't how statistics works. You can't remove uncertainty from a given set of data.

Oh that kind of uncertainty. I completely misread that post.

Whether it's hiring a statistician or an IT professional, it reads to me that a lot of people are hired to do work that their superiors simply don't understand (what else would they be hiring you for? :)

There's a lot of people who would think the whole point of data scientists is to look at the numbers and say, with certainty, what they see. But sometimes the numbers don't say anything, and that's what OP was talking about I think - sometimes a clear correlation just doesn't exist, at which point the person who has hired you probably thinks you're not looking hard enough, and might encourage you to 'clean the data' until you get a result with high confidence.

Reply on news.ycombinator.com