Comment by iandanforth | Hacker News Reader

Comment by iandanforth | Hacker News Reader

Parent Full thread

In 2003 I did the work that lead to this paper (http://bit.ly/I1qvuI). Could it be replicated? Probably not. Let me outline a few challenges that don't have to do with a result being true or not.

1. The passage of time

Unlike physics and biology, culture has a huge impact on the field of psychology, and cultures change. Additionally, my work was about addiction to online games. I have no idea if the same type of people exist today that did then. I do know that the game I was studying (Asheron's Call) does not exist in anywhere near the same form as it did then.

2. Copyright

While I was lucky enough to find psychological measures (aka tests) that were in the public domain, most are not. To replicate a result you may have to pay to access the tests and then most likely cannot re-publish them. At the very least this makes replicating work inconvenient and expensive.

3. Data

Because psychology is largely driven by statistics, to replicate a result you should really start by re-analyzing the data. The reality? This data simply isn't available. It is not published alongside the results, and I don't have it anymore. I might be able to find the cleaned data, but that's not important. Error and bias can just as easily be introduced during data cleaning. For example I removed many data points that appeared as outliers, but perhaps these should have been included. No one will ever know.

I hope barriers such as these are addressed in this investigation. It would be disappointing to impune the work of scientists when it is the process that could really use reform.

As a layman, your three points terrify me about the state of science. Well, numbers two and three do. There are still people addicted to video games. WoW and Minecraft probably moreso than Asheron's Call back in 2003.

As to point number two: Are these questionnaires or something that are used to get psychological data? Why would those not be accessible--don't you need to know the questions asked to determine if the results are valid?

As to point three: Data cleaning?! I would understand throwing out invalid results (i.e.-you find out somebody's lying, errors with data collection, etc), but to throw out results because they look like they don't belong feels disingenuous. I've done a lot of work with public company financials and operational systems. I cannot, on my worst day, imagine telling the SEC, "Yeah, well that return was really abnormal and was an outlier, so we decided not to record it." Why are these data points thrown out?

To me, again, a layman, it appears that if this is standard practice, these experiments are starting with a conclusion and then just going through the motions to get that conclusion published. Why are data cleansing and secrecy normal practices? What about the scientific method don't I understand?

You can clean the data according to certain metrics: e.g., "We removed all results that were greater than 6 standard deviations from the result".

There are other metrics of dropping data that is "out there".

As someone finishing my Master's, I will be releasing all my work, including test inputs and reproducibility results, as a Mercurial repository. Some of it depends on an exterior compiler, but the interesting ideas don't. I feel it is absolutely critical and honest to release source code and all data.

See the CRAPL: http://matt.might.net/articles/crapl/.

The line between an invalid result and one that looks like it doesn't belong is fuzzy. I believe I was removing the former, but if an independent replicator doesn't have access to the raw data they would never discover that I had been fooling myself.

As to secrecy, there is nothing being hidden, instead it's that nothing in the system requires you to publish all your data, and so because you're pressed for time you don't. Never ascribe to malice what laziness can explain :)

I'm not ascribing this to malice. It strikes me as a paragon of negligence if that's the way that the science of psychology is being done. In fact, it strikes me as an affront to science.

Take this in contrast to the neutrinos that were going faster than light at CERN. Instead of going, "Okay, well, those can't happen, so those are outliers," they reported their results, published their methods and data, and said to the world, "Help us validate this."

It seems to me like the modern psychology scientific method is the antithesis of this--"These are the results of my massaged data, and no you can't see how I got them, just trust me."

I'm not attributing this to malice, I'm attributing this to negligence and laziness. Putting your work in front of people for critique is hard. Really hard. But, it's part of science. If psychology as a field has convinced themselves that they're above that reproach, then I think that's a huge condemnation of the field.

When I first read the article, I thought it was a bit sensationalist to call psychology "under attack," but now I'm not so sure. I've assumed that reproducability is the standard litmus test of all science, but I guess not, and that leaves me with a bitter taste in my mouth toward the field of psychology.

Outlier detection is an important part of basic statistics, and has been for a long time. It isn't about just deleting a few data points that "look" strange. Some statistical tests are robust to outliers and some are not... it's always important to use the appropriate tests in the appropriate way and be open about them.

"It would be disappointing to impune the work of scientists"

I hate to point this out, but I've never even heard of any attempt to "impune the work" of any scientist that is anywhere near as damning as what you wrote above. You've described your work as temporary, local, and proprietary, and you've discarded the actual data. Being unable to replicate it is the least of the issues.

Note: I am not a psychologist. What do you mean by "some tests are copyrighted"? Are you talking about a statistical method being copyrighted, an implementation thereof, or something else entirely?

He's refering to this (example):

https://en.wikipedia.org/wiki/Beck_Depression_Inventory#Limi...

The BDI is copyrighted, a fee must be paid for each copy used, and photocopying it is a violation of copyright.

Correct. A portion of the original study involved a personality measure like big five (http://en.wikipedia.org/wiki/Big_Five_personality_traits). The tests for this are almost all under copyright.

Probably something like the Rorschach tests being copyrighted until recently. The actual test questions and the scoring rubric that, applied to a subject, yield a number or set of numbers.

Reply on news.ycombinator.com