Software faults raise questions about the validity of brain studies | Hacker News Reader

Software faults raise questions about the validity of brain studies

arstechnica.com

110 pointsingve21 comments

I spent a few years in the late 90's and early 00's working on fMRI and I've been stunned by many of the claims that appear to arise from it's use. What's being measured relates to blood flow changes that correlate with an input block paradigm, typically from a visual stimulus. This isn't brain activity in the electrical sense but rather which areas of the brain appear to regulate their blood flow in relation to the stimulus. The assumption being that these areas require more 'fuel' to support their increased neural activity in response to the stimulus. Or as I like to think of it, attempting to estimate your home electricity usage by monitoring your water bill. Doing so with MRI, a technology I've spent most of my adult life analysing the data from and still can best describe what it does as 'magical', adding another not insignificant level of complexity.

I'm know there is good science going on in this field, by people that understand the limitations of the techniques and the technologies. However, I worked with psychiatrists - a clinical discipline starved of quantitative measurements until fMRI - that would happily ignore statistically significant activation in the air around the subjects head whilst laying claim as to the importance of those in the frontal cortex. Seeing the visual cortex 'light up' in response to flashing chequerboards is one thing, isolating those areas of the brain responsible for 'forgiveness' is something quite different.

Of course the software has bugs, I personally know people who wrote the package in question and they are extremely smart and also very human. I doubt I've ever published a paper using software that wasn't bug ridden. That's why open source is such an important part of the process, laying bare every last detail of what was done and not just what felt worth mentioning in the paper. The medical imaging research community is particularly good at this with most of the industry software making source code available. The problem with fMRI is not the software.

I think you're missing a key thing when talking about the activation estimation.

The measure is not blood flow, but the change in oxygenation of the blood, is it not? This does have issues but I think it's different from the issues you raised.

You are right when you say the signal is BOLD and not flow directly. Flow regulation is a key component though. However, I don't think this negates my point. Oxygenation is being equated with brain activity. The approach is sensitive to BOLD signal changes that correlate with the input. I'm not a neuroscientist but I know many that would argue the brain is more complex than this and response are more complicated (what about areas of the brain that are on through out the experiment?)

I guess that if we're measuring bloodflow instead of neuron activity, then perhaps our knowledge of where in the brain something is happening might be off, but the results are not per se invalid. Things might only get tricky if more than one vessel feeds the same neuron.

Each blood vessel feeds hundreds of thousands of neurons.

That was not the point. Say you have a vessel A, and a vessel B, both feeding hundreds of thousands of neurons. Assume A and B both feed neuron X (one of those many neurons). Then A and/or B might fire up in a scan when X is active. The "and/or" is the tricky part.

Are there any ideas for new technologies that give a more meaningful signal than BOLD with reasonable spacial accuracy?

I've recently made the argument that science publications should include code and data. See comment history e.g. https://news.ycombinator.com/item?id=11606278

Glad to see that concrete good is coming from such efforts:

> The researchers took advantage of a recent trend toward making data open for anyone to use or analyze. They were able to download hundreds of fMRI scans used in other studies to perform their analysis.

The more data and code that's made openly available, the stronger science will be. I hope that we can work toward a future where most if not all code and data are expected socially and by funding policies to be included in the publication process.

Agree 100 %. The data of publicly funded studies should be available online so that people can run their own analyses on it. There could be a non-disclosure period of course to give the scientists that have done the original work an advantage when publishing their results.

I'm confident that we will see cloud-based systems for scientific data analysis in the near future though, especially for data which is uniform and for which good standards exists. For some areas like genomics we already see this, and I'm sure others will follow suit.

I also predict that being able to try out analysis algorithms on peta- or exabyte datasets will be a huge game changer for many areas of science, and will invalidate many current findings that are based on smaller datasets, while hopefully producing many new ones as well.

I think scientists already support that for the most part, but there are lots of barriers, e.g. cost of maintaining the data, privacy concerns, intellectual property, competition, etc.

Despite the title, software was not at fault here. Rather, the paper found a higher than expected false positive rate due to a choice in statistical modeling. The conclusion was that nonparametric tests can avoid the issue.

So, none of the results are truly invalid in the way of a hypothetical software bug, since we know what modeling assumptions were used in the previous studies. Personally, I don't see a major problem, since most fMRI papers are exploratory, and we should be reproducing the major findings anyway. We should certainly start using nonparametric tests from here on out though!

A 15 year old bug in a basic package used by fMRI software was, according to the researchers, causing numerous false positives in previously published studies. Which means some of the accepted fMRI based brain studies results are dubious.

They were dubious anyway

As I read it, the bug was a minor issue. The bigger issue is the general false positive rate, which isn't a bug exactly, but more like a statistical modeling question - with millions of vocals, what's the appropriate way to characterize an activation as spurious?

No, that was a side note about a bug that caused a ~10% increase in false positives. The real primary issue is that the clustering methods assume Gaussian autocorrelation distributions, which don't exist in reality. They estimated a 70% false-positive rate because of this

http://m.pnas.org/content/early/2016/06/27/1602413113

Related classic: Neural correlates of interspecies perspective taking in the post-mortem atlantic salmon: an argument for proper multiple comparisons correction http://pages.vassar.edu/abigailbaird/files/2014/06/bennett_s...

http://blogs.scientificamerican.com/scicurious-brain/ignobel...

http://scan.oxfordjournals.org/content/4/4/417.full

There is a real problem with incentives in science strongly encouraging bad science. It is a publish or perish world, so every researcher needs to be finding some (apparantly) statistically significant result on a regular basis. Quantity is rewarded more than quality. Being super careful with the stats makes it much harder to get publishable results but doesn't significantly increase the rewards (i.e. academic credit). So the culture tends to sloppy stats making garbage or weak results look strong. And there is no academic credit for being open with data, and more of a risk that someone will discredit your results if you are open, so that is rare as well.

There really ought to be some sort of PE-equivalent certification for mission critical software, where they're required to sign off on the code before it can be legally distributed. It seems like the majority of today's catastrophic failures are due to faulty software and currently there's no accountability when it happens. If engineers were routinely crashing aircraft and breaking MRI's they would be going to prison because of the accountability procedures that are in place.

I am not sure if this would really be much of a shock to researchers who have spent time working with fMRI data.

I'm a computer science grad student, but spend at least half of my time taking courses and reading literature from the fields of psychology and neuroscience. Brain imaging studies have become very popular within many subfields of psychology, yet often the published analyses I encounter make inferences which are not necessarily well supported by the data.

It is encouraging that more researchers are making their raw data available. Currently many academics treat fMRI studies as some kind of infallible truth, when in reality it would be wise to give more consideration to the many sources of error that contribute to conclusions reached from imaging data. Hopefully the availability of raw data as a supplement to publications will help us gain more complete understandings.

So how many neuroscience papers can now be used as a scratch pad?

I am surprised it took this long to discover this..

Reply on news.ycombinator.com