Hacker News Reader

Comment by cimmanom | Hacker News Reader

I think this is great advice about the psychology of learning. But also wanted to add an anecdote.

A friend is getting a degree in a field similar to biology. She and everyone else in her program have been processing their data (for at least the past 5 years) with a script written by the person in the department who is most competent with the programming tools at their disposal.

Last week, sanity-checking its output, she discovered a bug in the script that meant that every dataset processed by this program had been deeply flawed.

That means dozens, maybe hundreds of studies published based on meaningless data.

It might be worthwhile to also introduce a cautionary tale or two and encourage your students to take followup courses that go deeper into programming; learn automated testing; etc, so that they can be confident in the future that the output of their studies is correct.

View on HN

Oh my goodness! :-(

Even worse, I heard a very similar story in a science talk only last week. I wonder how many more of these bugs there are out there?

A frightening number, I'm sure. Consider the fact that even highly experienced programmers introduce bugs all the time. And that the vast majority of people writing data processing scripts for scientific use are not experienced programmers.

This is an anecdote but my guess is a lot. I was a software engineer for a decade and recently transitioned to a bioinformatics job. A cursory look at many of the repos that supplement published papers (even in top journals like Nature), will show you that there are often no tests or type documentation of any kind and quite often you'll find bugs when you use packages with your own datasets. So to the OP I would recommend either teaching fundamentals first or concurrently--but don't skip fundamentals!

This is why the push to open science is so important. Making data sets and code accessible for more in depth peer review should be required. Maybe it is a side effect of our antiquated publishing industry though.

To be clear though -- sometimes (often) this is not due to lack of education. The job of researchers is to produce papers and grants, not working software, and postdocs and grad students are junior researchers, not junior developers. It advances their careers to forgo the time to write tests (or, sigh, manually test other cases than their lab's data).

Big labs with large funding where funding sources allow hiring non-research full-time support staff as software engineers, places like the Broad, tend to have slightly better practices. But if you're using a package written by a graduate student.... his job is not to make yours easier. And since he's paid less than the guy who manages your local sandwich shop, maybe it's understandable.

There's a problem in science where we don't value scientists. Telling them "the world values software engineering" and turning them into software engineers is only going to make them go into industry faster.

But surely they (or most of them) value being able to trust their data?

I agree that we shouldn't ask them to be professional software developers, but then we need to provide them with access to professional developers if we want there to be such a thing as science that uses data and computing.

It's ridiculous to ask untrained people to write data processing scripts if you want to be able to rely on the output.

And then people wonder why we have a reproducibility crisis.

Yes, absolutely. It would make sense either for funding bodies to provide ways to fund contract developers or for even smaller institutes to be required to spend a portion of their budget on full-time developers (instead of spending it all on machines and student staff).

For the actual data science, as opposed to writing new software packages, it would be reasonable for students to maintain lab notebooks (except for computer experiments not wet lab ones), with exact versions of everything, exact steps, name, size and checksum of every dataset, and so on. I think things are definitely moving in that direction but this is a skill that is not beaten as much into grad students as recording wet lab work is. But in time it will be.

Though, my experience in wet labs is, both quality of the lab notebooks and quality of the labeling of the stuff in the fridge definitely varies.

Any recommendations for data analysis testing?

...I don't think there's a good book available on that.

Reply on news.ycombinator.com