Comment by tedmiston | Hacker News Reader

Comment by tedmiston | Hacker News Reader

For an engineering community manager, that was an awfully rant-y way to make the point: "It's frustrating that researchers don't publish their code."

Not to say I disagree with the frustration... but it's also not something new. It's been this way for decades. I'd much rather hear about who is doing work in this space and what they're working on. Here are the ones I know of:

1. The Center for Open Science (https://cos.io) is one such org trying to fix this with the Open Science Framework [1].

2. GitHub also recognizes the need for citable code and gives special discounts for research groups, in fact, Mozilla is one they work with [2].

Two smaller related startups are:

3. Datazar (https://datazar.com) - A way to freely distribute scientific data.

4. Liquid (https://getliquid.io) - A scientific data management platform. Somewhat like "Excel for scientific data as a Service".

Also, a related HN thread from some years ago: "We need a GitHub of Science" https://news.ycombinator.com/item?id=2425823

---

1: https://osf.io

2: https://github.com/blog/1840-improving-github-for-science

I like his rant. But at the same time, as an ex-CS grad student now doing biology, I wish my field was 1/10th as rigorous, tidy and beautiful as computer science.

Code is a joke, most data is processed using "pipelines", which in reality means some irreproducible mess. People don't generally do research trying to understand how cells or tissues work, they generally write papers about "stories" they found. Only a small minority are trying to do some serious modeling using serious math.

Now imagine being a patient reading those studies and knowing that.

Struck a nerve

Biology is related to medicine, but not researched quite the same way.

Medicine certainly has it's own issues, but they're working on it:

https://en.wikipedia.org/wiki/John_Ioannidis

http://www.theatlantic.com/magazine/archive/2010/11/lies-dam...

http://www.economist.com/news/science-and-technology/2159894...

I mean really, the Churchill quote about democracy applies here as well: the current way of doing research is the worst possible way of doing it, except for all the other ways we've tried before.

I was thinking more along the lines of the Myriad Database

http://www.theatlantic.com/science/archive/2015/10/filling-i...

VUSes that actually could be clinically significant are hoarded, and conversely, some companies claim to know that certain VUSes are not based on their own proprietary data that no one else sees/double checks. meanwhile the actual data comes from patients, who are left in the dark

That's just cruel. Comparatively, at least CS gets math formulations

Ah, I get your point now

> Code is a joke, most data is processed using "pipelines", which in reality means some irreproducible mess.

You're not wrong, and it's not limited to bioinformatics; Reinhart-Rogoff's findings were reversed when an additional 5 rows were included in a spreadsheet they used to calculate their correlation between GDP growth and debt ratios. And of course, they insist that despite the actual outcome being twice as strong and in the opposite direction, they still support their original position.

I wonder if one can get a CS PhD by producing enough retractions. Of course, it won't win you many friends in the academy, and would probably lead to less source code made available. But given the Perl code I've seen published who's termination condition is a divide-by-zero exception, one can argue that peer review in the information age has to include code review.

Didn't know about the Reinhart-Rogoff controversy [0], interesting! They state that they have been careful not to claim that high debt causes slow growth, but rather that it has an “association” with slow growth.

[0] http://www.huffingtonpost.com/2013/04/16/reinhart-rogoff-aus...

I read abt this quite a bit, the refutation is attacking a small portion of the data, because that small portion is trendy and hot in politics.

Judging academically, the original paper and the refuting paper is a healthy debate, but the dynamic of the society and politics ab-use them to attack a whole school of thought at large (the austrian school: less bailout, less intervention by government, less control over everything) in favor of keynesian school (more bailout, more government spending, more public debt, especially in recession and crisis).

Anyway, it remains a controversy, because theoretically one can do what one wants, but once it involves policy and real life matters, it is hard to argue for what method is right and what is wrong, in the presence of so many (ready to b angry) interest groups.

Excuse me? That single attacked paper was the intellectual blanket for an unprecedented victory march of the Austrian school after the financial crisis and the recession.

I agree that from a purely academic point of view this is nothing big to worry about, but this paper played a completely outsized role. And the authors stood by and let things run their course, without any attempt to reign in or moderate the debate.

https://en.wikipedia.org/wiki/Growth_in_a_Time_of_Debt

Fair enough, but i add that academic life is sad, one has to pursue one's endeavor at one's own cost. However, politicians and the public want too much from us researchers. So sometimes, we do believe that our sweating formulas have life impact, or to fancy, save the world.

Wow. It's scary that 5 rows have such a big impact. What about the next 5? Is this science or sorcery?

It's worse than just those 5 rows.

http://www.peri.umass.edu/fileadmin/pdf/working_papers/worki...

>>People don't generally do research trying to understand how cells or tissues work, they generally write papers about "stories" they found. Only a small minority are trying to do some serious modeling using serious math.

Speaking as a layman, that seems like a very strong claim for what one hopes is a hard science supposedly applying the very best practices of the scientific method (i.e. falsifiable theories vs. anecdotal stories).

Is this meant to be hyperbole to get your point across, or something that is generally known throughout the bio[med|tech] industry? As a sister comment pointed out, the latter scenario would be quite alarming.

EDIT: I'm aware of growing sentiment within the scientific community to reconsider using p-values, of which John P.A. Ioannidis and his body of work[1] helped to raise awareness of. Was this the "story"-like theme that you're referring to in cell and tissue papers?

[1] http://journals.plos.org/plosmedicine/article?id=10.1371/jou...

His claim is hyperbolic, but only from the perspective of the way that scientific research has been conducted for the last 10-15 years. Prior to this, many researchers were forced to tell "a story" because the sheer volume of data and complexity in fields like cellular biology was/is so massive. This does not mean that their findings were in any way intentionally deceptive, it's simply that it was impossible to cover all the ground necessary to tell the whole story.

However, as data analysis techniques have become more powerful and data more easy to produce thanks to advances in the scientific methodologies we use to access information about the matter we are studying, the datasets are getting bigger but also easier to manage.

Biology is driven by genetics, and it's what grounds the field. I think that, perhaps, your experience is more in bioinformatics, for which I share some of your skepticism. Some good work is being done there, but usually that has a strong genetic component.

We're at an interesting point where many of the useful mutant screens have been done to saturation, at least for the most common species. Expanding our genomic resources to screen more species is certainly helping. The way forward is just the same as it ever was; good mutant phenotypes with strong genetics.

I don't think 'serious math' is at all useful for most biological investigation. Modelling is only a means for generating hypotheses, and in my experience it's a terribly weak tool. A poor substitution for proper genetics.

I think the reason academics don't publish their code is that they don't want to get stuck into an endless discussion on the code along with people asking for "support" on it.

What I have found is that if you need the code for a specific data analysis then send an email to the authors of the paper. More than ever you will be surprised. They do share code with fellow researchers.

So - if you do really want code, write the email. Be forewarned though - the quality is often bad from a s/w engineering perspective and suits a very specific purpose. It will also come in a language of the authors choice (Tcl/Tk for ns2 scripts, matlab scripts, octave, etc.)

I've published code for most of my papers, and rarely, if ever, received support requests. When I have, it's usually been from someone else trying to do work with it, who asks good, intelligent questions (i.e., the requests you want to receive).

Maybe that just means I don't write very interesting papers.

I think you're doing pretty well if you've gotten more than zero supports requests, and they are all relevant to the topic too.

Should also include Gitlab as option to Github. I have no association with them. Just a user of both platforms.

Do they also give discounts to researchers or otherwise show awareness and support of this issue? Might be good to call attention to that fact.

Researcher here: I Use Gitlab because I get everything for no cost as a regular user. With Github I had to pay and get limited features.

GitLab.com and GitLab CE are free to use. GitLab CE is used by many universities. If you use GitLab EE you don't have to pay for student users.

GitLab CE in almost any environment with more than one team (say uni course with many groups doing projects) would need ldap group sync.

The vast majority of universities that use GitLab use CE that has LDAP login but no sync.

Reply on news.ycombinator.com