Comment by DasBlob | Hacker News Reader

Comment by DasBlob | Hacker News Reader

Parent Full thread

While I certainly share your sentiment that both, data and analysis software, should be available, I do see a conflict of interest here that cannot be easily resolved. On the one hand, we do assert (wrongly, IMO) that competition is a necessary element to drive scientific innovation and consequently require scientific actors to adhere to economic principles, i.e. publish more and better-received papers. On the other hand, allowing researchers to keep their competitive advantage, and having exclusive access to research data certainly is, contradicts the principle of reproducability and unneccessarily restricts further research that could be done based on that data. I don't see a way to easily resolve that in the current scientific framework. One (less than optimal) approach, that is frequently taken in astronomy, is to allow exclusive access to telescope data to the members of the consortium or the principal investigators for a certain amount of time, after which it is released to public. As for the software, it's easier. if the analysis method is sufficiently well documented (and it should be in any publication), people with access to the original data can easily attempt to reproduce the results and either way they'll get a publication out of it. So there is no incentive to keep the analysis software but there is a huge incentive to keep the original data.

The solution should rather be that society (and publishers) does not consider papers (or more generally: texts) as research that don't also publish everything possible for independent verification (in this case the source code and raw data). If this is not the case, the text is just a plain marketing statement (in contrast to research) comparable to the marketing statements that are made everyday by lots of companies.

The lack of source code does not make everything produced by software "a marketing statement."

There's more than one way to verify a conclusion.

For example, if I say that RSA-130 = 39685999459597454290161126162883786067576449112810064832555157243 × 45534498646735972188403686897274408864356301263205069600999044599 then you can easily verify it by multiplication - you don't need access to the factorization software I used to determine that result.

If I say that the energy difference between two chemical structure according to the Merck Molecular Force Field (MMFF) is X, then you can use any software which implements the MMFF to verify my statement using the raw data - you don't need the source code that I used.

In fact, it would be better to use an independent implementation to verify my result than it would be to re-run the same code. And there are many programs which implement the MMFF.

If you are used to RDKit, then it will likely be faster to use RDKit to reproduce and verify my numbers than trying to figure out the strange VM I provided, which is based around some home-brew code of mine implemented in MORTRAN.

> For example, if I say that RSA-130 = ... × ... then you can easily verify it by multiplication - you don't need access to the factorization software I used to determine that result.

I didn't write about independent computation, but about independent verification. In your case you need access to a software for multiplying numbers for an independent check. OK, this is rather trivial code to write. But if it weren't that trivial the author would have to provide it.

I gave the RSA example because it's better known and easier to understand than a molecular modeling example.

But I question the meaning of your phrase "independent verification".

If the author has to provide the program used for verification, how is that "independent"? At the very least, isn't there a range of dependencies, from "running the same code in an identical environment produced the same answer" to "running a different code in a different environment produced the same answer"?

Which verification do you trust more? What level of independence do you need before you can trust the results?

Let me give you a few other examples. "We docked 500,000 small molecule structures from CHEMBL against the target using 6,000 GPU days. We took the 1,000 best fits and measured binding affinity. The top 10 structures are shown in table 1. The third is our lead candidate, which reduces the chance of infection in rats from 80% to 10%."

Do you need to be able to validate the docking results in order to accept this paper as science?

"We used ProteinFitter 4.5 in simulated annealing mode to generate the best protein fold to the given constraints. The resulting structure has a R-factor of 2.3Å against the Hodgkin electron density, which is better than any other published model."

Do you need access to ProteinFitter 4.5 to verify this paper? Since all crystallography tools will generate an R-factor for you, why is it critical for the model provider to also supply an environment which contains a tool which does that?

Both are examples where it doesn't matter what's in the magic box, or if the magic box is buggy. It could as well be "I had a dream about a snake tying itself into a knot and eating its own tail. I woke up and realized the structure might be a trefoil. It fits all of the known data and makes intuitive sense and hasn't been considered before, so I'm publishing in the hopes that others can explore it more rigorously."

Is that science? I say "yes".

Now, replace "dream" with "computer program" - is it still science? I still say "yes".

I agree on the propriety of data for a limited time. I worked in the field of Radio Astronomy, and it was common to have a two year period where the data was private to the Primary Investigator (and co-investigators/consortium). I think this is a relatively acceptable approach, given how long it can take to process and fact-check data and papers.

I'm also for the publishing and tracking of code, as in many fields the data is becoming so vast that it's impossible to process or handle manually, and many pipelines and large scale systems are being built to handle this data. This inevitably means that a simple bug in code (or false/incorrect assumption by the author) can result in many lost hours and years of work.

But publishing code can also lead to biases, if there is an unintentional bias in released code or approaches that become popular within a given field.

In the field I worked in, there were two very common pipelines used by a huge majority of researchers, and I always wondered if there might be some unintentional selection effect being introduced. Well, it turns out someone else thought so too, so there's now an effort to rewrite one of these tools from the ground up without using any of the original code. It's only something like 60% complete right now, but hopefully soon it will be complete and we'll get to see if results compare, or if we spot any unintentional effects.

both data and analysis software should be available

I think that goes too far. The data alone is sufficient for independent verification, I don't think we should mandate that scientists publish their tools. However, I would very much support scientists using open tools to begin with. But if the tool is closed, that should not invalidate any results in the paper.

> But if the tool is closed, that should not invalidate any results in the paper.

As I said in https://news.ycombinator.com/item?id=12100008: If it cannot be verified independently as far as possible, this does not mean it is wrong, but also should not be considered as science (since independent verification and the ability to do so as far as possible is central to science). Instead it is just a marketing statement with the author providing evidence for truth.

Marketing statements are also not wrong per se (if they were, at least in Germany this would be a criminal offense because of the "Gesetz gegen unlauteren Wettbewerb (UWG)" (which according to the internet dictionary corresponds to "Act Against Unfair Competition" or "fair trade law")), but you strongly have to trust its author to tell you the truth, since you can hardly verify the truth independently.

Not sure if there is so much to support - they most likely run some archaic Fortran code that only 5 people in the world understand, compiled for some specific cluster, where it runs multiple days for one dataset, and is only numerically stable in that particular environment.

This is why on https://news.ycombinator.com/item?id=12099720 I also suggested to provide a virtual machine image if possible (unluckily providing one is in my opinion not always possible). On the other hand if another team ran the code independently and we really saw that it is numerically unstable in another environment, this would be, in my opinion, a strong case for the importance of providing the source code.

I don't think you understand what elcapitan means by numerically stable.

Numeric stability is a function of the environment. If I take an algorithm designed for 128 bit IEEE binary floats and implement it on a system with 64 bit floats, then it may be unstable.

That doesn't say anything about the correctness of the original method for the original environment. It only means the method wasn't designed for the new environment.

You are used to a world where computers are a commodity, likely based around the Intel architecture, and almost certainly using IEEE floats.

What do you do with a computer program written for Anton, a specialized computer for doing molecular dynamics built using specialized ASICs? https://en.wikipedia.org/wiki/Anton_(computer)

First: IEEE 754 defines exactly (bit per bit) what the result of any floating point operation in each of the five rounding modes as long as no NaN is produced (but in this case only not all bits are defined exactly, but it will still be a NaN).

Unluckily C allows some optimizations to be done to code containing floating point code that violates this principle of bit-for-bit reproducability of floating point code. If this causes problems the code can be considered as numerically unstable.

In other words: Writing code that uses the IEE 754 defined "gold standard" in its code should be the goal scientific results.

The correct way for independent checking, if we really need 128 bit floating point numbers, is to wrap these operations by a software floating point library.

> It only means the method wasn't designed for the new environment.

If this property (what kind of environment the algorithm depends on) of the algorithm is not documented properly (and an explanation is given why we need this property (say: size of floating points, in particular unusual floating point sizes as 128 bit) for the algorithm to work correctly) any independent reviewer should better not assume these properties to hold. If we indeed find out that this causes problems, this is a strong sign to me that there might be subtile errors in the code. In other words: We should be really careful of the results that the algorithm gave.

> What do you do with a computer program written for Anton, a specialized computer for doing molecular dynamics built using specialized ASICs?

I gave the answer in the post above: "[U]nluckily providing [a virtual machine] is in my opinion not always possible".

I don't follow the point of your first four paragraphs. I specifically said the code was numerically stable on 128 bits - why are you introducing those irrelevant facts, like how C compiler details can make some algorithms unstable?

Really it comes down to what you mean by "another environment" and what your case is for access to the source code.

Let me give a simpler example. I wrote code which expects ILP32. It's stable under ILP32. However, it produces different and non-deterministic answers under LP64. That is "another environment."

Does the numerical instability under a different environment cast doubt on its validity in the original environment? Why is it a strong sign that there may be subtle errors?

Reply on news.ycombinator.com