How a word sparked a four-year saga of climate fact-checking | Hacker News Reader

How a word sparked a four-year saga of climate fact-checking

theconversation.com

61 pointsnl59 comments

I see a large problem in the fact that (as it seems) the original data and the computer code is not published, too. This is in my opinion a contradiction to the scientific principle that any result or conclusion has to be able to verified independently as much as possible. Though not publishing data and computer programs (and ideally a virtual machine for independent execution of the program code, too) is unluckily accepted practise in science.

If it were otherwise any critic of the result could reapply the computer program to the non-detrended data and see that the conclusion will not change. On the other hand, if the computer program and data is not published he has to trust the conclusions that the researchers came to. So the only way for an independent person to check the result is to find indirect evidence that the conclusion could be wrong. And finding out that a different data set than (wrongly) claimed was used is exactly such indirect evidence (and in my opinion nearly as far as one is able to if one doesn't have access to the used data and computer programs).

While I certainly share your sentiment that both, data and analysis software, should be available, I do see a conflict of interest here that cannot be easily resolved. On the one hand, we do assert (wrongly, IMO) that competition is a necessary element to drive scientific innovation and consequently require scientific actors to adhere to economic principles, i.e. publish more and better-received papers. On the other hand, allowing researchers to keep their competitive advantage, and having exclusive access to research data certainly is, contradicts the principle of reproducability and unneccessarily restricts further research that could be done based on that data. I don't see a way to easily resolve that in the current scientific framework. One (less than optimal) approach, that is frequently taken in astronomy, is to allow exclusive access to telescope data to the members of the consortium or the principal investigators for a certain amount of time, after which it is released to public. As for the software, it's easier. if the analysis method is sufficiently well documented (and it should be in any publication), people with access to the original data can easily attempt to reproduce the results and either way they'll get a publication out of it. So there is no incentive to keep the analysis software but there is a huge incentive to keep the original data.

The solution should rather be that society (and publishers) does not consider papers (or more generally: texts) as research that don't also publish everything possible for independent verification (in this case the source code and raw data). If this is not the case, the text is just a plain marketing statement (in contrast to research) comparable to the marketing statements that are made everyday by lots of companies.

The lack of source code does not make everything produced by software "a marketing statement."

There's more than one way to verify a conclusion.

For example, if I say that RSA-130 = 39685999459597454290161126162883786067576449112810064832555157243 × 45534498646735972188403686897274408864356301263205069600999044599 then you can easily verify it by multiplication - you don't need access to the factorization software I used to determine that result.

If I say that the energy difference between two chemical structure according to the Merck Molecular Force Field (MMFF) is X, then you can use any software which implements the MMFF to verify my statement using the raw data - you don't need the source code that I used.

In fact, it would be better to use an independent implementation to verify my result than it would be to re-run the same code. And there are many programs which implement the MMFF.

If you are used to RDKit, then it will likely be faster to use RDKit to reproduce and verify my numbers than trying to figure out the strange VM I provided, which is based around some home-brew code of mine implemented in MORTRAN.

> For example, if I say that RSA-130 = ... × ... then you can easily verify it by multiplication - you don't need access to the factorization software I used to determine that result.

I didn't write about independent computation, but about independent verification. In your case you need access to a software for multiplying numbers for an independent check. OK, this is rather trivial code to write. But if it weren't that trivial the author would have to provide it.

I gave the RSA example because it's better known and easier to understand than a molecular modeling example.

But I question the meaning of your phrase "independent verification".

If the author has to provide the program used for verification, how is that "independent"? At the very least, isn't there a range of dependencies, from "running the same code in an identical environment produced the same answer" to "running a different code in a different environment produced the same answer"?

Which verification do you trust more? What level of independence do you need before you can trust the results?

Let me give you a few other examples. "We docked 500,000 small molecule structures from CHEMBL against the target using 6,000 GPU days. We took the 1,000 best fits and measured binding affinity. The top 10 structures are shown in table 1. The third is our lead candidate, which reduces the chance of infection in rats from 80% to 10%."

Do you need to be able to validate the docking results in order to accept this paper as science?

"We used ProteinFitter 4.5 in simulated annealing mode to generate the best protein fold to the given constraints. The resulting structure has a R-factor of 2.3Å against the Hodgkin electron density, which is better than any other published model."

Do you need access to ProteinFitter 4.5 to verify this paper? Since all crystallography tools will generate an R-factor for you, why is it critical for the model provider to also supply an environment which contains a tool which does that?

Both are examples where it doesn't matter what's in the magic box, or if the magic box is buggy. It could as well be "I had a dream about a snake tying itself into a knot and eating its own tail. I woke up and realized the structure might be a trefoil. It fits all of the known data and makes intuitive sense and hasn't been considered before, so I'm publishing in the hopes that others can explore it more rigorously."

Is that science? I say "yes".

Now, replace "dream" with "computer program" - is it still science? I still say "yes".

both data and analysis software should be available

I think that goes too far. The data alone is sufficient for independent verification, I don't think we should mandate that scientists publish their tools. However, I would very much support scientists using open tools to begin with. But if the tool is closed, that should not invalidate any results in the paper.

> But if the tool is closed, that should not invalidate any results in the paper.

As I said in https://news.ycombinator.com/item?id=12100008: If it cannot be verified independently as far as possible, this does not mean it is wrong, but also should not be considered as science (since independent verification and the ability to do so as far as possible is central to science). Instead it is just a marketing statement with the author providing evidence for truth.

Marketing statements are also not wrong per se (if they were, at least in Germany this would be a criminal offense because of the "Gesetz gegen unlauteren Wettbewerb (UWG)" (which according to the internet dictionary corresponds to "Act Against Unfair Competition" or "fair trade law")), but you strongly have to trust its author to tell you the truth, since you can hardly verify the truth independently.

Not sure if there is so much to support - they most likely run some archaic Fortran code that only 5 people in the world understand, compiled for some specific cluster, where it runs multiple days for one dataset, and is only numerically stable in that particular environment.

This is why on https://news.ycombinator.com/item?id=12099720 I also suggested to provide a virtual machine image if possible (unluckily providing one is in my opinion not always possible). On the other hand if another team ran the code independently and we really saw that it is numerically unstable in another environment, this would be, in my opinion, a strong case for the importance of providing the source code.

I don't think you understand what elcapitan means by numerically stable.

Numeric stability is a function of the environment. If I take an algorithm designed for 128 bit IEEE binary floats and implement it on a system with 64 bit floats, then it may be unstable.

That doesn't say anything about the correctness of the original method for the original environment. It only means the method wasn't designed for the new environment.

You are used to a world where computers are a commodity, likely based around the Intel architecture, and almost certainly using IEEE floats.

What do you do with a computer program written for Anton, a specialized computer for doing molecular dynamics built using specialized ASICs? https://en.wikipedia.org/wiki/Anton_(computer)

First: IEEE 754 defines exactly (bit per bit) what the result of any floating point operation in each of the five rounding modes as long as no NaN is produced (but in this case only not all bits are defined exactly, but it will still be a NaN).

Unluckily C allows some optimizations to be done to code containing floating point code that violates this principle of bit-for-bit reproducability of floating point code. If this causes problems the code can be considered as numerically unstable.

In other words: Writing code that uses the IEE 754 defined "gold standard" in its code should be the goal scientific results.

The correct way for independent checking, if we really need 128 bit floating point numbers, is to wrap these operations by a software floating point library.

> It only means the method wasn't designed for the new environment.

If this property (what kind of environment the algorithm depends on) of the algorithm is not documented properly (and an explanation is given why we need this property (say: size of floating points, in particular unusual floating point sizes as 128 bit) for the algorithm to work correctly) any independent reviewer should better not assume these properties to hold. If we indeed find out that this causes problems, this is a strong sign to me that there might be subtile errors in the code. In other words: We should be really careful of the results that the algorithm gave.

> What do you do with a computer program written for Anton, a specialized computer for doing molecular dynamics built using specialized ASICs?

I gave the answer in the post above: "[U]nluckily providing [a virtual machine] is in my opinion not always possible".

I don't follow the point of your first four paragraphs. I specifically said the code was numerically stable on 128 bits - why are you introducing those irrelevant facts, like how C compiler details can make some algorithms unstable?

Really it comes down to what you mean by "another environment" and what your case is for access to the source code.

Let me give a simpler example. I wrote code which expects ILP32. It's stable under ILP32. However, it produces different and non-deterministic answers under LP64. That is "another environment."

Does the numerical instability under a different environment cast doubt on its validity in the original environment? Why is it a strong sign that there may be subtle errors?

I agree on the propriety of data for a limited time. I worked in the field of Radio Astronomy, and it was common to have a two year period where the data was private to the Primary Investigator (and co-investigators/consortium). I think this is a relatively acceptable approach, given how long it can take to process and fact-check data and papers.

I'm also for the publishing and tracking of code, as in many fields the data is becoming so vast that it's impossible to process or handle manually, and many pipelines and large scale systems are being built to handle this data. This inevitably means that a simple bug in code (or false/incorrect assumption by the author) can result in many lost hours and years of work.

But publishing code can also lead to biases, if there is an unintentional bias in released code or approaches that become popular within a given field.

In the field I worked in, there were two very common pipelines used by a huge majority of researchers, and I always wondered if there might be some unintentional selection effect being introduced. Well, it turns out someone else thought so too, so there's now an effort to rewrite one of these tools from the ground up without using any of the original code. It's only something like 60% complete right now, but hopefully soon it will be complete and we'll get to see if results compare, or if we spot any unintentional effects.

>any result or conclusion has to be able to verified independently as much as possible

Well, collecting the data yourself and writing the code yourself would be even more independent.

> Well, collecting the data yourself and writing the code yourself would be even more independent.

I wrote "any conclusion". This does not preclude collecting the data or writing the code yourself (and indeed you should if you doubt, say, the data). But it implies that the code that was used to come to the conclusion must be available.

> Meanwhile, our team received a flurry of hate mail and an onslaught of time-consuming Freedom of Information requests for access to our raw data and years of our emails, in search of ammunition to undermine and discredit our team and results.

> The mammoth process involved three extra rounds of peer-review and four new peer-reviewers. From the original submission on 3 November, 2011, to the paper’s re-acceptance on 26 April, 2016, the manuscript was reviewed by seven reviewers and two editors, underwent nine rounds of revisions, and was assessed a total of 21 times – not to mention the countless rounds of internal revisions made by our research team and data contributors. One reviewer even commented that we had done “a commendable, perhaps bordering on an insane, amount of work”.

So one side you do this "mammoth" and "insane" task to review paper. On other side you block people from doing independent audit?

Climate studies decide how billions (if not trillions) of dollars are spend. People should be allowed to cross examine raw data and program source code.

If its coming with hate mail and or has a generally offensive tone, I don't think it unreasonable to be unhappy. The author doesn't appear to have blocked access. One can cross examine without being a dick.

Read linked article, offensive language was used AFTER they refused to provide raw data. It took lawsuit to get them.

Context was not provided however - an article that calls researchers bimbos is not one I take on its word. Was the raw data refused? And why was it refused? Perhaps because they were, as the original article states, prepping the study for publication, or because they had put it on hold? Or because they put the paper on hold and wanted to focus on doublechecking it before spending work on making the data available? Perhaps they still had to retrieve the rights to publish the data, or find a way to publish it in a good way? Context is lacking to state that there was a non-benign refusal of data.

I read the article. I saw nothing about a lawsuit.

I also read that the final paper wasn't published until very recently. Why should a research group be forced to distribute raw data years before the paper is finished?

In the US, for example, http://biotech.law.lsu.edu/IEEE/ieee36.htm says "The term, Research Data, is defined as the recorded factual material commonly accepted in the scientific community as necessary to validate research findings. It does not include: preliminary analyses, drafts of scientific papers, ... "

I know nothing about the Australian equivalent.

Paper was published first time several years ago. Lawsuit is mentioned in other article.

My understanding is that for climate science, a lot of effort is spent to debunk claims, by well-financed groups who will publish without independent peer-review but with a clear goal in mind. Allowing those groups to access the same data means that the researchers will have to spend a lot more effort explaining why those claims are unfounded. That means the researchers are not really encouraged into sharing the data.

There is a more general issue in science or any data-intensive practice (Google ranking is another example) where the argument “People should be allowed to cross examine raw data” fails to acknowledge that ‘People’ generally do not have the ability or know-how to, and those who can generally have interested financial backing. In many cases, it is positive (Open Source Map is a good example of that: my employer is interested in identifying, say, bike parking spots) but there are examples where de-anonymisation, competitive pressure, make “sharing data” a pragmatic question that needs more than principles.

Will more people who can contribute positively to climate science help if they had free access to those data? It is the case for astronomy; I am less convinced for climate science.

Paranoid talks about "well financed groups" have no place in science.

Climate science is way more important than astronomy. It should have much higher standards for transparency, not other way around.

Get rid of copyright law, then everyone can have the data. People keep ignoring the fact that you can't just release data like that all over the place, because researchers often do not have the IP rights to do so in a general way (whereas within the field other researchers probably do have IP rights to use the same data).

> People keep ignoring the fact that you can't just release data like that all over the place, because researchers often do not have the IP rights to do so in a general way

While I agree that this is a problem in the current copyright law, there is still a way for researchers to "circumvent" this copyright restriction without violating the scientific principle of independent verifiability that I outlined in https://news.ycombinator.com/item?id=12099720:

Let's say that the data is generally distributed by the owner in a standard format, say, a zip archive, which we will name foo.zip here (for simplicity). If you can't distribute the data itself, you write into your research paper a short explanation of the license terms that disallow the distribution of foo.zip. But you also add a notice what the SHA256 sum of foo.zip is. If some person who has also access to the data (say, from another university) has interest that the research result can be checked independently, they will leak foo.zip somewhere (I can imagine that sci-hub would be willing to provide server space for this).

This way access for independent verifiers can be "assured" without the original researchers having to violate copyright law.

The US[1] doesn't recognized the "sweat of the brow" doctrine for copyright. There is no copyright on a "mere collections of facts", and no IP rights cover collected data.

Give that, what role does copyright law play (at least in the US) in preventing data sharing? I thought it was limited based on patient confidentiality, collaborator agreements, end-user agreements, desire to maximize publications from the same data, and other non-copyright issues which were at play.

([1] Other countries are more restrictive than the US in this regard.)

If astronomy includes the possible detection of previously unknown sizeable objects in space which might put an end to our activities on this planet, then I don't agree. Climate alarmism (that's not intended as a perjorative word: it's sensible to be alarmed if there is something to be alarmed about) is based on the validity or not of statistical conclusions regarding climate data. Objects in space can and do collide with Earth if that's their trajectory. No theory or hypothesis is required - just data.

> Paranoid talks about "well financed groups" have no place in science.

It is not paranoid, it isn't even secret.

Koch Brothers Exxon Heritage Foundation

are just 3 examples. It is absolutely within the purview of science to consider these things, as is anything. You can't just declare it off limits. We observe and have records of these groups paying people to write fake sciency-sounding reports on climate science constantly.

There is absolutely nothing paranoid about systematic and well documented efforts from the oil industry to discredit scientists.

> It should have much higher standards

Yes, but for the quality of the result, not intuitions set-up as principles to replace the real pillars of science: experimentation and peer-review. Transparency is only useful if it helps getting peers; in this case, there are well documented reasons to doubt it, and that the people benefiting from that transparency are not just enemies of science, but of humanity.

> Will more people who can contribute positively to climate science help if they had free access to those data?

I think if we only consider it as science if it can be verified independently, also the "opposite side" (your "well-financed groups") has to provide evidence of the same high standard, which would strongly increase the level on which the debate is pursued (i.e. the opposing side also cannot get away with outlandish "evidence").

But then I wonder, how do you enforce the latter rule? ('the "opposite side" has to provide evidence of the same high standard'). Generally it seems like a lot of the media at least, which serve as a gatekeeper of sorts for which information gets disseminated to the broader public, do not really seem capable of or interested in enforcing such a rule. Policymakers, who should be quite interested in climate studies, might be more persuadable to having standards, at least when they're not in some interest group/ideological pocket.

(EDIT: of course, journals could serve as enforcers, but then the "opposite" should have to publish their science in well-policed journals. The OP's article claims that this generally isn't the case) (EDIT: disclaimer, I am not trying to argue against the broader point that data should be open)

I think it is easy (in the sense: takes little time) to check if the opposite side provides the data and source code of the computer program that lead them to the conclusions. If not, one can simply give out this reason.

If they do, on the other hand, I consider it as sufficiently scientifically intesting to analyze what different kind of model their program used (that lead to different conclusions) and why this model is better/worse etc.

The problem is that budgets for science are controlled by the relatively uneducated (as is seemingly the case for most legislators), at least when it relates to the fields of science. And they are often easily swayed by unreviewed and unverified articles in public magazines/newspapers/blogs/etc. Even more so for the public, who hold a strong sway over legislators due to elections and what-not.

So it is in no way easy to force the opposition to uphold the stringent requirements of science, when they are clearly ignoring many of the facts staring them in the face (at least in the case of climate change deniers).

Indeed. Though I think we need to resist the slide towards post-truth politics.

That seems to be a sane way to go about it; but that is what you would do. Question is: would others. The broader argument that Bertil makes (I think) is that these decisions aren't being made based on aversion to your individual choice of behavior but based on aversion of a sort of media/public opinion war where one group isn't acting in good conscience and where the battlefield is not level (because the "opposite side" is well-financed and avoids peer review). I am not necessarily agreeing or disagreeing with that, but if you accept that premise, for the level on which the debate would be pursued to strongly increase, it would be necessary that someone held up both sides to the same standard.

There has been enough raw data and code out there for an amateur to confirm that global warming is real for decades. I did it myself ~10 years ago. Here is a small subset of data and code that is public domain:

http://www.realclimate.org/index.php/data-sources/

I'm personally inclined to support the climate change folks, being a scientist and stuff, but it's a bit crap to claim that it was a one word typo when the original text apparently read:

> For predictor selection, both proxy climate and instrumental data were linearly detrended over the 1921–1990 period to avoid inflating the correlation coefficient due to the presence of the global warming signal present in the observed temperature record. Only records that were significantly ([pre]p<0.05[/pre]) correlated with the detrended instrumental target over the 1921–1990 period were selected for analysis.

That isn't a typo, it is just being wrong. I'm glad they re-did the work to confirm it was still true, but it's hard to argue with folks who didn't think the claim was supported by the data.

Well, what you expect from people who happen to turn around in their (unverifiable) models 180 degrees (Amazon jungle net produces oxygen! No, it's a net consumer!) and who routinely produce models that use some physical value (heat capacity in greenhosue effect), but fail to account for major substances influencing the value (water concentration in atmosphere)?

I'm not an expert on the history of opinions on amazon oxygen, but

>but fail to account for major substances influencing the value (water concentration in atmosphere)?

is absolutely false.

OK. Now show me any study that accounts that the water in the atmosphere (0.4% on global level, after Wikipedia, not 1% on the sea level) has ten times higher concentration than methane (0.00018%) and CO2 (0.039%) combined, and almost twice as big heat capacity (75 J/(mol * K)) as each of those (36 J/(mol * K) for methane, 37 J/(mol * K) for CO2). To sum up, heat capacity of atmospheric water being twenty times bigger than CO2 and CH4.

The one word typo was in the code, not the paper. It was setting DETREND to FALSE instead of TRUE. Their assertions were based on thinking they were working with detrended data instead of the raw data that they actually used.

In which case, one shouldn't write

> Instead of taking the easy way out and just correcting the single word in the page proof, we asked the publisher to put our paper on hold

Global warming denialists are attempting a denial-of-service attack against global warming science. The goal is to keep the science unsettled so no major policy changes are made while those making the most profits from climate-changing activities continue to do so.

I was about to respond to your comment and argue that some of them generally have good intentions, and that your comment was paranoia about a conspiracy....but then I realized you pretty much nailed it. I used to doubt climate science...15 years ago. Now I don't.

Yup; just like tobacco, acid raid, DDT, and the hole in the ozone layer. Very very sad. It shows you how powerful greed is, when people are willing to sacrifice humans lives and even the future of our planet just to maximize profits.

This is one of the main failures of our capitalist economy, IMO. The system needs a built in way to assign a value on sustainability and human well being, rather than just profits.

See https://en.m.wikipedia.org/wiki/Merchants_of_Doubt for mor on that, if you are interested. Very good book.

For another view of this issue: https://climateaudit.org/tag/gergis/

General observation is that neither the mathematical models nor the data is perfect.

For example, oceanic thermocline data, from what I understand, is not really collected.

We have an imperfect model being fed incomplete data? Why be surprised at debate over it?

People laugh today at ether and phlogiston debates, but such acrimonious debate pushed forward science to the point where we had exactly the right answer.

The problem is that we're not really having debate, as this article makes clear. We are having scientists attacked by 2 groups: climate deniers funded by large corporate masters, and random groups of crackpots. Real debate would be good but I think we're well past that point.

No idea why this is a surprising, even controversial result. We are are on the warming side of a recent (geologically speaking) ice age (actually glacial period) and we expect long term temperatures to be trending ever upwards.

No we're not. The complete opposite in fact: the planet would naturally be cooling now were it not for the increased amount of greenhouse gases in the atmosphere.

> "Technically speaking, we’re living during an ‘Ice Age’ today – in the sense that we have a world with glaciers. However, the present time is a relatively warm phase within a period of geological time when glaciers and ice sheets have typically been larger and more extensive than now."

Taken from: http://www.rgs.org/OurWork/Schools/Teaching+resources/Key+St...

There are other sources, but this is one example that makes it clear we have been in a warming phase the last few thousand years.

Also from the same site:

> "Since the industrial revolution in the 18th century, the concentration of carbon dioxide in the atmosphere (an important greenhouse gas) has risen significantly above the level it would be naturally. It now stands at over 400 parts per million (ppm) in the atmosphere. Evidence from ice cores (see Lesson four) tells us that the normal level for CO2 in the atmosphere during ‘interglacial’ times (such as the Holocene in which we now live) is 270 to 290 ppm, and that at no time over the last 800,000 (the time covered by ice cores) has the CO2 level been as high as it is now. (CO2 is thought to be at its highest level for three to five million years.)"

http://www.rgs.org/OurWork/Schools/Teaching+resources/Key+St...

You misunderstand me - I am not talking about man made climate change. I am saying that having data that shows current temperatures are highest ever in 1000 years is not surprising, given that temperatures have been rising for last several thousand years.

If one had a time machine and went back 1000 years, took temperatures and compared them to those from previous thousand years, one would not be surprised to see that on average they were higher.

No one is misunderstanding you, you are just wrong.

Even if you were right and we were in a "warming" phase, the velocity of the change should be on a geological scale; that is, very very very tiny. We are seeing rates of warming orders of magnitudes higher, which cannot possibly be anything but man made.

At no point have I mentioned scale of change or rate of change. That is irrelevant to the point I am making: that during a warming phase, average temperatures will be the highest they have ever been at the end of the time period under consideration. So a result confirming that is hardly news.

4000 years ago there were still mammoths and far greater glaciation than now, so clearly climate has been warming over the last few thousand years.

Sure. But if someone is saying "the house is on fire" and you reply, "Ah, Yes, clearly we should expect it to be getting warmer, after all it's springtime now", it's not much of addition to the conversation.

No, that example is correct but you misread it: we're in a warmer period. Not a warming period. The warming stopped about eight thousand years ago. We've been in that warmer period since then.

For example: https://en.wikipedia.org/wiki/Temperature_record#/media/File...

We would naturally now have started cooling again were it not for the additional greenhouse gases in the atmosphere. As I've already explained.

Reply on news.ycombinator.com