Hacker News Reader

Comment by astrodust | Hacker News Reader

Yeah, I'm sure you have the time, inclination and expertise to look over billions of data points and make your own conclusions.

View on HN

A layperson named Steve McIntyre did just that, after reading the IPCC's 3rd Assessment Report and seeing the prominently featured "Hockey Stick" based on a paper by Mann et al., and wondering, hmm... perhaps this study is reproducible?

The rabbit hole that McIntyre and his colleagues have found themselves in over the last 10 years has, in my view, demonstrated the value of hacker-types taking an interest in climate science.

See here for a paper detailing the point where McIntyre's descent into the rabbit hole began: http://www.uoguelph.ca/~rmckitri/research/McKitrick-hockeyst...

From your link:

"Of crucial importance here: the data for the bottom panel of Figure 6 is from a folder called CENSORED on Mann’s FTP site. He did this very experiment himself and discovered that the PCs lose their hockeystick shape when the Graybill-Idso series are removed. In so doing he discovered that the hockey stick is not a global pattern, it is driven by a flawed group of US proxies that experts do not consider valid as climate indicators. But he did not disclose this fatal weakness of his results, and it only came to light because of Stephen McIntyre’s laborious efforts."

Geez, that's so damning.

IIRC, despite the criticsim, the 'hockey stick' study and its author were proven right. Lots of criticism doesn't make it wrong, especially in politicized debates.

How do you mean? Did you read the paper I linked to? In what way were Mann et al. "proven" right?

If read in isolation, the paper seems pretty convincing that something is amiss. However, it looks like some other scientists have generally confirmed the findings from Mann et al.: https://en.wikipedia.org/wiki/Hockey_stick_controversy#Furth...

I haven't read those other papers.

If you can link to the raw data so that I can wget it I will take a look at it.

You should just stop with this line of thinking. You have to understand the scientific context and the limits of the data to learn anything. (Source: I know several contributors to the IPCC AR5 report, and I try to have the proper respect for their expertise.)

Here's a good place to start: ftp://ftp.cdc.noaa.gov/Datasets/20thC_ReanV2c/Monthlies/gaussian/monolevel/air.sfc.mon.mean.nc

It's netcdf global monthly mean air temps. Pick some grid points and plot time series. Have fun.

Thanks, Ill take a look. I hate HDF format !!!!

You aren't capable of interpreting it, as is no one individual which is rather the point you're missing.

I'm not sure I understand. Can you explain?

Are you saying people can not understand this data?

You know, why don't you help us out and wget some data from CERN and give them some insight into their ongoing LHC experiments.

I'm sure they could use a nudge in the right direction, and I bet a little bit of Perl or Python is all they need to solve some deep mysteries.

Oh, and you have 6,000TB of disk space to download the data from the ATLAS sensor, right?

Once you're done that there's all kinds of data regarding cancer treatments you can crunch through. I bet that's a weekend of work at the outside.

You're right. I couldn't process 6 petabytes in perl.

I'd use C for that amount of data.

Great, so you're capable of writing highly parallel cluster-scale code in C that does intensive precision numerical analysis? Most choose vectorized FORTRAN or a combination of C++ and CUDA, but hey, knock yourself out.

CERN has a 3700 core supercomputer to crunch through this kind of data. You can rent that on Amazon for about $800 an hour, so I guess you're good to go.

Sorry to be so harsh here. While there's always desirable amount of "constructive naivety" necessary to try the impossible, you need to recognize that there's considerable amounts of expertise required to process and analyze data of this complexity at scale.

This is not like a movie where six minutes of furious typing can solve any problem.

I bench marked fortran vs. x86_64 SSE extensions in C and .... C's fine.

I'd rather have local clusters than Amazon or Google "cloud" any day of the week.

Spotting a methodology bias is not that hard.

Why the heck would you need CUDA ? NVIDIA ???

C'mon man.

Are you simply trolling at this point?

If you're so confident in your ability to process this sort of data, please, post your follow-up on HN.

Reply on news.ycombinator.com