Hacker News Reader

Comment by BenoitEssiambre | Hacker News Reader

God I hope this becomes a trend in other fields as well.

Spending time in universities has made me very cynical of the research that comes out of them. There is just too much incentives for profs to ignore biases in their research. I've seen it happen many times. For them, it is the difference between becoming a prof or a lab tech and it is literally worth millions.

You can not trust any research done where the career of the researcher depends on them finding results.

Universities, by hiring based on research credentials that currently translate roughly in the amount of positive results a person has generated, completely render worthless the research going on in their departments.

Aggravating the situation is the fact that the peer review system is too incestuous to be relied on especially when the peers are probably also 'bias ignorers' with incentives to keep the flaws in the system.

I strongly believe that for hiring purposes, the research skills of profs should be evaluated on criterion that are incidental to research. Mostly math, probability, statistics (yes even for psychology) then methodology skills and also maybe leadership, communication skills and dedication to science (last only because it is difficult to measure).

update:typos

View on HN

"I take the hard view that science involves the creation of testable hypotheses" - Michael Crichton

I'm not trying to be a troll. Really, I promise. But I really hope that the next field they do this in, is in Climate Science, specifically articles on climate change. It seems like bulk of their work is based on computer models and I just don't know how you're supposed to replicate any of that. How do you isolate the millions of variables to show cause and effect with a closed source computer model. That is my biggest problem with climate science.

> "It seems like bulk of their work is based on computer models"

Then it seems that you are confused. Climate science is about gathering data, putting forth a testable hypothesis and seeing how future data fits the prediction. Yes, they use computer models. In the same way that you or I might gather data from a wind tunnel test and make a computer model of wind flow from that data to extrapolate a theory and create new testable hypotheses of a more-aerodynamic shape.

But so long as new data is used to test those hypotheses, the use of a computer model is irrelevant. Science is being done. Testable hypotheses are being created and new data is being compared to those predictions. As long as we aren't running around half-cocked, believing the predictions of hypotheses that have not yet been tested, there's no concern.

And the core hypotheses under the umbrella of Climate science have been tested. Some disproved by data, leading to changes in thinking, further research and new testable theories. Some supported by data. None of the 'alternate' hypotheses can say the same. (No, it's not the solar cycle. No, it's not distorted temperature readings from data stations in urban areas.)

The very reason that there is a general consensus around the so called 'tent-pole' theories in Climate Science is precisely because they have been tested and have been shown to make accurate predictions and continue to do so. Some of them, for decades now.

Somehow I find that thought uncomfortable that the complexity of factors impacting something as complex as climate is getting reduced to a wind tunnel test.

One is a highly controlled scenario where all(most all) the externalities can be controlled. The other some extremely complex modeling, which will, at least in my view, always depend on some consensus among the participants which factors are more important than others.

I am far from a climate change skeptic, but the best example for the fundamental failure of grasping complex scenario with some factor-based prediction systems has been demonstrated with the complete failure of economics in predicting something as "simple" as the financial crisis. But then again, I am coming from a social science perspective assuming that as soon as humans are involved data and data analysis will always be subjective.

You are not "far from a climate change skeptic". You are a climate change skeptic. At best, your post is the vague claim that complex systems cannot be modeled or tested empirically, and is empirically wrong in its claim that no-one predicted the financial crisis.

Based on the second claim of stfu’s post, “as soon as humans are involved data and data analysis will always be subjective”, I think he is really trying to highlight the unpredictability of a model that must generalize human choices using statistics. Since even conservative scientists will agree that climate change has some anthropogenic dimension, we should understand the limitations of modeling it.

Being skeptical of the model, however, does not prohibit you from accepting what the model is suggesting. Historical data and scientific opinion can be enough to convince us that climate change is occurring. But no model can predict how our reaction to that knowledge will affect climate change in the future.

Call me a skeptic too in that case. Chaotic systems are inherently unpredictable to us, though they are ofcourse deterministic.

The climate is chaotic (http://www.mendeley.com/research/nonlinearities-feedbacks-an...).

Chaotic systems have a horizon of predictability (http://www.britannica.com/EBchecked/media/3227/Sensitivity-o...), so climate models are still useful to an extent. They will tell us, with some degree of accuracy, roughly what will happen in the near future. The long term remains a mystery and there's little that can be done about that.

It goes much too far to say that the climate is chaotic and therefore totally unpredictable in the long run. After all, if you buy that whole hog, you're also led to say that if you moved the Earth to Mercury's orbit it would be impossible to predict that the Earth would get hotter, in the long term. This results from the flaw of confusing weather (which is indeed chaotic) with statistics about weather (i.e. climate).

A butterfly flapping its wings can cause a hurricane. But it can also prevent a hurricane. All those perturbations don't change the fact that, within an order of magnitude, we can predict how many hurricanes will occur in a given year. Alternatively, think about the classical three body problem: even though it's the preeminent example of a chaotic system, you can still make broad predictions about it. Center of mass, average speed of the bodies, average collision time. This isn't some obscure controversial field, either: in physics, these emergent statistical properties form the basis of statistical mechanics and therefore thermodynamics as a whole.

Chaos doesn't mean you have to throw your hands up and give up on making any long term predictions at all about a system.

>Chaos doesn't mean you have to throw your hands up and give up on making any long term predictions at all about a system.

But we can't tell what the weather will be in 45 days in New York.

One of the attributes of chaotic systems is their self-similarity. The behavior on one level is similar to the behavior on another - in this case weather is similar to climate. We can make predictions about the weather in the next few days, just as we can with climate, but in the next few months or years (or centuries or millenia for climate) is another story. The tiny pertrubations in the system which we do not model (the butterflies) end up throwing our predictions off in the long run.

Yes, and that's why nobody is claiming to be able to predict the weather in the long run. Nevertheless, some things still can be predicted, even in chaotic systems. What is your point?

My point is that pollution should be reduced, climate models should still be developed and that at the same time we shouldn't take their predictions as dogma and continue to question them.

I agree with all of what you say above and am skeptical of the models as they exist now, as another comment of mine elsewhere suggests.

I'm more interested in the theoretical question here, though. I'm unconvinced that self-similarity in chaotic systems means that since weather is a chaotic system, climate is itself a chaotic system that's unpredictable in the long term. One way of thinking about chaos is that at some point in the future, a system's position in phase space will become wholly uncorrelated with the system's initial position.

So for weather, tomorrow's weather is correlated with today's weather, but weather two weeks from now (or whenever) is wholly uncorrelated. But with climate you do end up with persistent correlations, year on year, decade on decade, millenia on millenia. Temperate regions show (anti-)correlations of deviation from annual average temperature between 180 day periods, and daily solar energy impingement is correlated with positive deviation from global mean temperature.

I've always made a mental analogy between that and a gas (the n-body problem): even though the positions of gas molecules rapidly become uncorrelated from their starting position, certain statistical properties--e.g., the pressure the gas exerts on the walls of its container--are highly correlated from one moment to another. Not just initially but into the indefinite future.

I'm open to arguments that that analogy is flawed and could see some assumptions that I'm not 100% confident in, but I don't see any obvious flaws. What's going wrong here, in your view?

The point is, while the models are inaccurate, the data (and there is a lot of data) indicate that warming is happening. This is shown not merely in temperature measurements; but in ice coverage changes, both in the Arctic Ocean and at mountaintops; and in biological change: increased growing seasons, earlier spring flowering, earlier bird migrations (and in some cases increased overwintering bird populations), increased ranges of species, including pests; not to mention permafrost melting and increased droughts and other severe weather anomalies in many areas.

That argumentation is a bit silly. Chaotic systems are not necessarily unpredictable. For example take the pendulum that swings among the magnets (one of those classic examples). So the exact path of the pendulum might be unpredictable, but it is predictable that it will reach equilibrium eventually. What I mean is: some aspects can be predictable while others are not predictable. Therefore your argument is wrong.

Predicting the financial crisis is probably orders of magnitude more complex that climate change. Climate science is based on well known (although complex) systems of equations and laws that have been studied independently from each other for hundreds of year (heat diffusion, fluid flow, etc...). The tricky part is combining all the pieces is a system the size of the planet where a lot of quantities are still relatively unknown.

Compare that with the financial crisis where there are no hard laws and the behavior of the system depends directly and inextrinsibly from how a large group of people scattered around the world will reach to certain hypothetical scenario in an unknown point in history. Will investor A be rational? Panic? How about B? What happens if B sees A panicking? Or vice versa?

I'm not an economist, by can give you an example on epidemiology I worked on recently. Starting with one of the simplest epidemic models (SIR), as soon as you add a simple behavioral assumption (people can become afraid of the disease and behave more carefully) the behavior of the systems changes completely.

Predicting the financial crisis is probably orders of magnitude more complex that climate change.

Orders of magnitude simpler, I'd think. "If real wages and household incomes keep dropping while household indebtedness keeps rising, you will eventually have a financial crisis."

Marx never needed any math to work this out, just the good sense to realize that money never appears out of thin air.

Don't mistake a bubble with a financial crisis.

There's as much content to that prediction as there is to the prediction "if you release carbon dioxide into the environment at an accelerated rate, the average temperature of the Earth will increase".

What people are meaning by "prediction" here is something much more specific than what you mean and you know it.

Complete failure of neoclassical economics, perhaps. There were plenty of people who predicted the GFC with varying degrees of accuracy; Steve Keen being but one.

The problem is that these people (and others) don't know enough to realize their economic predictions are worthless. There is truth to the old saw, "Economists are good at predicting recessions. They've predicted eight of the last three. . . "

In many cases the prognosticators are fully aware that their predictions are worthless but persist in the facade to build up a reputation. It takes only a few lucky coincidences to get set up for life. People love to hear bold predictions.

> One is a highly controlled scenario where all(most all) the externalities can be controlled. The other some extremely complex modeling, which will, at least in my view, always depend on some consensus among the participants which factors are more important than others.

You do know that precisely modeling just a few seconds of turbulent flow would take more computer time than all the climate simulations in history, right? They're both approximations of intractably complex situations.

It will never be possible to credibly predict financial crises: if you learn that in a month your stocks are going to drop 50% you will try to sell them now. But, since other people have heard the same prediction, they will do the same, and the drop would happen today, rather than in a month. Such a prediction can only be possible if it's made by someone people do not believe. Which is, accidentally, completely in agreement with very classic economic theory.

That being said, you are right in that climate models are woefully inadequate.

Predicting the precise moment of the "crisis", the pop, is semi-impossible due to markets being anti-inductive, yes.

Spotting the structural, macroeconomic factors that will create a financial crisis is actually pretty easy.

The mathematics of economies falls in the field of evolutionary dynamics. This is a relatively young field, having its origins in the 1960-1970 time frame, and much of the fundamental principles are still being worked out. For one thing, evolutionary dynamics necessarily involves quite a bit of graph theory, much of which is difficult or impossible to describe with continuous functions.

The mathematics of climate, on the other hand, falls in the field of statistical mechanics. While still a rather complicated field, statistical dynamics is much older, starting with Boltzmann in the late 19th century, and considerably more of the principles have been worked out. Statistical dynamics also deals more with continuous functions which, while not exactly easy to work with, can still be manipulated more easily than graphs.

Are you seriously arguing the analogy?

As long as they are producing testable hypotheses and gathering new data to (in)validate their predictions, they are doing science. And if their theories make accurate predictions, it literally does not matter if those theories came from computer modelling, mathematical proofs, inspiration or divination.

If we could make accurate predictions about the function of the universe by casting chicken bones, that would be science. Even though we wouldn't have the first clue what mechanisms might make it so.

There's no shortage of such situations in science, where theories produce useful results despite no understanding of the underlying, clearly complex systems that drive them. e.g. The Placebo Effect, Gravity, etc.

You seem to be arguing that every time science approaches a complex situation, which it doesn't understand and can't explain from first principles, that we cannot apply the Scientific Method. And that is patently false. That is exactly how we have come as far as we have. By creating falsifiable hypotheses, testing their predictions and refining those hypotheses.

Your hand-waving doubts and fears about complexity and methods is fundamentally anti-science. You are not only a 'climate skeptic', you are apparently skeptical of man's ability to use science to investigate and understand the universe at all.

I think computer models just mean you do replication the other way around. Of course anyone who runs the same program gets the same results, so ordinary replication is pointless--instead you answer the same question with a totally different program and see if your answer is close.

And then, of course, you wait and see if the predictions of the models come true. You can't reset the world and re-test it, but you can re-run the models and ask for more prediction in the future and wait some more.

Climate scientists do both of those things all the time because they're in one of the most heavily scrutinized fields.

Isolating variables just means you compare two setups with everything the same except one. That's actually one of the things the models are for. You can't re-run the world without humans, but you can re-run the model with humans turned off.

Then someone else can do the same with their totally different model. And if both of your answers match reality with humans on and each other with humans off, well maybe the difference between humans on and humans off is the impact of humans. Or maybe not. But adding more different models helps.

In short: climate science generates testable hypotheses, does replication, and isolates variables. It's possible they're wrong (and publishing source code is a good idea) but they don't have a methodology problem. And they're probably right.

Also, Michael Crichton basically wrote Hollywood scripts in novel form. He's not a good source on anything.

> Of course anyone who runs the same program gets the same results, so ordinary replication is pointless

That would be true, if anyone actually distributed their actual code. Pick a journal article at random in any field that describes results from a computational model, and 99% of the paper will describe the results and not the model. The paper will never contain the complete code (which is fair enough, since it would be too long); 1 paper out of 100 will have excerpts of the code, and another 10/100 will have a URL that claims to have the code. If you actually follow that link, you'll find that 2-3 times out of 10 the code won't actually compile or run, and 9 times out of 10, the figures in the paper were generated by tweeking some parameters not defined in the paper whose particular values the author never recorded, and so even the author couldn't reproduce what he actually published, even if he wanted to.

Some people are trying to make institutional change here, see http://www-stat.stanford.edu/~wavelab/Wavelab_850/wavelab.pd...

But really it's a pretty sad state of affairs.

> He's not a good source on anything.

Perhaps you mean this in the sense that we should trust in science and not in authority? Because as hack writers go, his academic qualifications are better than most. He's an intelligent guy, and it seems fair to give at least some weight to his opinion. Equally, it seems unfair to presume a priori that he's not a good source of information. At the least, you'd should argue "He's looney about X, which we all agree is false, therefore we should not trust him on Y". Simply saying "He's looney about X" doesn't add much information and merely pits your authority against his.

I agree with your summary of computer models, and your basic judgement of climate scientists, but fear that many of the influential climate science papers don't adhere to this standard. Frequently the technique is to tweak the parameters of a number of models until each creates "realistic" results, generate a small number (1-3) of simulations with each model, and then create an unweighted average of this ensemble so that the high and low estimates cancel. The meaning of this is much harder to interpret than the case where all the models predict a similar outcome with identical inputs.

http://www.ipcc.ch/ipccreports/tar/wg1/347.htm#tab91

----

CRICHTON, (John) Michael. American. Born in Chicago, Illinois, October 23, 1942. Died in Los Angeles, November 4, 2008. Educated at Harvard University, Cambridge, Massachusetts, A.B. (summa cum laude) 1964 (Phi Beta Kappa). Henry Russell Shaw Travelling Fellow, 1964-65. Visiting Lecturer in Anthropology at Cambridge University, England, 1965. Graduated Harvard Medical School, M.D. 1969; post-doctoral fellow at the Salk Institute for Biological Sciences, La Jolla, California 1969-1970. Visiting Writer, Massachusetts Institute of Technology, 1988.

http://www.michaelcrichton.net/aboutmichaelcrichton-biograph...

Crichton's scientific standards can't have been THAT high, considering how he got hoodwincked into believing Jack Houck's whole spoon bending spiel. I've attended one of Houck's parties and the whole thing was quite remarkable, not for the spoon bending but for the gullability of 90% of the people there.

That certainly could be a legitimate case of "He's looney about X therefore we shouldn't trust him about Y", but I'm not familiar with either Houck or Crichton's beliefs. The quote I can find from him on it seems within reason: 'I think that spoon bending is not "psychic" or bugga-bugga. It's something pretty normal, but we don't understand it. So we deny its existence.'

On the other hand, he also says "More than seeing adults bend spoons (they might be using brute force to do it, although if you believe that I suggest you try, with your bare hands, to bend a decent-weight spoon from the tip of the bowl back to the handle. I think you'd need a vise.)"

http://www.crichton-official.com/qa-travels.html

I just grabbed a spoon and tried it. As expected, contra Crichton, I had no trouble bending the handle to touch the bowl -- no vice required. And no trouble twisting it 360 after bending. But it would be a little surprising that one could exert that much force without noticing. And there was some interesting annealing and tempering going on: it was much harder to untwist than to twist, easier to unbend than to bend, and subsequent bends preferred new locations to repeat bending. So the scale tips a little toward looney, but I'd have to read more before discounting him. And I'm willing to believe there might be some metallurgical property worth exploring here, although I'm pretty sure it has nothing to do with telekinetics or psychics.

But you've actually been to a spoon bending party, and I haven't. Do you have a loonier link?

Just google Jack Houck, you'll find plenty.

Funny anecdote - I honestly tried to use his method of 'feeling the metal get soft and then quickly use this moment to bend the spoon'. I didn't feel anything, so after seeing everyone around me get into some kind of ecstasy, I decided to actually bend the spoon to get an idea of how hard it was. It wasn't hard at all! (Just get some low quality, cheap spoons and forks, they're very easy to bend.) Now, when Jack Houck came around, I showed him my spoon with a sad face and told him it hadn't worked for me, and I had just 'used my muscles' to bend it. He took a few moments to examine it, then proclaimed that he could see in the metal that it had actually melted, that there were features inconsistent with 'cold bending' and that I had very great mindpower but just didn't realize it.

The crowd at this party was very much into new age stuff, crystal healing and all that. In fact, Jack Houck was doing a seminar the next day to teach people healing powers using the same 'energy' that was used to bend spoons, which he had come to consider as a party trick of little interest compared to the healing powers.

There's an interesting issue in climate science, and I say this as someone who thinks that dealing with climate change is probably the most important task of our generation. The (small subset of) professional statisticians I've spoken to are both similarly suspicious of climate models (not climate change itself), though for more technical reasons than you mention.

At the same time, the physics dictate that the general sign and order of magnitude of CO2 forcing in climate models is correct, as surely as doubling solar output would increase mean surface temperatures. The physics also don't preclude the effects being even more severe than the models predict. But there's considerable uncertainty involved in both directions, and when people hear "well, things might end up better than scientists think is the average case, and at no cost to us!" they jump for that option and ignore the average prediction, let alone the worst case scenario that's within plausibility.

Coupled with an extremely well-funded group of fossil fuel industrialists putting hundreds of millions of dollars into the outright shutting down of climate research, good scientists and especially not-at-all-scientific activists end up on the defensive, overemphasizing the finality of the models and using them as solid predictions instead of tools to vindicate the general thrust of the physics.

As far as your particular point, think of the the models in climate journals as tools to understand the issue instead of the final word. It is somewhere between very difficult and impossible to come up with a climate model where CO2 forcing doesn't cause significant warming, but individual parts of those models need to be and often are tested. Indeed, those are the main points of dispute in the legitimate research and end up being thoroughly vetted.

There are a billion dollars worth of research money for group-think infested alarmist journals for every million dollars available from the fossil fuels industry. Your life as a climate scientist is MUCH easier if you shut up and find a hockey stick than if you take Exxon funding.

You have got to be fucking kidding. Scientists have billions, and Exxon has millions for this fight?

Hell of an extraordinary claim. Got any proof?

http://www.climatesciencewatch.org/2010/02/02/president-obam...

http://tomnelson.blogspot.com/2012/02/with-tiny-budgets-like...

You're welcome. Actually, the funding for the actual climate skeptics who are actually fighting this goliath is so close to zero as to be indistinguishable. The significant players all basically amateur bloggers. I think Anthony Watts got $40K from Heartland, but AFAIK it was for a contract not officially related to his blog at all.

The good news is - there's really nothing wrong with being wrong by five orders of magnitude. So long as you keep an open mind and are willing to learn from it.

I'll try to interpret this as charitably as possible =) I think there are two separate issues you're bringing up here.

The first is the implication that climatologists are all involved in a conspiracy with power hungry politicians to institute a kind of global eco-Stalinism, in exchange for research grants. It's the only imaginable scheme where you can treat all government-funded ecology, meteorology, clean-energy tech, and climatology research as part of a coherent but corrupt bargain (As you must to get your billions of dollars figure. I would also add that your mere seven figure fossil fuel budget is grossly underestimated.). Frankly, I don't think you actually believe it, as it's implausible rhetoric that could come straight from the fevered fantasies of Fox News and Rush Limbaugh.

The other is that there's group think among scientists. The idea seems to be that academics and scientists are all too often willing to get caught up in petty vendettas, battles for turf and recognition, and back scratching, instead of focusing on the angelically pure pursuit of knowledge. The issue with that is... well, there isn't an issue. It's totally true, as anyone who's spent much time in research knows. Hence the CRU emails.

It's a fair criticism. But science has soldiered on despite it through the centuries, and scientific institutions, even being plagued with those flaws, have consistently produced better explanations of the world than hacks-for-hire employed by Big Tobacco, Lysenkoist Communists, or the fossil fuel industry.

There's only one small problem with your reasoning: the Lysenkoists called their work "science" as well. As do of course the other two.

Unless you adopt a tautological definition, in which "science" does not include pseudoscience, "science" is whatever the people in your society who practice and organize it choose to call "science."

More specifically, since basically all "science" is government-funded, you'll find that your actual working definition of "science" is "whatever my government funds and calls science."

So your statement boils down to: climatology can't be pseudoscience, because it's funded by the US Government. And Washington (unlike Moscow) would never fund pseudoscience, and call it "science."

This is a pretty interesting epistemology to say the least. Do I have it right? If not, where's the error? If so, what information do you, as (no doubt) a rationalist, have about the US Government that justifies this extension of trust?

And if USG is not the institution you're trusting, what is? What set of human beings are you investing your trust in? If the field of climate science as presently practiced was not in fact scientific, but rather pseudoscientific, who would you expect to have stepped in and shut it down?

[Edit: see also the links to the actual funding levels a couple of posts down. If you're interested in reconsidering your position on this issue, the blog to read is Steve McIntyre's.]

Go read James Hansen's 1981 predictions, and its match with current reality, and weep. Climate change is real, and is probably going to be disastrously significant in the lives of many now alive.

http://www.theatlantic.com/technology/archive/2012/04/now-th...

I think we see this graph very differently. I see a linear rise in measured temperature, and a prediction that shows an inflection point at the year 2000. If I were trying to match the actual to the prediction, without knowing the labels, I'd probably match it to 2c. This turns out to be the hypothetical coal phaseout in 2000.

http://www.realclimate.org/index.php/archives/2012/04/evalua...

After looking for a bit, my questioning side kicks in:

How did they choose the zero point for the overlay? Since it was published 1981, shouldn't the measured match up with the prediction until then? If not, why not?

For that matter, what exactly is the overlay? Is the pink line a time smoothed average of the red or a pencil? Would it create a different impression if extended to 2012?

In what way is a prediction that is 30% off a good prediction? Has it been long enough that we should have seen the change in slope? If anything it seems like it levels off.

I don't think that I ask these questions because I'm a "climate skeptic", but because I'm "generally skeptical". If you tell me a graph tells me something, my first instinct is to doubt you, and then see if the evidence supports your position.

This one feels more fuzzy to me than terrifying, and there are lots of things about the direction of the world that terrify me. What about it makes you weep more than the rest of the news?

I agree with you about the rest of the news. This (and the whole climate denial[not skepticism]) is just one more piece of ongoing tragedy, which is going to get quite a bit worse before it gets better (if it does).

Climate science is much closer to the "hard sciences" than psychology, and thus much less prone to shenanigans.

As someone who accepts global warming (and is uncertain how much alarmism is warranted), I completely agree with you. The best way to debunk (or validate) the climate skeptics is to attack the theory as vigorously as possible, and to see whether is holds up or not.

Of course, I still think in the meantime we should be building nuclear plants, investing in solar tech, etc, but there are many reasons to wean ourselves off of oil and coal rather than just climate change.

You are very right. Things like rising cancer rates, alarming increases in extinction, massive destruction of natural habitats, destabilization of entire ecosystems, etc. Some may attack anthropogenic climate change despite strong evidence, rightly or ignorantly, but it's an absurd (and tragically common) the degree to which actual environmental damage that is happening right now is ignored. Some of it is likely due to global warming, but there are a lot of other externalities that are really dangerous for our planet and even our species.

You are soooo right. One of the worst fields, in my opinion, is nutritional science. Gary Taubes has written two books, "Good Calories, Bad Calories" and "Why We Get Fat" which eviscerate the core studies that are foundational to modern nutritional science.

You can not trust any research done where the career of the researcher depends on them finding results.

As an aspiring researcher, I quite agree. I know many researchers who have similar feelings.

But of course, this means getting off the capitalistic grant-publication-grant model of research.

Not all fields rely on grants -- but in all fields the careers depend on results. Which is not a bad thing in itself; but it would have been nice if "results" were not quite as narrowly defined.

I actually jotted down a note to myself wondering: is there anyone who would pay to have studies replicated?

  I strongly believe that for hiring purposes, the research 
  skills of profs should be evaluated on criterion that are 
  incidental to research. Mostly math, probability, 
  statistics (yes even for psychology) then methodology 
  skills and also maybe leadership, communication skills 
  and dedication to science (last only because it is 
  difficult to measure).

I like the sentiment, but the problem is that to do good research requires real initiative and creativity. Testing for the skills you listed would filter for people who are skilled, but it doesn't distinguish at all between those who can do new research versus those who are merely good at following instructions from other people.

What's your evidence for that? Don't you think it's possible that initiative and creativity are important for getting your research noticed by jey on Hacker News, but not for actually doing the research?

This was a big problem in my computer science. It's easy to run the experiment over and over until you get the desired result.

Reply on news.ycombinator.com