I would offer a pseudo-Bayesian[1] answer to that question. Parapsychology aims to prove hypotheses that lacks theoretical foundations. Our current understanding of physics and biology weigh strongly against the existence of psychic phenomena. Even before any experiment is conducted, we must admit that psychic phenomena are unlikely to exist. Our experimental results must be evaluated in light of that prior probability.
Thus, parapsychology is and should be held to a higher burden of proof than other branches of science. We should demand more rigorous experimental designs, stronger effects, and smaller p values. This XKCD presents a similar idea, if you substitute "psychic phenomena exist" for "the sun has gone nova":
http://www.explainxkcd.com/wiki/index.php?title=1132:_Freque...
[1] I say "pseudo" because I'm not a statistician by trade. I'm basing my argument on my rather superficial understanding of Bayesian statistics. I still think it's a valid argument in its own right, but I don't claim that it's an accurate representation of Bayesian statistics.
This is the lynch pin for most scientists. Before you can have a hypothesis, you must have a theory, and you can work to prove or disprove that theory by experimenting to create or observe results that the theory predicts. Parapsychology doesn't have good, testable, theories. Rather it has some interesting unexplained correlations.
Do you discount all evidence that doesn't fit into your world view? Maybe theory hasn't caught up with evidence yet.
For the topic at hand, ESP would most likely invalidate quite a bit of physics no one is questioning for other reasons and there is no proposed theory to explain the effects. Our confidence in the studies is rightfully close to zero.
The germ theory of disease and experiments supporting the theory predate medical sanitation. (Pasteur's work did come after, but he wasn't the first.) The medical community's initial rejection of handwashing was not due to a scientifically motivated demand for a sound theory. Rather, it was due in large part to doctors' unwillingness to believe that they were the ones spreading disease from patient to patient. Additionally, the medical community at the time did not embrace the scientific method to the extent that it does today. Had it, handwashing would have been evaluated in a controlled study and proven effective.
> Despite various publications of results where hand-washing reduced mortality to below 1%, Semmelweis's observations conflicted with the established scientific and medical opinions of the time and his ideas were rejected by the medical community. Some doctors were offended at the suggestion that they should wash their hands and Semmelweis could offer no acceptable scientific explanation for his findings.
He published a lot on the subject and was rejected because it didn't fit in with how doctors saw the world and he had no theory to back up his findings. It wasn't until Pasteur that germ theory gained any widespread acceptance so it's pretty irrelevant that other people thought of it first.
Plenty of things are evaluated in controlled studies and still rejected today. For example, the article that you presumably just read discusses one such thing.
Science is not based on a single experiment it's based around replication of experiments by different people at different times using slightly different methods. Parapsychology repeatedly fails this test to the point where there is a million dollar prize for anyone that can demonstrate anything in a controlled setting.
Science is the testing of a hypothesis by applying an instrumental injunction to reality, apprehending and analyzing the results, and sharing/verifying the results in with a community of experts who have also performed the same injunction (reproducibility).
Randi's process shares none of these characteristics with the scientific process. It is entertainment, a publicity stunt that has little to do with science other than to muddle the waters when discussing these topics.
However, the important point in the article is that in order to make this inference in an intellectually honest way, you need to significantly increase your estimation for the probabilities of widespread experimental error in ALL scientific studies that use similar methods. Since the methods in parapsychology are pretty good, this has quite a far reaching effect.
In a sense, your question about "on what grounds can people claim one field to be nonsense but not others?" is exactly the same question asked by the author. Except the author isn't implying that parapsychology isn't nonsense, they are implying that many other fields are nonsense too. This makes sense because physics is almost certainly not nonsense.
tl;dr: Yes, that is intellectually dishonest, which is a huge problem. Scientists have two options: (1) accept parapsychology as real, or (2) accept that the "scientific method" (in social "sciences", at least) is insufficient. The problem seems to be that Bem (the author of the study) did almost everything "right", and if we increase the bar of scientific proof so that his study is excluded, so are many others...
The reason why parapsychology is excluded: the real world doesn't support it (noone is earning huge amounts of money on the stock market using psi). However, the real world often doesn't support very small effects (e.g. Einstein's relativity). Fortunately for physics, they can afford to make their scientific methods much more rigorous, so they can study large effects as well as exceptionally tiny effects. Social studies can't, for the time being.
> Yes, that is intellectually dishonest, which is a huge problem. Scientists have two options: (1) accept parapsychology as real, or (2) accept that the "scientific method" (in social "sciences", at least) is insufficient.
I don't get why the whole thing is such a huge problem. The entire problem rests on needing parapsychology effects to not be real. If that need did not exist, we could just go "Okay, interesting, seems likely that there's something to it then. Let's do more research!" because, you know, we take that approach everywhere else. So my question remains: what is it about parapsychology that makes option two even valid to consider? All I can see is people just not liking that that may be how things work.
(But these experiments are as good as many in psychology / social science - suggesting that many "proven" results in psychology / social science could be false)
Conflicting results don't mean one set is impossible (cf., the long-standing apparent conflict between QM and relativity within physics). Apparently conflicting results without a methodological error in either imply that the explanatory model that appears to be supported by at least one of the results (if not both) is, while useful within its own domain, in some way incorrect.
The whole idea that the models validated by scientific experimentation are binarily true of false is, well, missing the point badly. While over time we hope they approach truth with them, what they are is useful (that is, they have predictive power) to a greater or lesser extent. And quite often the models with the greatest predictive power in two different domains conflict when either or both are extend outside of their own domain.
EDIT: The real problem with parapsychology is that there's little in the way of explanatory models being tested anywhere in the field. There's a lot of hypotheses without models and some experiments testing them, which (concerns about methodology aside, for the moment) might raise interesting questions and serve as inspiration for developing and then testing theoretical models to explain the effects, but very little has been done there -- which makes "parapsychology" more a collection of potentially unexplained phenomena more than a branch of science that provides an explanatory model for some set of phenomena.
Which is very different from most of the social sciences.
When scientist thought they found particles travellig faster than light speed they checked the results, then the equipment, and then they assumed they had made a mistake and asked other people to check the numbers and the experiment. They realised that they had an extraordinary result and they wanted very high degree of rigour.
Some parapsychologists appear to rush to publish weak results and to claim success for flawed experiments.
And as it turns out, it was a mistake after all. Physics is solid to a satisfying number of digits after the comma.
So do some sciences in the supposedly "better" fields [1].
I think the fact of human nature at issue here isn't specific to any particular field of study.
[1] For a particularly well-known example, consider http://en.wikipedia.org/wiki/Cold_fusion#Fleischmann.E2.80.9...
Similarly, with the faster than light neutrino, we spent far more effort looking for mistakes that would make our answer bigger than it should have been, which introduces the same systematic bias.
The solution to this is to realize that science is a time consuming process, and it is okay to take a while to arrive at the right answer. But, if we are aware of these problems, we can get there faster.