Posted on 14 March 2011, 22:13

Journal of the Society for Psychical Research 74: 156-167 (2010)

“Heads I Lose, Tails You Win”, Or, How Richard Wiseman Nullifies Positive Results, and What to Do about It: A Response to Wiseman’s (2010) Critique of Parapsychology by Chris Carter

INTRODUCTION

Psychologist Richard Wiseman is a well-known British critic of parapsychology, frequently appearing in the British media to “debunk” psychic research.  In his recent (January/February 2010) Skeptical Inquirer article “‘Heads I Win, Tails You Lose’: How Parapsychologists Nullify Null Results”, Wiseman argues that parapsychologists have tended to view positive results as supporting the existence of psi, yet have adopted various strategies to ensure that null results do not count as evidence for the non-existence of psi.  However, I am going to demonstrate that throughout Wiseman’s career, he has tended to adopt a “heads I win, tails you lose” approach to parapsychology’s research findings, viewing null results as evidence against the psi hypothesis, while attempting to ensure that positive results do not count as evidence for it.
 
In his article Wiseman levels the following criticisms against parapsycholgists:

• “Cherry Picking New Procedures”.  By this Wiseman means that positive findings in parapsychology have “emerged from a mass of nonsignificant studies.  Nevertheless, they are more likely than nonsignificant studies to be presented at a conference or
published in a journal”.

• “Explain Away Unsuccessful Attempted Replications.” Wiseman argues that parapsychologists come up with various excuses for not accepting failures to replicate positive results as evidence for the nonexistence of psi.

• “Meta-Analyses and Retrospective Data Selection”. Wiseman argues that meta-analysis provides evidence against the existence of psi, but that parapsychologists retrospectively decide only to analyze data that fits with the existence of psi. 

Cherry Picking New Procedures

In this section Wiseman wrote:

Parapsychologists frequently create and test new experimental procedures in an attempt to produce laboratory evidence for psi. Most of these studies do not yield significant results.  However …they are either never published … or are quietly forgotten even if they make it into a journal or conference proceedings. (Wiseman, p. 37)

But how does Wiseman know that “most of these studies do not yield significant results”?  He provided not a shred of evidence for these claims, yet continued:

Once in a while one of these studies produces significant results. …the evidential status of these positive findings is problematic to judge because they have emerged from a mass of non-significant studies.  Nevertheless they are more likely than
non-significant studies to be presented at a conference or published in a journal. (Wiseman, p. 37)

Again, Wiseman offered no supporting evidence for these claims. When he remarked that “The evidential status of these positive findings is problematic to judge because they have emerged from a mass of non-significant studies”, he refers to what is known as the “file-drawer” problem: the likelihood that successful studies are more likely to be published than unsuccessful studies, which are more likely to end up discarded in someone’s file drawer.

It has long been believed that in all fields there may be a bias in favor of reporting and publishing studies with positive outcomes. Given the controversial nature of their subject, parapsychologists were among the first to become sensitive to this problem, and in 1975 the Parapsychological Association adopted a policy opposing the withholding of non-significant data, a policy unique among the sciences. In addition, the skeptical British psychologist Susan Blackmore (1980) conducted a survey of parapsychologists to see if there was a bias in favor of reporting successful ganzfeld results, and concluded that there was none. 

Still, since it is impossible in principle to know how many unreported studies may be sitting in file drawers, meta-analysis provides a technique to calculate just how many unreported, non-significant ganzfeld studies would be needed to reduce the reported outcomes to chance levels.  In a ganzfeld debate between skeptic Ray Hyman and parapsychologist Charles Honorton, Hyman had raised the possibility that the positive results were due to selective reporting.  However, once Honorton calculated that the results could only be explained away by a ratio of unreported-to-reported studies of approximately fifteen to one, it is not surprising that Hyman concurred with Honorton that selective reporting could not explain the significance of the results. (Hyman and Honorton, 1986, page 352)
 
In the following case Wiseman seems to engage in a “cherry picking procedure” of his own:

The Natasha Demkina Case

In September 2004 Wiseman took part in a classic debunking exercise, claiming that a young Russian girl who had seemingly psychic powers of medical diagnosis had failed a test he and his fellow skeptics designed.  In fact, the girl scored at a level well above chance.

Natasha Demkina, then 17 years old, claimed that she could look deep inside peoples’ bodies, examine their organs, and spot when something was wrong.  As part of a test broadcast on television by the Discovery Channel, Natasha was given a set of seven cards, with a medical condition indicated on each.  Medical subjects with these seven conditions (one of which was “no condition”), each bearing an identifying number, stood in a row and Natasha had to mark each card with the number of the person whom she thought had the condition indicated on the card.  Under the tightly-controlled conditions imposed by the experimenters, she identified four of the seven correctly.  The odds of getting 4 hits or more out of 7 by chance are more than 50 to 1 against.  Another way of expressing this would be to say the probability that the null hypothesis is correct – that is, that Natasha displayed no genuine ability but merely got lucky – is less than 2 percent. However, Wiseman declared the test a “failure.”  He was only able to do this because the experimental protocol, to which Natasha and her agent had been asked to agree, curiously states:

“If Natasha correctly matches fewer than 5 target medical conditions, then the Test Proctor will declare that results are more consistent with chance guessing and does not support any belief in her claimed abilities.”
 
Accordingly, it was announced that Natasha had “failed the test”.  Brian Josephson, a Nobel Laureate in physics, investigated Wiseman’s claims about this test and found them to be seriously misleading. http://www.tcm.phy.cam.ac.uk/~bdj10/propaganda/

Keith Rennolls, Professor of Applied Statistics, University of Greenwich wrote a letter that appears in the 2004 December 17th issue of the Times Higher Education Supplement.  In part it reads: I have reviewed Professor Josephson’s arguments, published on his web page, and findthem to be scientifically and statistically correct.  In contrast, the statement of Professor Wiseman, of CSICOP, ‘I don’t see how you could argue there’s anything wrong with having to get five out of seven when she agrees with the target in advance’, demonstrates a complete lack of understanding of how experimental data should be interpreted statistically.

 
The experiment is woefully inadequate in many ways. The chance of the observed 4 successes 7 subjects by pure guessing is 1 in 78, an indication of a significantly non-random result, as claimed by Professor Josephson. The experiment, as designed, had high chances of failing to detect important effects. 

Here we have a case in which Wiseman nullified a positive result by arbitrarily ignoring the fact that the girl scored at a level commonly accepted in scientific experiments as strongly and significantly above what chance alone would predict.  In other words, Wiseman “cherry picked” an experimental design that had high chances of failing to detect important effects. 
 
Explain Away Unsuccessful Attempted Replications

Regarding follow-up studies of successful psi experiments, Wiseman complained:

However, any failure to replicate [the original effect] can be attributed to the procedural modifications rather than to the nonexistence of psi. Perhaps the most far-reaching version of this “get out of a null effect free” card involves an appeal to the
“experimenter effect,” wherein any negative findings are attributed to the psi inhibitory nature of the researchers running the study. (Wiseman, p. 37)
 
It is somewhat hypocritical for him to dismiss appeals to experimenter effects, especially remarks about psi-inhibitory effects, since one of the best documented studies demonstrating this effect involves himself. (Wiseman & Schlitz, 1997)  Wiseman and Marilyn Schlitz ran identical studies in the same location using the same equipment, in order to see if participants could detect whether or not the experimenter was staring at them.  Wiseman’s results were not significantly different from chance, while experiments involving Schlitz produced results significantly higher than chance would predict. 
 
In the following case Wiseman attempted to explain away a potentially embarrassing successful replication.

A Dog that Knew when its Owner was Coming Home.
One of Wiseman’s most highly publicized experiments concerned a dog named Jaytee.  His owner, Pamela Smart, claimed that the dog could anticipate her arrival home.  Pam’s parents noticed that Jaytee seemed to anticipate Pam’s return, even when Pam returned at completely unpredictable times.  It seemed as though Jaytee would begin waiting by the window at about the time she set off on her homeward journey (the following was first described in Sheldrake, 1999a, 1999b and Sheldrake 2000).

In April 1994 Pam read an article in the Sunday Telegraph about research into animals that seem to know when their owners were coming home, being undertaken by biologist Rupert Sheldrake.  She contacted Sheldrake and volunteered to take part in his research.
 
After receiving a grant from the Lifebridge Foundation of New York, Sheldrake began videotaped experiments with Jaytee in May 1995.  Between May 1995 and July 1996, thirty videotapes were made of Jaytee’s behavior under natural conditions while Pam was out and about.  Pam’s parents were not told when she would be returning, and Pam usually was not sure herelf.  The results showed that Jaytee waited at the window far more when Pam was on her way home than when she was not, and this difference was highly statistically significant (p

<.000001).

The researchers discovered early that Jaytee responded even when Pam set off at randomly selected times. This was an important discovery, as it seemed to clearly rule out an explanation based upon routine, or expectations based upon the behavior of her parents. Consequently, twelve more experiments were videotaped in which Pam returned home at random times, determined by the throw of dice after she had left her home.

See complete article here.

www.sheldrake.org/D&C/controversies/Carter_Wiseman.pdf

www.spr.ac.uk

www.subversivethinking.blogspot.com