2. Flip a coin. Sort out all participants that were wrong.
3. Repeat step 2 until there are 10 people left. These 10 people now have a track record of predicting the future. Some of them even will continue to predict the future for a number of times. Eventually, all will be wrong.
The thing is, you don't know who will be the expert forecaster in the first place and the same is true for the Good Judgement Project.
> The big surprise has been the support for the unabashedly elitist “super-forecaster” hypothesis. The top 2% of forecasters in Year 1 showed that there is more than luck at play. If it were just luck, the “supers” would regress to the mean: yesterday’s champs would be today’s chumps. But they actually got better. When we randomly assigned “supers” into elite teams, they blew the lid off IARPA’s performance goals. They beat the unweighted average (wisdom-of-overall-crowd) by 65%; beat the best algorithms of four competitor institutions by 35-60%; and beat two prediction markets by 20-35%
If you insist on a heads or a tails then your experiment is broken by design and tells you nothing.
They should turn it on itself - make anonymous profiles of the participants, and see if they can predict who will make a good forecaster!
Here's an entertaining video from Derren Brown that shows how to use this technique to convince people you have psychic powers: https://www.youtube.com/watch?v=9R5OWh7luL4
Edit: also see the comment about the Miracle Man scam in this thread.
Your coin example is a game of pure chance while trading on current events have much more context to inform predictions. It's really quite a difference.
The only difference is that, in a coin toss, the statistics are much clearer. In real event forecasting, the perversities of language can make a failed forecast seem successful. Here's an example:
A seer in the Greek city of Troy issues a warning: "Beware of Greeks bearing gifts." The citizens don't know what to make of this saying, so they proceed as before.
After the now-famous Trojan Horse incident, the seer repairs her reputation by revising her earlier warning -- "I actually meant to say, 'Beware of gifts bearing Greeks'."
The bottom line is that, whether flipping coins or evaluating real-world predictions, one can separate successes from failures with equal efficiency, and an equal lack of meaning.
http://en.wikipedia.org/wiki/P-value
Quote: "In statistical significance testing, the p-value is the probability of obtaining a test statistic result at least as extreme or as close to the one that was actually observed, assuming that the null hypothesis is true."
Chance occurrences can be made to look very convincing. Consider the "Miracle Man" scheme used in investing -- on first hearing about it, many people assume it's not possible:
http://www.arachnoid.com/equities_myths/index.html#Miracle_M...