Meh. Start a site, like longbets, where you retroactively compare pundits' predictions with what actually happened.
I should start adding Mike Arrington quotes, he's almost as bad as Scoble.
Of course, ideas are cheap...
Only then can you sort out intelligent predictions from noise.
And wouldn't it make sense to quantify the vagueness/concreteness of statements made? It's easy for a pundit to say something is "dangerous for America" without giving a specific consequence, and people who make specific predictions that are borne out deserve more credit than people who make vague claims that can't be either affirmed or falsified.
And from there, why not keep track of how many times a commentator uses equivocating devices like "borders on", "all but certain", "just short of"?
And then, what about commentators who keep predicting a specific thing will happen but ascribing different causes to it?
You can also measure how "substantive" a commentator tends to be by looking at the frequency with which concrete facts or examples are cited to support their claim, vs. how many times vague subjective judgments are invoked, vs. how many times nothing is brought forward to support the claim at all.
You can measure how frequently a commentator tends to repeat the same news event or same analysis.
You can take transcripts of their shows and put them into a program that determines what reading level, (in terms of school grades like 7th grade, 9th grade...) their discussion was judged to be at.
And there are probably tons of others.
Granted, systems like this, if they caught on, could turn around and bite us like those stories about wish-granting genies with a sense of irony- surely this would result in a whole host of unanticipated structural biases generated by such stats that change the character and quality of the discussion; surely pundits would find, even discover, new ways to misinform and adapt to the structure accordingly.
But that has already happened. Pundits already speak in a language designed to make as many claims as possible without being on the hook in any way that risks their credibility. So while it would create new biases, I think it would overturn a whole bunch of pre-existing biases that I think are much worse (or at least shed light on them).
This is a long-winded way of saying I really want something like this to happen, and to work.
Strictly speaking, there is no prediction in the article. This is a claim about the present. Granted, he has already had to amend the original claim ("late stage" vs. "early stage" talks), so take him to task over that rather than imply a prediction is being made.
http://punditwatch.hubdub.com/
TechCrunch's score is a telling 13%.
The right measure is to take the difference between market and pundit opinion (you know like a real market, except unlike companies these always end at 1/0 on a certain date - like a covered option) and average that.
This is their real performance.
There is only so much fudging that can occur, of course. But this kind of on-the-margins behavior is a new challenge that this relatively new breed of sites will have to deal with.
I don't have a point here, this just strikes me as a pretty weird situation.