A database that list all pundits' specific prediction and give them a rating based on if they succeeded or not?
You would have to give each prediction an "audacity factor", how unlikely it seems.
The system would work in fashion where a 50% accuracy rate and a 30% one would actually be considered the same accuracy. Either you get higher than a 50-60% rate OR you get some of your predictions rated as having a high audacity factor.
The basic idea of a prediction market is like a stock market for outcomes with an end point. If you make good predictions you will make 'money' and if your predictions are correct but go against the common idea your rewarded even more handsomely. Of course a true prediction market has a shifting price point where you can choose to sell your 'shares' at any point, yada yada, but I think the concept applies.
The trick is you have to have the pundits participate in order to make it work. I suppose there could be some third-party that pulls their articles and 'ghost trades' for them?
However I think the truth is that pundits DON'T WANT to be held accountable. Even if they are shown to be 'the most correct' they are still going to be wrong some of the time and those will always be used as examples to attack them and their views.
Just look at another comment in this thread that tears into this pundit for his poor job predicting the Sony vs MSFT vs Nintendo war over this console generation. This is a SINGLE data point being used to try and dismiss his views.
edit: For anyone interested in prediction markets check out Inkling (http://inklingmarkets.com/). I don't work for them, but I just think they have done an awesome job of creating some useful software. I am trying to push for this to be adopted internally for creating 'confidence metrics' of how stable a build is going to be, or if a deadline is going to be hit.
It's not really realistic to comprehensively log all of this, of course. But it sure would help me assessing the relative quality of some sources, especially now that sources for news are so varied and fragmented. I suppose in the past the reputation of a publication served some of this purpose (i.e. WSJ or NYTimes having stellar reputations).
I knew of a couple of databases for equity analyst performance like Starmine and Bulldog Research. So those do exist. Not sure if either are still around since I'm no longer in the finance space.
from Google cache: Site: http://bit.ly/9JiRrz & FAQ: http://bit.ly/9cQeTb