538 indeed kicked ass. They also got all of the Senate races right, except for the Steven race in Alaska (way off, apparently, but who'd have expected a felon to win?) and possibly Franken/Coleman in Minnesota, where they gave Franken a 52% chance.
A better way would be to check their results against other 'odds producing' predictive mechanisms. So if a another method has Stevens at 20% and 538 has Stevens at 30% then that actually counts as a win for 538 because they were 'less wrong'. The bigger the difference in their odds the bigger the win. In the real world this would translate to 538 winning money from the other method.
I'm sure there are better and more valid statistical techniques for doing this though. I'm hoping that someone gets around to it because I'd be keen to see the results.
"Indeed, it seems possible that the number of "questionable" ballots could be quite high. So far, about 220 thousand votes have been processed in Alaska. This compares with 313 thousand votes cast in 2004. After adding back in the roughly 50,000 absentee and early ballots that Roll Call accounts for, that would get us to 270 thousand ballots, or about a 14 percent drop from 2004. It seems unlikely that turnout would drop by 14 percent in Alaska given the presence of both a high-profile senate race and Sarah Palin at the top of the ticket."
http://www.fivethirtyeight.com/2008/11/what-in-hell-happened...
Read on for more of his WTF explanation. We should all stay tuned for this one.
Perhaps seeing the coverage caused some voters to change their behaviour (ie. not vote)?
(Nate muses on the same possible explanation in his post).
I would be curious to see if it outperformed a naive averaging of the major polls.
They did seem to do better than Pollster which is a naive averaging.
The site compiles polling data through a unique methodology derived from Silver's experience in baseball sabermetrics to "balance out the polls with comparative demographic data"[1] and "weighting each poll based on the pollster's historical track record, sample size, and recentness of the poll."