Why political journalists can’t stand Nate Silver
markcoddington.com
markcoddington.com
Consider that Mr Silver has been given awards and accolades most journalists would kill for: Time's list of 100 most influential people in the world, best political blog Webby award, Rolling Stone's 100 agents of change, the editor in chief of Politico listed him as "one of the most powerful people on earth", he's given a prestigious lecture at Columbia Journalism School, etc. He's received accolades, even from traditional journalism sources, that most journalists, even at the NYTimes, would never dream of. Speaking of which, the Public Editor of the Times wrote "he’s probably (and please know that I use the p-word loosely) its most high-profile writer at this particular moment."
All this for someone who has never worked a proper newsroom since college, never had a television show, cable or otherwise, and just started a blog four years ago because he was (supposedly) annoyed at laws that threatened his livelihood - online poker.
So imagine seeing someone like this receiving all the awards you've coveted since you started journalism in earnest. While the recoil against Mr Silver might be clothed in intellectual differences for some, the reason they care probably has more to do with seeing someone who differs from their traditional approach getting so much recognition.
Finally, I'd note that there are many traditional journalists who support Mr Silver's approach - his awards from traditional journalist organizations speak to this. Rather, it's a very specific group that seems to feel threatened.
Nate was also an incredible asset to the early online poker community. He was one of the most respected members of 2+2 and really helped pushed poker strategy forward. His detailed and analytical approach to the game helped make a lot of people a bunch of money during the poker boom.
The news media were still touting it as a close horse race when Nate knew that it was already over. Obama already had enough delegates that it was almost impossible for him to lose. He saw it as blatant deception on the part of the news media as a way to sell more newspapers. When he pointed this out, he just flat out embarrassed them.
I really miss the original 538 blog. Nate would give snarky commentary on whatever the political news of the day was. Now that he works for the NY Times all he does is crunch the numbers. It's not nearly as entertaining.
Nate could be pretty hilarious in print. I want the old short snarky articles back.
It's popular to mis-use "per se" when you mean "necessarily", as in, "she's not my girlfriend, per se", but this is not correct, either. I expect this is how you meant it -- "he's not really an employee of NY Times, he's more like a contractor".
Here's an appropriate use of "per se":
"An aggressive psychological interrogation, per se, is not torture. It only becomes torture when it causes physical harm or lasting psychological trauma to the prisoner." <-- note, this is just a language example, not a statement of my position :)
Why wouldn't it be parsed as that?
"Ohio Has 50-50 Chance of Deciding Election"
http://fivethirtyeight.blogs.nytimes.com/2012/10/23/oct-22-o...
Perhaps some pundits dislike Mr Silver because he is able to rationalise what they only have a gut feeling for.
I disagree. I much prefer the article's explanation for why they invariably invent a tight race: journalistic 'objectivity'. As Coddington pointed out, when all the highest sources on both sides of the campaign say it's tight, who is Silver to disagree?
I don't think that can always be replaced by just data-crunching, in part because you just move the same problem to the 2nd-order problem of interpretive frames for data, which requires the same in-depth ethnographic field work to get right. The proper balance is a big debate in qualitative vs. quantitative sociology, though.
I do think that when it comes to predicting the outcome of elections, on the other hand, the decision is not hard, because it's almost a best case of a prediction that can be quantified based on available data. So I'd say this is more an issue of punditry vs. careful methodology (of any kind), rather than qualitative vs. quantitative epistemologies. In this specific case, it's mainly just partisanship: some people don't like that Silver's data shows Obama with an ~80% chance of winning, so plug their ears.
If you really want to know which side feels less-than-confident about their chances, the best indicator is "leaks" from campaign insiders that talk about the candidates' future plans. Laying the groundwork early is an irresistible urge for those who think they might be in trouble.
News flash: It's no secret that the news media (except for Fox) has a distinct Democratic tilt. I agree with your premise that journalists are closet partisans, but I disagree that they'd be upset at an analysis that favors Obama for partisan reasons.
Bonus points if their run talk show with a political slant like Colbert.
EDIT: thanks for the downvote, but I'm still waiting for triplets.
Sigh...
talk show: Glenn Beck? Rush Limbaugh? Pat Buchanan is on the communist PBS show McLaughlin Group.
Colbert isn't a journalist, but talk radio is 99% conservative or if not outright conservative then advocating "common sense"-y solutions that somehow always come out conservative.
Didn't check wikipedia or even look up articles.
These charts (from [1] and [2]) are particularly interesting:
http://www.stateofthemedia.org/files/2011/01/31_Cable_Revenu...
http://stateofthemedia.org/files/2011/03/20_Cable_Cable-Chan...
http://www.stateofthemedia.org/files/2011/01/5_Cable_CNN-Lea...
http://www.stateofthemedia.org/files/2011/01/21_Cable_CNN-Re...
http://www.stateofthemedia.org/files/2011/01/22_Cable_The-Sa...
In short, when compared to CNN: despite operating few domestic and foreign bureaus, Fox News pulled in more revenue and had nearly as large an audience (as measured by viewers tuning in for 60 minutes or more monthly). Fox also allocates over 70% of its budget to "program expenses" (including salaries for its hosts), whereas CNN's program expenses are about 44%. The difference in staffing figures are also drastic: ~4000 for CNN, 1272 for Fox.
[1] http://stateofthemedia.org/2011/cable-essay/
[2] http://stateofthemedia.org/2011/cable-essay/data-page-2/
However, that doesn't necessarily mean the actual news stories are biased. There may be tilts in the sense that, say, a Keynesian and a Hayekian might sincerely intend to write the same story objectively but still reveal a bit of bias in the finished result -- but isn't that qualitatively different from the kind of editorializing we see on Fox (and, since they decided that counter-programming Fox was their ticket to success, MSNBC)?
I disagree with this. Basically, the journalistic "epistemology" the article describes is that journalists get access to information the rest of us can't see, so we have to take their word for whatever they say about it. That doesn't seem like a good thing to me. I don't want journalists, or anybody else, to pre-digest my information for me. I want to see the raw data. If they want to show me their spin on the data in a separate article, that's OK, but if I can't see the actual data they base their spin on, I don't trust their conclusions.
I had to edit it because of the eighty character limit. Sorry about that.
The central point of the article is about misunderstandings between people with different 'modes of knowledge', and not really about current USA politics (in which case I wouldn't have posted this here,I'm not a US citizen and so detached from the partisanship.) Hopefully the discussion here won't get into political argument.
And unfortunately, that's true for most of the people, most of the time - if you asked your physician to substantiate any non-trivial recommendation he makes, you'd find that data supporting those recommendations is severely lacking or irrelevant. But most people want the magical insider (the doctor, in this case) to tell them what's happening, rather than the facts.
I remember an Issac Asimov story about a future where only one person had to vote, because, based on that person's vote, a supercomputer could predict who would have won the election. That's the strange future of 538. Politics is no longer the art of the possible, it's just like betting on sports.
[1] Insider campaign coverage is certainly interesting, but that's different from being important. I'm interested in behind-the-scenes narratives in just about everything.
http://www.nybooks.com/articles/archives/1988/oct/27/insider...
Note: don't try to read it from a left/right perspective. That's not where she's coming from.
but just the weather service is not enough
(what I would have said had I more time first time)
For the same reasons, Chris Christie and Michael Blomberg feel safe throwing Romney under the bus.
Good point that they act extremely numerate (is that a word?) in how they respond to polling numbers.
Got any data to back this up?
Nate is a hybrid. The details of his model are not on Github or anything, but he does describe his general approach, and he works from publicly available information.
It was much better when journalism was a trade that you went into. Street savviness was the journo's hallmark. They had no loyalty to power, because they came from the wrong side of the tracks, and their livelihood depended on exposing secrets. Great journalists were ugly, ink-stained, alcoholic wretches.
Those people are gone. Now there's two kinds of professional journalists. Middle class people who were educated to do journalism at university, fighting over the few scraps of remunerative work left. And people who want to shill for one of the dominant political coalitions. These groups get to the gym a lot more often, but are constrained by their loyalty to the system.
So, journalism would be better if it was less respectable. We also need to put a stop to all the social mobility that's happening. When you block an entire group of people from attending university or holding office on some irrational basis like ethnic origin, you usually get some really good journalists. But there is cause for hope. Perhaps the end of privacy means we'll get more people in the media like Eliot Spitzer.
Wang sometimes disagrees with Silver, but supports the notion of a data-driven approach. Fascinating stuff on his blog, too: he's running Bayesian prediction models, Random Drift, and has a popular meta-margin that is worth checking out.
He also takes some delightful shots at journalists and others who are trying to keep the focus on political "horse race" reporting, instead of using more rigor. And he has a great sense of humor - his "Nerds Under Attack" post is hilarious - http://election.princeton.edu/2012/10/29/nerds-under-attack/
They're not against his methodology, they're against his current results, which don't make the race as interesting as it could be. (If Nate was the only one saying it was a close race, they'd be pushing his narrative above all others.)
News is biased, but not how most people think, it's biased towards conflict (and novelty, doomsaying, and sensationalism).
It's just entertainment, there's no permanent epistemology to entertainment.
They could, however, be using those entertainment factors to cover, say, the fall of real wages, or the current state of unions, or what's actually in a Congressional budget, or any number of other topics that seem to get very little air-time.
I did work for a startup that specifically showed un-biased news on a variety of pertinent topics from a variety of great sources.
Yet the videos that made the most money for them (advertising revenue) were the stupid shit you'd expect. Gossip, memes, sexual related content...
At the end of the day the most mainstream news stories/clips were far more lucrative than the topics a typical HNers/New Yorker reader might enjoy.
The typical entertaining news story "writes itself". Gossip, stupid pet tricks, etc.
Indeed the entertainment industry is itself very conservative in its choices of subject matter.
Exactly. They're called "stories" for a reason.
Conflict itself is a news value. Something may be news precisely because there is conflict, and exponentially so if that conflict affects a news organizations' readers via proximity or other news values. Where no conflict can be found, it can be (and often is) generated via balance. Science writing is full of it, same with education writing and politics. Find one source that says one thing, another source that says the opposite and you've generated built-in conflict. If the conflict is sensational enough, your editor puts it on the front page.
This is a real problem in journalism. Journalists are specifically trained as generalists. There's a culture of avoiding becoming experts in their beats, instead they strive for access to experts. The argument is that by not being an expert, one can write for a general audience more clearly. Just follow the formula.
Anyway, news is narrative. That is the chosen format. Compelling narratives do better at selling news than dry factual reporting.
There's a reason why, for example, when newspapers do polls that are outliers from the consensus they blare that information as loudly as possible. It's not because they honestly think they're right, it's that they've spent upwards of $20,000 conducting the survey and now they're going to get their worth out of it.
To put it another way if his model suggested, e.g., Obama has a 100% chance of winning, and everyone knew that, his supporters would be less inclined to vote and he could lose. So there's a chaotic element to these things that's just difficult to predict, and journalists are better placed to handle this chaotic element (another example: Hurricane Sandy). That said, given the abundance of polling data, I think his model stands to be a much better predictor than journalistic intuition.
Fittingly, pundits typically dislike actual journalists for the same reason: they attempt to report things accurately rather than through a party or ideological filter.
This is in part because it takes longer to fact check things, but there is tremendous competitive pressure to be first with news. So the on-scene reporters report what was said in near-real-time, and then the fact check reporters look through and report on how true it was.
I get the rationale, but it doesn't do what reporters need to actually do to be an effective part of the political process.
It's also an insight into the state of journalism and the press in this country. It's pretty hard to find objective reporting these days, sources of 'news' seems to be full of highly opinionated punditry rather than unbiased reporting.
It's not for nothing that one party is considered the anti-science party.
The difference is that generally when the polls sour for Dems, they blame themselves collectively. When Repubs see the polls sour, they blame the polls. Neither is objective, but one of them is somewhat less isolated from reality.
[1]: http://mediamatters.org/research/2004/07/14/33-internal-fox-...
Take for instance discussions around social security. I swear most of the arguments I hear have no real data to back it up. Nate Silver represents a new generation of reporting that threatens the status quo journalist. Notice I don't say traditional journalism because I don't believe you can replace true journalism (go out and actually do investigation and reporting). I just believe Nate Silver is leading the way to more fact based approach.
http://www.theatlanticwire.com/politics/2012/10/people-who-c...
e: The article is now up for me, and it definitely covers the same topic.
He's dealing with terribly sketchy data. Response rates to political polls in America are south of 10%, and there's no proof that the portion of the population willing to have a conversation about their political preferences with a stranger is representative of the population as a whole.
He's also dealing with incomplete data. The 'likely voter' screen pollsters use to determine who's actually going to the polls isn't always revealed, and it varies from organization to organization. Polling is a cash-strapped industry and some polls' likely-voter screens are much more porous, cutting down on staff required and voter contacts needed to generate a 'statistically-significant' sample. But details of the likely-voter screens and response rates aren't always available.
His results are also completely unverifiable. Let's say his final prediction is 80-20 in favor of one candidate, but the other candidate wins. Well, his model did say that'll happen one time in five, so you can't really criticize it.
The appropriate amount of precision to conclude with, given all of the fuzzy inputs, is something like 'well, we can't tell who's going to win - it's close, although this one candidate's chances look slightly better'. It's not 'Candidate A's chances improved from 74.6% yesterday to 76.5% today'.
That's what makes Nate Silver so irritating - he doesn't know any more than the journalists, but he claims he does.
That's a good point. He is using percentage chances everywhere but then running only one trial (election).
Romney win:
J: +1000
N: -1000
Obama win:
J: -600
N: +600
With those payouts each side has an expected value of +200 dollars for this wager, using the specified odds they believe to be true, so both should be willing to take it.
He runs (continuous?) Monte Carlo simulations over his inputs, and when he says that there's an 80-20 chance in favour of candidate O, he's saying that 80% of the simulations resulted in a win for candidate O, not candidate R.
What he does different - and his isn't the only model built off of polling data, nor are 538's probabilities that different than other models - is just trying to decipher all the available polling data to reduce the noise and focus on the signal.
I also don't think he sees himself as being in the business of picking a winner, as in saying "I believe X will win tomorrow".
One thing you might be missing is that his predictions are verifiable at the state level - a result where he predicted the overall winner but only got 30 of 50 states right versus getting 49 out of 50 states right are pretty different.
Polling aggregation and models are as much "science" as the practice of polling is in the first place.
This really isn't true, because something like 35-40 of the states aren't even remotely up for grabs and anyone intelligent could predict them after 15 mins of looking at recent polls and previous election results. So he's trying to pick winners for 10-15 states where the race is closer, but for probably half of those, it's really not that close and you'd have a pretty good chance of getting them right if you just eyeballed it. So now we're talking about 5-7 really tight swing states, and you're basically saying that predicting 44/50 states is very different than predicting 49/50. Not so sure that it is, especially given that it was a 50/50 choice for each of those states, and he did it for one election. Also, lots of pundits and bloggers made predictions in 2008; you'd expect someone to be mostly right. Fooled by Randomness and all that.
For what it's worth, I think Silver probably is right and Obama will win (though I'd prefer he didn't), but his methods do lean towards being relatively unverifiable on a short timeline.
Perhaps. But I think a lot of his critics fall into this trap of thinking that what Silver thinks is that he is offering some sort of revolutionary and perfect prediction, or that he thinks he is doing some sort of groundbreaking science.
He's just trying to make a prediction based on all of the available information. If election predictions are inherently "relatively unverifiable", Nate Silver isn't capable of changing that - just working with what he is given (and doing a great job of explaining all of the ins and outs of this stuff to the layman).
Silver also publishes projected margins of victory (with margins of error), and he does so for every state and every senate race, not just the battleground ones. As such, it's possible to do a fairly substantial review of his model against actual results.
Statistics are never perfect, but his predictions are far more transparent and verifiable than anything else on the market.
The state by state predictions are the primary method by which you can judge the accuracy of his model. If his model forecasts a state as 60% for one candidate, you can assess that level of accuracy by looking at all 60% predictions.
Let's take some examples from todays date on FiveThirtyEight.
* Florida 54.8% chance of Romney win
* Virginia 67.0% chance of Obama win
* Nevada 67.9% chance of Obama win
* North Carolina 79.6% chance of Romney win
* New Hampshire 80.4% chance of Obama win
* Iowa 80.7% chance of Obama win
* Nevada 88.7% chance of Obama win
Other states like Texas, Utah, Idaho, Wyoming are projected at 100% for Romney and New York, California, Oregon, and Illinois at 100% for Obama.
If any of the 100% states go to the opposite candidate, that is a model problem. If some of the ones specified with extremely high percentages 95% go against his predictions with a high margin of victory, again that is a model problem.
Finally, some of those close races should go against the models prediction.
Let's take the 7 states listed above. 5 of the 7 are projected for Obama and 2 for Romney. However there is only about a 37.68% chance of that exact distribution happening. I break it down as follows:
Obama-Romney * 0-7 0.02%
* 1-7 0.14%
* 2-5 2.26%
* 3-4 12.14%
* 4-3 31.48%
* 5-2 37.68%
* 6-1 16.00%
* 7-0 0.28%
Each of these numbers are probabilistic statements about the likelihood of the overall event occurring based on the probabilities. They are from a single simulation of the 7 state probabilities run 10,000 times. Each of them has it's own distribution, eg 5-2 was 37.68% in the first run, 37.38% in the next, then 38.03%, then 36.85%, etc.
This is a very simple model prediction, but by taking all 50 states into account you can get a very clear assessment of how well his model is actually predicting the outcome of the election. Complicating matters is the time series nature of the predictions.
This type of model is precisely the way you get away from "Fooled by Randomness and all that". His model is clearly articulating the amount of uncertainty in the forecast.
EV.com had Missouri as a tie (McCain ultimately won by 0.1%, 538 got the state right) and had Indiana wrong (as did 538).
I'll take a thoughtful statistical analysis any day over the words of a single television journalist giving me second-hand information from a 'source within the campaign' especially when any campaign insider has an absurdly strong incentive to manufacture public perception in his candidates favor.
Your post shows a horrible misunderstanding of how this all works. When you have sketchy data, you don't just disregard it -- it's still telling you something and the role of statistics here is to extract, from amidst all the noise, that information.
Just because something is hard doesn't mean it's not worth the attempt.
>His results are also completely unverifiable.
Kind of. But how his methodology performs over all, over a wide range of elections, is verifiable. Wikipeida mentions this:
>The accuracy of his November 2008 presidential election predictions—he correctly predicted the winner of 49 of the 50 states—won Silver further attention and commendation. The only state he missed was Indiana, which went for Barack Obama by 1%. He also correctly predicted the winner of all 35 Senate races that year.
Whether his models perform like this consistently is a very real way that the methodology can be verified.
1) His model is easy to verify... wait for the elections to be over, and see how much his predictions correlate with reality. He predicts a great many races, and the results speak for themselves.
"The accuracy of his November 2008 presidential election predictions—he correctly predicted the winner of 49 of the 50 states—won Silver further attention and commendation. The only state he missed was Indiana, which went for Barack Obama by 1%. He also correctly predicted the winner of all 35 Senate races that year."
http://en.wikipedia.org/wiki/Nate_Silver
2) Also, his model accounts for the response rate figures you cite - that was actually the point of his most recent blog post. The reason Nate gives Romney any chance at all to win is because his model predicts that there is less than 1/5 chance that the polls are systematically biased (based on data since 1968). He thinks that if he's wrong, it's exactly for the reason you say - that those who respond to the polls are a small group that don't represent the population. He has a large enough sample size to eliminate sampling error, and we're close enough to election day to discount error due to polls being a snapshot in time. So there is only systematic bias left to discuss...
"So why, then, do we have Mr. Obama as “only” an 83.7 percent favorite to win the Electoral College, and not close to 100 percent? This is because of the other potential sources of error in polling."
And that error is simply that polls don't reflect reality, and he thinks that is about a 15% chance.
http://fivethirtyeight.blogs.nytimes.com/2012/11/03/nov-2-fo...
FWIW, he got a bunch of congressional races wrong in 2010.
No, as far as I know the actual model and its parameters are not public. I understand why this is the case but without that information the basis for the conclusions are effectively as obscure as those produced by the conventional journalists decried in the article.
The fact that he doesn't include specific equations in his NYTimes blog is likely only to improve accessibility to non-technical readers.
That said, he has given enough detail for others to roughly replicate his work. See [2]
Thank you for the link to the attempt to reproduce Silver's results. I will have to spend more time to determine how closely they agree -- it isn't obvious after a cursory look.
I wonder whether Silver would be responsive if you emailed him asking about parameters.
Sure, a few ignorant ones say things to earn a bit of attention, but they are neither the standard nor the rule.
Now where have I heard that before . . .
http://gist.io/4007765 (Source: https://gist.github.com/4007765)
Journalists do it through their prism, statisticians do it through theirs (and yes Nate Silver is a statistician and so his assumptions will affect his predictions)
Other journalists perform analysis on secret or private facts. The leaks, off the record briefings of normal political discourse, as well as actual real investigative journalism (both of them who are left)
Pundits who don't get the difference are doomed to be replaced by pundits with a copy of SASS.
Investigative journalism is just doomed.
The author uses the word "objectivity" when the right word is "subjectivity". 'savvy', 'news sense', 'gut feeling' are never objective. The author himself says so a few sentences later:
Where political journalists’ information is evaluated through a subjective and nebulous professional/cultural sense of judgment.
And yet, he concludes by going back to "objectivity" :
When journalistic objectivity is confronted with scientific objectivity, its circuits are fried.
The bottom line is: this is the age-old war between what I'd call 'science' (Silver, given his methods) and 'art' ( the journalists ).
Journalistic objectivity is about reporting the news without your personal bias. I think you can give your subjective opinion (making predictions, etc) and still have it be considered objective journalism.
Thats why his tldr is about the clash between the two objectivities.
Oh, bullcrap. Silver is hand weighting different polls based on whether or not he thinks they're reliable. That, in turn, is based on his own " subjective and nebulous professional/cultural sense of judgment".
In other words, he's doing exactly what the political journalists are doing and then smearing a thin layer of math over it. This isn't "scientifically based" at all.
-- <social science> != Science
... but you're technically correct :)
I'm not certain that there is. The modern political media largely follows politics like sports, and analyze whether a particular event is likely to be good or bad for a candidate, and such shallowness does not make the electorate more informed.
However there is a case to be made that more certainty would lower voter turnout, and that that would be bad.
I think that the only thing that could clearly improve our political situation is proportional representation and limits on consecutive terms.
In a world where people place less importance on what people like Mark Halperin and Dick Morris think, news organizations would still have plenty of reason to conduct polling - that would be a world in which people value data over what "thought leaders" are telling us what they think.
Well, the campaign insiders do have access to internal polling data that the rest of us don't. *
* Although, Nate Silver apparently did get to see that data in 2008.
The losing campaign will try to claim the race is close because they don't want their supporters to be discouraged and not turn out.
The winning campaign will try to claim the race is close because they don't want their supporters to get lazy and not turn out.
For instance, Obama's electoral vote EV and chance of winning, according to 538, sharply plummets after the first debate before turning around. Between the 3rd and 12th of October, Romney actually tripled his chances of winning from about 13% to about 39%. Then it turned around and Romney dropped to about 16%. It's not difficult to build a dramatic story around that.
In fact, this kind of thing is exactly what sportswriters and business journalists do all the time. Which means ESPN is on a (slightly) higher plane of journalism than most political commentary.
Here's the quote on pundits saying the election is a toss-up: "then you should abandon the pretense that your goal is to inform rather than entertain the public".
And what does Nate Silver use? That's right.
[1] http://chester.id.au/2012/07/29/review-expert-political-judg...
Source: fivethirtyeight.blogs.nytimes.com/2012/11/03/nov-2-for-romney-to-win-state-polls-must-be-statistically-biased