Hacker News Reader

Comment by ninth_ant | Hacker News Reader

> I did too much bragging in the media and didn’t anticipate the extent to which public opinion toward FiveThirtyEight would shift once we became a corporate-backed incumbent rather than an eccentric upstart

Can’t speak for everyone else, but it wasn’t this for me. It was about 2016 presidential that lost me.

He tries to justify this later about how theirs was better than other outlets but I don’t care. Call it emotional, naive, unfair or whatever you want, but regardless I had zero interest in reading any of their predictions or analyses after that.

Not even mad, just that to my experience they had one job and they didn’t fulfill it at the most important time. They went from appearing insightful to just one opinion amongst so many others.

View on HN

I had complaints about 538, especially the early days, but don't understand this critique at all. A 30% chance hitting is completely unremarkable, and it was a perfectly reasonable reading of the evidence at the time. Nate isn't wrong that conventional wisdom was way off, with even supposedly statistical models giving Hillary a 99% chance of winning.

Elections, like many things, have some inherent uncertainty. A several point polling error is normal, so a candidate who is down a couple points on election day has a decent shot of winning.

Not just that, predictions also impact voter participation.

I think this is all true, but it dodges the bigger issue. A presidential election has a binary outcome: yes/no, win/lose. If your statistical model doesn’t contain this single bit in its output, then it doesn’t meet the minimum requirement for being a prediction.

Now you might say that it was on me as a consumer to understand this in 2016, but I remember the look of total shock on Nate Silver’s face when he called the winner on live TV that night, so clearly he didn’t really understand it either. Lesson learned for all of us, I guess.

> If your statistical model doesn’t contain this single bit in its output, then it doesn’t meet the minimum requirement for being a prediction.

I would be fascinated to know how you model a coin flip.

Well, that’s my point. You can model a coin flip, but you can’t predict it. A model and a prediction are two different things.

I don't understand why this is surprising. People didn't go to FiveThirtyEight to marvel the science behind it. The science was just supposed to give you what you came there for: the actual election results.

In the end, it turned out that predicting elections is still very hard, and that for all the fanfare, FiveThirtyEight performed only slightly better than what you could find in any other reputable newspaper, so it kinda lost its appeal.

538 was never about magically making polls more reliable, and only people that don't understand what polls are could think that (caveat: lots of people don't understand how polls work).

538 was about analyzing and communicating the information from those polls in an easily accessible form. If you came to the site for that, you weren't mad that they "predicted poorly something that was impossible to predict from the data sources they used" ... you were just mad at Trump for winning (despite polls suggesting otherwise).

Again, I don't think any of this matters. People were not coming there to have "information communicated to them". They were coming there for the satisfaction of knowing the results before everyone else. And FiveThirtyEight couldn't realistically deliver on that.

I'm very curious to see how polymarket fairs, compared to the news agencies. I suspect prediction markets will be the norm, going forward. Polls can't fully capture the element of anonymity that's required for an accurate poll of something controversial.

My experience was that prediction markets were lagging indicators and basically followed something akin to an aggregate opinion of polls.

This is especially viewable if you watch them during the 2020 election.

I think anyone who actually operates that way is very misguided, but it’s a fair point. But either way, 538 was such a nice site for just looking at the data in a fresh way at the time, and it’s a shame that went away.

If people are expecting anyone to have a magic prediction algorithm for things like this… I mean there’s only so much one can say. It’s not realistic.

That makes as much sense as visiting ESPN and expecting them to tell you who will definitely win the Super Bowl next year. Anyone expecting that is going to be disappointed often no matter what.

I thought it went without saying but a good analyst can't predict the future in politics, sports, or anything else. What they can do is make good probabilistic estimates of what is likely to happen. 538 wasn't pretending to do anything more than that.

If people want magic predictions there are plenty of touts and scammers willing to offer them, they don't need to waste time with charts and numbers though.

> a good analyst can't predict the future in politics, sports, or anything else. What they can do is make good probabilistic estimates of what is likely to happen. 538 wasn't pretending to do anything more than that.

Well, sure, but how big is the market for that, really? Particularly for a binary outcome like an election, knowing who's going to win is fun, reading a pundit telling you who's going to win can be fun, but ultimately the man in the street is going to take whatever the pundit said and reduce it to candidate X or candidate Y, and you can only do so much better than replacement level at that.

Polls became much less interesting as an Entertainment category once we all had experience with how unreliable they are.

> FiveThirtyEight performed only slightly better than what you could find in any other reputable newspaper

FiveThirtyEight gave Trump double the odds of the next highest reputable prediction, which was The New York Times Upshot (15%). Princeton Election Consortium gave Trump less than 1%.

That is not "only slightly better" to anyone who's statistically literate.

A FiveThirtyEight reader in 2016 was significantly better calibrated regarding Clinton’s chances than a reader of other reputable newspapers.

This embodies what 538 and its defenders miss about 538's appeal:

People didn't come to 538 for explanations on subtle points of statistical literacy (although those were provided). They came because, for whatever reason, they wanted to know who would win the election.

People not trained in statistics treated like the scoreboard at a football game- it's always better to be winning, but score is a near perfect predictor in the last minute.

Once 538 stopped delivering perfect predictions and started delivering "Actually the difference between 1% and 30% are way bigger than you think" lectures, the appeal disappeared. There are better places to learn math from.

Speak for yourself. That's not why I read FiveThirtyEight.

The purpose of FiveThirtyEight was never to be an oracle for the average person. It was always a deliberately wonky site for a wonky audience. They were very clear about that in the articles they published and topics they covered.

If we’re brutally honest the vast majority of 538 readers read it to be assured that the right outcome was outcoming.

They went to the wrong place then.

----

Nov. 1, 2016 — Election Update: Yes, Donald Trump Has A Path To Victory — https://archive.is/kwdab

> Tuesday was another pretty good day of polling for Donald Trump.

> Trump remains an underdog, but no longer really a longshot: His Electoral College chances are 29 percent in our polls-only model — his highest probability since Oct. 2 — and 30 percent in polls-plus.

> This isn’t a secure map for Clinton at all. In a race where the popular vote is roughly tied nationally, Colorado and New Hampshire are toss-ups, and Clinton’s chances are only 60 to 65 percent in Wisconsin, Michigan and Pennsylvania.

> If you want to debate a campaign’s geographic planning, Hillary Clinton spending time in Arizona is a much worse decision than Trump hanging out in Michigan or Wisconsin.

----

Sept. 16, 2016 — How Trump Could Win The White House While Losing The Popular Vote — https://archive.is/rxP5l

> Using a prototype of a demographic election calculator that FiveThirtyEight will be unveiling in the next few weeks, I decided to simulate a few election scenarios.

> The result? Clinton would carry the popular vote by 1.5 percentage points. However, Trump would win the Electoral College with 280 votes by holding all 24 Romney states and flipping Florida, Ohio, Pennsylvania, Iowa and Maine’s 2nd Congressional District from blue to red.

----

Jun 29, 2016 — Donald Trump Has A 20 Percent Chance Of Becoming President — https://archive.ph/ryIkP

> A 20 percent or 25 percent chance of Trump winning is an awfully long way from 2 percent, or 0.02 percent. It’s a real chance: about the same chance that the visiting team has when it trails by a run in the top of the eighth inning in a Major League Baseball game. If you’ve been following politics or sports over the past couple of years, I hope it’s been imprinted onto your brain that those purported long shots — sometimes much longer shots than Trump — sometimes come through.

----

FiveThirtyEight was probably the worst reputable source to read if you were looking for maximum assurances that Clinton would win.

> it turned out that predicting elections is still very hard

So maybe we shouldn't be doing it. The value of predicting an election in the large out in public seems kind of dubious, and it's more like gambling than actually being useful. A candidate only runs, and continues running, if they think they can win. All predictions like these do is confuse voters leading up to election day and while they are voting. It keep candidates from making strong cases for their platform, keeps the voters from listening to the candidates' platforms, and encourages team-based partisan politics.

Predictions are like exit polls, aren't they, in that they're both able to provide a check on official election results?

Think that's outweighed by the negatives?

No. They can’t be used to verify election results. There’s too many confounding factors.

Discussion of stats models is always complicated by the fact that a lot of people will read "30%" as a "no" prediction and claim your model is wrong if the thing happens. On the one hand, one strategy is to "hide" the numbers a bit behind a blaring headline that says "we are not sure!!" It's a bit of an art to decide when to be "sure" or not. On the other hand, in research for example you can just say screw it, I care if the correct people are correct, not if a bunch of wrong people are wrong.

I feel like the correct strategy for 538 when it was actually niche was to be precise, but then it went viral and maybe should've hit the IDK button much harder and more often after that.

The real caveat is that 538 was a Monte Carlo model, and is only as good as its inputs. "Here's what the current spread in polling numbers is *given our model and the current polling and their reported uncertainties.*" Polling uncertainties are themselves computed under certain models, and those models are subject to errors. I don't think 538 hid this, but it's a difficult caveat for people to reason about because the sorts of modeling errors that have the most influence usually represent "unknown unknowns".

Regularly referring to that ~30% spread as "one polling error" made this a lot more understandable than most statistics for many people.

Building a model for predicting the ultimate winner of a US presidential election is particularly difficult, because you are dealing with noisy input data and nonlinear effects, i.e. just a few thousand votes in a few key states can completely flip the outcome. If you then have poorly calibrated polls with a large margin of error, there is really nothing much you can do.

On the other hand, it does raise the question how valuable the 538 models for something like this really are if the outcome is a coin flip anyway.

Exactly, and correlated errors, where a polling error in one state predicts similar errors across the board.

I disagree that it's all pointless though. Most basically it's smart for campaigns to have a good model and let that inform strategy where appropriate. Since the president is a big deal other people's decisions are also impacted, and in the long run it pays to have good predictions of those chances. Also, the outcome sometimes is fairly certain and that isn't always easy to see.

I agree, it's far from pointless. The 538 model is arguably close to the best you can do considering how difficult the task is, but it's important to understand it as purely a reflection of the polling data (and 538's reliability scores for polls), and that polling data is inherently flawed. After all, there are only 2 ways to perform a perfectly accurate poll: either know the outcome a priori, or run the election. We shouldn't be too surprised when models like 538 fail to correctly predict the outcome, because that's not what they represent. It's an analytical tool for understanding the current state of polling.

That's a core mechanic in games like Dispatch.

People don't like seeing a 95% chance of winning and then losing. The game tweaks the odds, so certain thresholds become gimmes (something like "if the displayed odds are better than 75%, treat them as 100%").

That's stupid. That would piss me off.

Seems weird that it would piss you off, if you were really that invested in the cold hard stats you'd know that if it was fair rng you could still have been the 1 in 100,000 player that got lucky on 75% 40 times in a row.

Fire Emblem does something complex with averaging random numbers to do the same thing - a 95% chance to hit becomes 99.5, and the reverse for low percentages.

Conversely weather forecasters report a 40% chance of rain when the actual chance is 10% or similar.

So I have a bit of sympathy for people who don't have a good intuition for probabilities, given that the world is constantly gaslighting them.

> Discussion of stats models is always complicated by the fact that a lot of people will read "30%" as a "no" prediction and claim your model is wrong if the thing happens.

I've even heard things like "70% chance of Hillary winning means she gets 70% of the votes!" (and tangentially, my far-too-long argument with someone on Reddit who insisted "there is no way in hell 50% of the people in this town make above the median income"...)

I can't find the source anymore since 538 is no more, and I recall Nate even describing what could (and did) happen, which was that one swing state moving to the right had a high likelihood of them all moving to the right.

Yeah, Nate has talked a number of times about polling errors being correlated across states. In fact that's probably one of the most common mistakes models can make, treating correlated inputs as independent. There's a long history of that mistake in financial markets as well.

In 2024 the single most likely outcome his model had was trump winning all 7 swing states. The second most likely was Harris winning all 7.

It was 30% in the end, before the nomination it famously gave him a 2% chance of getting nominated. All the talk about 30% is disingenuous

https://en.wikipedia.org/wiki/FiveThirtyEight#2016_U.S._elec...

They had him at 30% at a time when most reputable media had him in single digits at best. You might not remember now, but the result of that election was a real shock.

From the article:

> Being dismissive of Trump’s chances early on in the Republican primary was the biggest analytical mistake of my career, and I think I deserved a lot of blame for that.

He went on to write a whole retrospective, which is exactly what you'd want to see:

https://web.archive.org/web/20160527093831/https://fivethirt...

So unless your standard is "pollsters must be perfect at all times", I dunno what else should be asked of Silver here.

I'm fine with him[0]. I don't like the apologists who explain patronizingly that 30% odds happen 3/10 times. He gave Trump much much lower odds early on and only put it at 30 in the last few weeks.

[0] I'd say the better defense of him is: the whole model is built on polls, the pollsters stink, people should complain about pollsters, not their aggregators.

If I say you have a 50% chance to win a coin flip and you lose it, that doesn't mean I'm wrong.

A key thing though is 538 did regularly test the calibration their models: https://web.archive.org/web/20190410030104/https://fivethirt...

> "What you’ll find, though, is that our calibration has generally been very, very good. For instance, out of the 5,589 events (between sports and politics combined) that we said had a 70 chance of happening (rounded to the nearest 5 percent), they in fact occurred 71 percent of the time. Or of the 55,853 events that we said had about a 5 percent chance of occurring, they happened 4 percent of the time."

Huh. I first started listening to 538 in the run-up to the 2016 election and started really paying attention to them precisely because their 30% figure was so much higher than all the other polls. It was shocking to me then (and still is now reading your comment) that people didn’t seem to understand that 30% in the context of that particular election and that particular candidate suggested a remarkably high chance of winning, not a really low chance of winning. It’s a strange thing where people seem to think that less than 50% = not happening.

I really think a majority of NYTimes and ABCnews consumers don't know the difference between a 2/3 chance (super close) of winning and 2/3 of the vote (a landslide).

Apparently neither do a big chunk of HN readers.

That’s like saying “there was only a 30 percent chance of rain today and it rained, so I will never look at the weather forecast again.”

We should have a drinking game in Nate Silver thread anyone complains about 2016 prediction. Then everyone piles on to point out how probabilities work.

It wasn't even a complaint, just a personal anecdote to help share some context as to why the site may have failed to retain consumer interest post-2016.

But yes I'll join you with the liver damage and drink 17 shots.

I long ago stopped commenting on threads (in /r/CFB) about the SP+ college football model because it go so, so, so tiring correcting the exact same misunderstandings about things as fundamental as what the model was even purporting to do let alone, how it worked. It is incredibly disheartening to me to see the staggering level of statistical illiteracy on display in any online conversation about a statistical model. I've come to the conclusion that it's not worth my, or any one else's time. I'm glad these things exist, and the communities who want them will self select. It's not use trying to convert haters.

I've long resigned myself to the fact that a lay audience will usually struggle to understand basic statistics. On HN though I don't expect to see those same issues much but have been surprised to see how many posters have trouble differentiating between 30%, 50%, and 99% probabilities.

Because they said trump only had a 30% chance to win?

What if they had said 49%? Would that have made their prediction worthless?

I mean everyone said he had a snowball’s chance in hell and then we ended up with him for two terms because the Democrats can’t stop fighting over the worse possible candidates to back that no one is asking for.

I find it interesting to blame it on the democrats. We ended up with him because enough people voted for him.

The blame is put on democrats because when they lose its because they don’t turn out and when they win its because they do. It is quite simple really. Republicans are far more reliable voters. You can look at vote totals and see this pattern. Massive delta for democrats election over election and usually half that delta for republicans.

And he won the popular vote if you believe that all U.S. elections are secure and sacrosanct. He is diabolical at getting people to talk about him and think about him constantly.

Joe Biden on the other hand was a senile wrecker for Build Back Better and the party finally made "the switch" to unelected Harris far too late in the process. Even if she was a great candidate, with her odd laughter and fascination with buses, there was not enough time to shape her candidacy. Her VP candidate choice was hobbled by rising anti-semitism in the party against Shapiro and perhaps concerns of being outshined by him. No, the Democrats did not do themselves any favors in the '24 election.

Carter, Clinton and Obama were media creations, vaulting to national prominence out of nowhere. It helped that Clinton and Obama were great, charismatic choices.

Now the traditional media is fragmented and weak. You're not seeing furtive vaulting attempts for potential phenoms like Newsome gain any traction. Who is the media going to be stuck with next time? Will it be take-two for Harris?

WHEN, not if, Harris loses bigly to Vance, then the Democrats will absolutely be to blame. Where are their all new shiny, beautiful, erudite candidates that would need all four years to gestate and promote? Shouldn't we be getting acquainted with them now? I wager they're not going to appear, and we'll get more flunkies. My theory as to why is that those currently in power in the party do not share; they're aging out and hollowing out the party in the process. We're to the point now of collapse. I'm surprised a third party on the left hasn't yet formed.

Yeah I think the presidential election system may be the one to blame. Only two parties, both very much "to the right", doesn't feel like a sane system anymore. IMO the better way is to have the government represent the people, and representing the people generally means enough politicians and having to go for a consensus.

Somehow "centrist" has taken on a negative connotation in Democratic circles, but I don't understand it. Far left politics are wrecker politics that have their moments and then (hopefully) get sent back to the political desert for a generation.

Governing in a sustainable way usually means big tent politics with give and take on the small stuff. Bill Clinton epitomized that style.

> Far left politics are wrecker politics that have their moments and then (hopefully) get sent back to the political desert for a generation.

Are you talking about US politics here? I don't see anything that would get close to "left", so "far left"... unless they reach the far left by going enough to the right I guess?

1619 project is Jacobin-style, revisionist history politics. Defund the Police, Jacobin. To flood drugs, homeless and immigrants into sanctuary cities is intentional Jacobin tactics to cause normal people to become fed up and to overturn the system. Revolutionaries are gonna revolt. Too bad everyone is too tired and distracted to give a crap about their utopian visions.

In rare and extreme circumstances it makes sense to have a revolution because the people are oppressed. For the rest of history, revolutionary fervor always comes from power hungry psychopaths who'd jail and kill everyone that stands in their way and then rule perpetually over everyone that's left if they could.

Thankfully, their time is usually short. They had a good run this time, very sneaky and devilish, but back to the political desert they go.

> Even if she was a great candidate, with her odd laughter and fascination with buses, there was not enough time to shape her candidacy.

I don't think anyone who would have ever voted for her actually cared at all about her laugh. I do think that she'd have done much better with more time though. I also agree that Democrats are too invested in themselves and the status quo to put forward a candidate who will make the kinds of meaningful changes that Democratic voters actually want. If a third party were viable I think a lot of registered democrats would be eager to jump ship, but in order for that to happen we'd need elected officials willing to make major voting reforms which at this point would take a third party.

> I also agree that Democrats are too invested in themselves and the status quo to put forward a candidate who will make the kinds of meaningful changes that Democratic voters actually want.

The old saying "the customer doesn't know what they want" seems true of the average Democratic voter. I look at the Democratic party planks as primarily boomer-era causes increasingly misaligned with technological progress and social evolution.

I see average Democratic voters as wistful and earnest, but ultimately not (yet? ever?) grounded with a cohesive vision for modern/future American society _at scale_. In my opinion, the moment for a legitimate new vision to emerge was Occupy Wall Street. All that movement seemed to yield for the grassroots was an acquaintance with homelessness culture.

The Democrats reliably back the color "beige" as a candidate. Obama was different, and he won back to back. Biden succeeded, barely, because Trump was fresh in everyone's mind. But for some reason the Democrats have been allergic to charisma for far too long.

Voting in the US, it feels like I am forced to choose between evil and incompetence.

Your description sounds more like evil vs boring, which is an easy choice for me.

And frankly I’ll vote for incompetence over evil too. Because, y’know, evil.

Hell, when it comes to government, I easily prefer boring to not-boring in an executive all else being equal. Which it wasn't here, but still.

I want the firebrands with big ideas to improve things in the legislature, and steady hands on the tiller in the executive. Unfortunately it seems that my compatriots want firebrands in both places.

I don’t think the choice is necessarily boring vs firebrands.

Obama is a great example. He was not a firebrand, not boring, and was elected twice in a row.

Charisma matters.

Obama is a very unique case. I wouldn't exactly call him a firebrand, but his public persona managed to give the perception of being a steady hand on the tiller who also had big ideas. The only other presidential candidate in my lifetime to give a similar impression was Al Gore, but he leaned more into the "steady hand" thing to his detriment. He also, as you note, had far less charisma than Obama.

I think you would be surprised by how not completely unique Obama was!

The Democrats have had a number of pretty high quality, charismatic candidates, but for some reason they never receive the party backing on the national stage, at least not for the presidency.

The Democrats seem to lean in to almost inheriting nominations. Hillary Clinton, Biden, Kamala, etc.

“The heir apparent”.

Honestly the most unique thing about Obama Was that they broke from “the heir apparent” and ran someone who had risen up from the state level and was merely a high quality congressman for a short period of time.

I mean, I followed the 2020 primary. Basically everyone who thought they had an outside chance seemed to be running there. I liked Buttigeg and Booker then, because they most fit the Obama mold -- well-spoken, charismatic, unifying rhetoric, credible claims (at the time) that they could take on the establishment, no serious baggage. Realistically, both were closer to Al Gore than Obama in execution, but still.

Booker clearly never had a chance in the primary to begin with, even moreso than Harris. Buttigeg suffered from never finding a strong enough constituency -- he wasn't loud enough to pull from the populist wing (who went to Bernie), but never made a serious play for any of the other major wings. He couldn't do that the way Biden did (by calling in favors with other popular Democrats) because he didn't have the connections, but didn't even try to do it the way Warren did (by advocating specific policies).

Meanwhile, Bernie never really tried to expand his coalition outside the populist and leftist wings (which he basically took all of). It turns out that those groups are not enough to win a democratic primary on their own. He was also perceived as a liability in the general -- remember, Democrats hadn't totally given up on Florida yet, and embracing the term "socialism" is a massive liability there.

Warren was somewhat effective in building a broad coalition, and had a similar strategy to Obama. She maybe could've pulled off that style of insurgent campaign. Her candidacy was doomed because of two factors: first, she is just not charismatic enough to compete with Bernie for the populist and leftist wings, but they are absolutely required for an insurgent campaign. Second, even if she had won the primary, she is a massive liability in the general election for a lot of reasons -- which ensured she was never going to win a primary when the primary concern was stopping the bleeding that Donald Trump was seemingly-deliberately causing.

Once you've ruled out Booker, Buttigeg, Sanders, and Warren, you're basically left with Biden vs. a crowd of other boring people. Is it really surprising that the guy who can barter for endorsements and trade on Obama's legacy managed to beat people like Klobuchar?

Now, you might say it's possible that those people (Booker, Buttigeg, Sanders, Warren) would have all made better candidates than Harris in 2024, and could have succeeded with Biden out of the field. I'd agree with that! I am not trying to argue that none of these people are better than Harris; I'm trying to argue that none of them are what Obama is. They were either trying to do something very different than what he did, or much worse at it. They all also lost significant "outsider" cred by 2024 for various reasons.

I call it incompetent because this style of uncharismatic beige has repeatedly failed to be elected, but keep being run in general elections.

I vote for the incompetence too, it’s just very fucking frustrating.

Every politician is to blame for their losses, just as every politician own their wins. The people voted for Trump because the Dems failed to get the people to vote for them.

Yes and no. It's a two-party system and a lot of people vote for candidate X to stop candidate Y getting in. Indeed IIRC a lot of democrat messaging around the previous election was explicitly about stopping Trump rather than the merits of whoever their guy was.

Enough people did not vote against Trump is my point. "The democrats" voted against Trump, otherwise they don't really count as part of "the democrats" for this election, right?

> Enough people did not vote against Trump is my point.

The argument is that there were enough people who voted against Biden-Harris, or against the DNC process, to swing it.

do you think nate silver is part of the problem or part of the solution?

the turnout-of-demographic-groups-based election model is surely the underlying intelligence failure here.

Everyone said that except for 538. That's why 538 was worth reading.

Democrats will not let people choose candidates because that may be too dangerous for their interests. We'll never get good candidates as long as the current leadership is in control.

Do you not have primaries where you are?

What position was Kamala Harris in the DNC Primaries before she was appointed as the General Election Candidate? First place? Second Place?

Surely she must have been in the top 3?

Not sure what you think they should have done? There is no way to reorganize primaries that late in the year.

It totally sucks that nobody tried to convince Biden not to run again before the primaries started though.

Plenty tried, old Joe thought he’ll clutch it out again. People were trying to dissuade him from running in 2023 already.

Denying that the democrats had two high-profile situations where the “wrong” candidate ended up running is denying the obvious. It’s manifest that Obama wasn’t to win the primary, and that the superdelegates exist for reasons.

When pressed before the election, Silver did not explain where Trump's much higher probability of winning came from. He predicted a Trump loss, Trump won, and he claimed victory because he gave Trump a better chance of winning. There's no way that strategy could have failed.

Nate Silver is not a magician! He can't magically make polls reliable!

All he (or anyone) can do is interpret or analyse poll results, and then surface their findings in a way a larger audience can understand. 538 did that better than any other poll analyst ... but they all got it wrong because the polls themselves were faulty.

TLDR; You can't get water from a stone, and no one (not even Nate Silver) can get perfectly accurate predictions from (inherently flawed) polls!

> All he (or anyone) can do is interpret or analyse poll results, and then surface their findings in a way a larger audience can understand.

He (or anybody) can make adjustments to the data. He was challenged to explain why his predictions were so different, but he wouldn't do it.

> 538 did that better than any other poll analyst

He made a binary prediction, and it was wrong. There's no such thing as "better" when you only have one outcome. Your prediction is either right or wrong. If by "better" you mean he was wrong but assigned a higher probability to a Trump victory, the best forecaster would have been someone that mechanically changed the probability of a Trump victory to slightly less than 50% no matter what the data said.

Silver claimed that his model was better because it predicted a high correlation between PA/MI/WI.

A model that predicts a 30% chance of winning the election will be wrong 1 out of 3 times, which is not quite a coin flip but close enough.

538 used the example of Trump having approximately the same chance of winning the 2016 presidential election as the Cavaliers had of winning the NBA championship round vs the Warriors. Both Trump and the Cavaliers won with a ~25% predicted chance.

538 made very clear with this analogy that both Trump and the Cavs were underdogs, and that both had a solid chance of winning.

FiveThirtyEight gave Trump a 30% chance. Their reporting did make clear that with margin was within the range of a normal polling error. And sometimes you get more than a normal polling error.

It doesn't help that the US has a terrible election system that often leads to small margins in some states being decisive.

> FiveThirtyEight gave Trump a 30% chance.

I know I'm being super conspiratorial here but why wouldn't all forecasters predict just between 30% - 70%? That way if they're "right" they can take the credit for it and if they're wrong they can say "well, we weren't that wrong". That's probably what I'd do anyway...

Of course there's more than that. Predicting higher uncertainty than warranted would be a different failure in the model. But that didn't really happen in that election.

It implies a close race or a strong reason to believe there's some sort of systemic polling miss, and if it's a blowout you still look pretty bad. Especially if you don't have some kind of good explanation for the miss/you keep making those kinds of misses frequently.

Also there's more going in those forecasts besides just the "% chance to win". There's expected results in terms of %'s of the vote for the candidates, and that's what people tend to focus on for actually analyzing your performance and credibility after the fact.

You getting the outcome correct but being off by 20 points on the margin is a much worse performance than you getting the outcome wrong but being within 0.5 points of the margin. (ex: Results are 49.75/50.25, you predicted 30/70, another outlet predicted 50.25/49.75).

You're completely right, although I believe this number was not decided personally. They just happened to pick algorithms that have this "nice" property because it will lead to the same result.

538 claimed that they post-checked thousands of elections and their percentages were pretty close. (E.g., 30% chances happened about 30% of the time, million to one chances happened every single time)

> I know I'm being super conspiratorial here but why wouldn't all forecasters predict just between 30% - 70%? That way if they're "right" they can take the credit for it and if they're wrong they can say "well, we weren't that wrong". That's probably what I'd do anyway...

For anyone making many predictions, you can analyze the outcomes to see how accurate those percentages are.

For anyone making few predictions, you should never trust their track record even if it's technically perfect.

I think this gets to the core of why a lot of this election prediction stuff doesn't work. People just don't parse the numbers the way the authors intend.

FiveThirtyEight had Trump at a 30% chance of winning, and he won. The model wasn't wrong. The less likely of two outcomes occurred. Even if they'd had him at 1% they still wouldn't technically have been wrong though I think complaints might be more warranted.

If they had Trump at 49% would you have still been angry? What about at 51%? Would it have been okay then?

Technically this is right. But if that is the case (and it seems to be), then a coin flip is better than their models. Because we only care about the current election, not a sequence of 1000 elections (which will not happen, by the way).

I don’t disagree, I think the tea leaf reading is ultimately pretty futile.

But at the same time I do think it’s valid to say it’s more than a coin flip. The polling data over the election cycle showed that Trump had a smaller but still legitimate chance of winning. The data was different in 2020, when he lost.

> But if that is the case (and it seems to be), then a coin flip is better than their models.

If a coin flip is the necessary mental model to remind you both things can happen, then sure.

People just love horse race coverage. Silver gave us the most accurate horse race coverage. Maybe the lesson is stop following horse race coverage.

But most people went back to the tea leave readers. That way when the election was over, it can justifiably be the charlatan's fault that viewers got over-invested in their predictive capabilities.

So use a coin flip to predict the weather then

We already do that in Denver.

Hard to believe that even 10 years later, people who don’t understand probability are still hung up on an event with a 30% chance of happening, happening.

> Call it emotional, naive, unfair or whatever you want

Yep definitely all those.

Why is it so hard to admit 30% is not 0%?

A dice roll has a 16.6% chance of landing on any given side, meaning an 83% chance of not landing on that side.

If you guessed a "two", and it landed on "two, I wouldn't really be that impressed, even though there was an 83% probability going against you.

> Call it emotional, naive, unfair or whatever you want

We would need a pass from the mods lol.

As a younger man I would have been with the commenters mansplaining probability, but I've aged into realizing that thinking of the election like a marble pulled from an urn whose contents we have probed with polling is just as bad as thinking of it as deterministic. The reason people read fivethirtyeight, probability-savvy or not, was almost entirely to be told what was going to happen, which is sort of incompatible with feeling you can do anything about it. In that way it's probably worse than old-fashioned pundit-driven horse race coverage because it has an air of scientific authority.

If you use 538 data, on average you make money betting. Its more correct than not

Nate was a huge outlier in that prediction, he gave trump a better chance than almost anyone else that I can recall, so why are you mad at him about that?

What made me mad is Nate seemed to turn into a MAGA troll himself after that election.

Reply on news.ycombinator.com