Polls, Predictions, and Russian Roulette

In the aftermath of the US election, prediction websites have been heavily criticised for giving Clinton a high chance of victory. This is not the fault of the sites themselves; I think it has more to do with the mindset of the users.

This is really safe, right?

I want to distinguish between polls and predictors. Polling companies provide the raw data; sites such as FiveThirtyEight, Upshot, and the Princeton Election Consortium aggregate the data, use it as input to mathematical models, and give us the probability of different outcomes.

The polling companies got it badly wrong. The RCP final polling average was Clinton 46.8, Trump 43.6, for a Clinton lead of 3.2. The result of the popular vote was Clinton 47.9, Trump 46.9, for a Clinton lead of 1.0. In 2016, national polls were wrong by 2.2%.

Update 2017-01-25: On updated figures, the popular vote result was Clinton 48.2, Trump 46.1, for a Clinton lead of 2.1%. The national polls were wrong by only 1.1%.

A lot of third-party voters switched to one of the two main candidates: Trump and Clinton's combined vote in the final polls was 90.4, but that rose to 94.8 on Election Day. This largely benefited Trump; he gained 3.3 points, whereas Clinton only gained 1.1.

The polling industry will rightly be asking itself hard questions about its methods, where it went wrong, and how it can improve. But this degree of error is not unusual.

It's been almost universally overlooked, but in 2012, the polls were even more wrong.The RCP final polling average was Obama 48.8, Romney 48.1, for an Obama lead of 0.7; the popular vote was Obama 51.1, Romney 47.2, for an Obama lead of 3.9. In 2012, national polls were wrong by 3.2%.

In both 2012 and 2016, the polling errors were not just noise; if that was the case, we would see some polls overestimating the frontrunner's vote, and others underestimating it, so that the average was fairly accurate. Something more systematic went wrong. It's hard to say exactly what that was; 538 has some interesting speculation.

It will be cold comfort to the polling industry, but their collective accuracy has improved over 2012. It just so happens that last time, Obama had been expected to win anyway, so few people took much notice of his margin of victory.

Now we turn to the aggregation sites. They don't gather any data themselves; all they can work with is what the polling companies give them. The polls have a margin of error, so the predictions are not a sure thing. Different assumptions and mathematical methods give different results.

Final predictions for Trump's chance of victory were 29% (538), 15% (Upshot), and 6% (PEC).

(PEC had a headline >99% "win probability" for Clinton, but if you delve further into their results, they gave Trump a 6% chance to win the Electoral College.)

538 have accordingly received kudos for their "good" prediction, but I think this is mistaken. The actual result -- a clear Trump loss in the popular vote, with razor-thin victories in a few large states giving him the Electoral College -- was well within the range of plausible outcomes under the other models.

Let's take PEC with its 6% prediction. Its founder, Sam Wang, has apologised for his misleading estimate, and followed through on his promise to eat a bug if Trump won. In hindsight, some of his model's assumptions may have been poorly chosen. But I think his only serious mistake was to emphasise the very high probability of Clinton winning the popular vote, instead of her somewhat lower chance of winning the Electoral College.

In the scheme of things, 6% is not trivial. It's more likely than an NFL kicker missing an extra point; snow on Christmas Day in London or Washington DC; or drawing two pairs in a hand of poker.

Are these events mildly unusual? Yes. Would I bet the proverbial farm against any of them? Absolutely not.

Similarly, Upshot's 15% chance is slightly less than that of of getting a bullet in Russian Roulette. No sane person would bet his or her life on those odds. If somebody tied you to a chair, and announced you were going to play a single round of Russian Roulette whether you liked it or not, you would rightly be terrified.

A Trump victory isn't quite as bad as losing at Russian Roulette; but it is very, very bad.

Many people who were afraid of a Trump victory approached the situation backwards. They were looking for reassurance, any reassurance, and seized on prediction websites to provide it. Eighty-five percent chance of a Clinton win? That's pretty high, right? That means we're going to be OK, right?

No, it doesn't, and it never did. The numbers were made into a security blanket, a talisman, a lucky rabbit's foot, which they never claimed to be. I really do understand the temptation, and succumbed to it a little bit myself, although I did point out before the election that it was far from a sure thing for Clinton.

The lessons to be learned are not very comforting: Polls are subject to error, and a three-point lead is not a place of safety. There is no such thing as an infallible crystal ball; mathematical models can quantify uncertainty, but they do not remove it.