Election2016: A comparison of predictions
NOTE: This text is being regularly updated. Current data is from October 30th 2016.
In addition to our central prediction method (the BASON survey described in the previous blog) we will use a series of benchmarks for comparison compiled from a multitude of available polls, predictions, models, markets, etc. whose predictions we intend to update regularly (every five days).
Polling aggregation sites
These benchmarks are separated into several categories. The first includes sites that use a particular polling aggregation mechanism. Namely, Nate Silver’s FiveThirtyEight, New York Times’ the Upshot, the Daily Kos, the Princeton Election Consortium, Real Clear Politics average of polls, Huff Post, and PollyVote.
For each site we track the probability of winning for each candidate, their final electoral vote projection, and their projected vote share (if any). For some of these sites we have only partial data meaning that we do not always feature all three categories. In some cases we also force a conclusion. For example the Upshot does not deliver a final definitive prediction on the electoral votes, instead listing many of the leaning tossup states. We sum up the electoral votes based on even a small lean towards one or the other candidate. The specific methodology for each of these can be found on their respective websites, with each of them employing a commendable effort in the election prediction game (except RCP which is just a simple average of polls).
We will compare each of these with our BASON survey based on all three criteria (where applicable). We will provide a forecast for all three outcomes; the chances of winning, the electoral college vote, and the final vote share. Our final comparison will also be on a state-by-state basis. This comparison will be made on election day featuring, where applicable, the precision of each prediction made by the benchmarks we use.
Models
There are two kinds of election prediction models we look at. The first group are political-analyst based models done by three reputable non-partisan newsletters/websites analyzing US elections: the Cook Political Report, the Rothenberg and Gonzales Political Report, and Sabato’s Crystal Ball from the University of Virginia. Each has built a strong reputation in correctly predicting a number of electoral outcomes in the past, and each is based on a coherent and sensible political analysis of elections. For each of these we report the tossup seats given that the reports themselves do not provide numerical information on a state level in order to determine where the state is leading towards. The reason these are a category for themselves is because they only report potential electoral college votes and do not give out probabilities nor predictions for total vote share.
The second group of models are pure models based on a number of political, economic, socio-demographic etc data. These are all conveniently assembled by the website PollyVote. This website does its own forecast aggregation (which is why we have placed it in the first group), however it also assembles a whole number of different prediction methods, including prediction markets (the Iowa Electronic Markets to be specific), poll aggregators (similar to above), citizen forecasts (asked every once in a while by selected pollsters), and even their own expert panel. What we are interested in are the so-called index models based on some form of an economic indicator, important campaign issue, candidate attribute etc. In addition to these there are a number of econometric models which also use a series of economic and socio-demographic factors to predict the election result. It is worth saying that most of these don’t use any polling data.
The downside is that the pure models only predict the final outcome of who the winner will be (and by what margin) but they don’t offer a state-by-state prediction of electoral college votes.
Prediction markets
The next obvious step are the prediction markets. Prediction markets were also historically shown to be even better than the polls in predicting the outcome (at least in the US – they failed considerably for Brexit, giving, on average, an 80% probability for Remain on referendum day). Their success if often attributed to the fact that they use real money so that people actually “put their money where their mouth is”, meaning they are more likely to make better predictions. We use a total of nine such markets: Iowa Electronic Markets (IEM), PredictIt, PredictWise, Betfair, the CNN prediction market, Pivit, Hypermind, Ig, and iPredict. Each market is given a weight based on the volume of trading, so that we can calculate and compare one single prediction from all the markets. The prediction markets, unlike the regular polls, don’t produce estimates of the total vote share, neither do they produce state-by-state predictions (at least not for all states). They instead offer probabilities that a given outcome will occur, so the comparison with the BASON survey will be done purely on the basis of the probability distributions of an outcome.
Superforcasters
And finally we will compare our method against the Superforcaster crowd of the Good Judgement Project. Superforecasters are a colloquial term for participants in Phillip Tetlock’s Good Judgement Project (GJP). The GJP was a part of a wider forecasting tournament organized by the US government agency IARPA following the intelligence community fiasco regarding the WMDs in Iraq. The government wanted to find whether or not there exists a more formidable way of making predictions which would improve decision-making, particularly in foreign policy. The GJP crowd (all volunteers, regular people, seldom experts) significantly outperformed everyone else several years in a row. Hence the title – superforecasters (there’s a number of other interesting facts about them – read more here, or buy the book). However superforecatsers are only a subset of more than 5000 forecasters who participate in the GJP. Given that we cannot really calculate and average out the performance of the top predictors within that crowd, we have to take the collective consensus forecast of all the forecasters in the GJP.
Finally, similar to the prediction markets, the GJP project doesn’t ask its participants to predict the actual voting percentage, nor does it ask the prediction of the total distribution of electoral college votes, it only asks them to gauge the probability of an event occurring. However it does ask predictions for some swing states, so we will at least report these for later comparison. They are doing the current election questions in collaboration with Washington Post’s Monkey Cage blog.
Powered by Oraclum Intelligence Systems Ltd.
0 Comments
Trackbacks/Pingbacks