We called it! How we predicted a Trump victory

As our regular readers know by now our last week’s prediction on Donald Trump winning the US election was spot on!

We correctly predicted all the key swing states (PA, FL, NC, OH), and even that Hillary could get more votes but lose the electoral college vote. Here are our results as I presented them in a Facebook post on the eve of the election:

The story got covered first by the academic sources. University of Oxford published it as part of their main election coverage, as did the LSE EUROPP blog. Our idea of a scientific-based prediction survey was also covered in the New Scientist in the week before the election.

More news coverage soon to come!

Details of our prediction 

The results nevertheless came as an absolute shock to many, but it was the pollsters that took the biggest hit. All the major poll-based forecasts, a lot of models, the prediction markets, even the superforecaster crowd all got it wrong (we have summarized their predictions here). They estimated high probabilities for a Clinton victory, even though some were more careful than others in claiming that the race will be very tight.

Our prediction survey, on the other hand, was spot on! We predicted a Trump victory, and we called all the major swing states in his favour: Pennsylvania (which no single pollster gave to him), Florida, North Carolina, and Ohio. We gave Virginia, Nevada, Colorado, and New Mexico to Clinton, along with the usual Red states and Blue states to each. We only missed three – New Hampshire, Michigan, and Wisconsin (although for Wisconsin we didn’t have enough survey respondents to make our own prediction so we had to use the average of polls instead). Therefore the only misses of our method were actually Michigan, where it gave Clinton a 0.5 point lead, and New Hampshire where it gave Trump a 1 point lead. Every other state, although close, we called right. For example in Florida we estimated 49.9% to Trump vs. 47.3% to Clinton. In the end it was 49.1 to 47.7. In Pennsylvania we have 48.2% to Trump vs. 46.7 for Clinton (it was 48.8. to 47.6. in the end). In North Carolina our method said 51% to Trump vs. 43.5% for Clinton (Clinton got a bit more, 46.7, but Trump was spot on at 50.5%). Our model even gave Clinton a higher chance to win the overall vote share than the electoral vote, which also proved to be correct. Overall for each state, on average, we were right within a single percentage point margin. Read the full prediction here.

It was a big risk to ‘swim against the current’ with our prediction, particularly in the US where the major predictors and pollsters were always so good at making correct forecasts. But we were convinced that the method was correct even though it offered, at first glance, very surprising results.

Read more about the method here.
 
The graphics
 
Here is, once again, our final map:
 
finalmap_large

For the key swing states:
final-result-swing-states-1
 

And here are the actual results (courtesy of 270towin.com):

uselection16_results-map

Here is, btw, what the other poll-based forecasters were saying (more on that here):
chances-table_0811

In addition to these other forecasters we were tracking were even more confident in Hillary taking all the key states. As you can see no one gave PA to Trump, some were more careful about FL and NC, although they too were mostly expected to go to Hillary. However the reason I think PA was key in this election is because everyone thought Hillary’s victory was certain there. Not to mention the shocks of losing MI and WI as well. If Hillary got these three states, even by losing the toss-up FL and NC, she would have won (278 EV). This is why, we believe, all the forecasters were so certain (some more than others) that Hillary will pull it off. Holding on to what was supposed to be her strongholds (all three states were last Red under Reagan in 1984) was to be enough for victory. But Trump’s dominating performance in the Rust Belt shattered the firewall strategy of the Clinton team and won him the election. 

Election 2016: Our final prediction

The time has come. We are now finished with our prediction poll.

First and foremost we would like to thank everyone who participated and gave us their predictions. We ran them through our best model and here are the results:

It looks like a Trump victory.

Probabilities for Clinton to win the elections
Clinton has a higher probability to win the elections because she needs just one of the key swing states, while Trump needs all of them. We ran 100,000 different simulations for the total number of votes for Clinton and the maximum of the distribution of probabilities is at 274 votes, which is above the needed 270 to win.

We’re predicting Trump to take the key swing states: Florida, North Carolina, Ohio, New Hampshire, and even Pennsylvania. This should bring him to about 284 electoral college votes. On the other hand Hillary is likely to stay at 254 votes. However, the result is extremely close, with our method giving Trump just a slight edge in the key states. Having said that, all it takes is for Hillary to overturn either Florida or Pennsylvania and she will win the election. Actually, her probability to win the elections is 57.3%, compared to the Trump’s 42.7% because she has more ways to reach 270 electoral votes (see the probability graph above). We can easily say that the 5 states are toss-ups and give them to no one, thus relieving ourselves from any post-election responsibility, but where’s the fun in that?

final-result-swing-states-1

Our method

A few words on the method before we examine the results. Unlike most polling aggregation sites, prediction models, or markets, we are using an actual poll. So we don’t just pick up all the ‘low-hanging fruit’ polling data and run it through an elaborate model. We need to get actual people to come to our site and take the time to make a prediction for their state. So instead of just picking up raw data and twisting it as much as we can, we need to build our own data. Given that we were doing this with limited resources explains why our sample size is rather small (N=445).

However even with a small sample we still think the method works. Why? Our survey asks the respondents not only who they intend to vote for, but also who they think will win, by what margin, as well as their view on who other people think will win. It is basically a wisdom of crowds concept. The idea is to incorporate wider influences, including peer groups, that shape an individual’s choice on voting day.

Why might this work? When people make choices, such as in elections, they usually succumb to their standard ideological preference. However, they also weigh up the chance that their favored choice has. In other words, they think about how other people will vote. This is why people sometimes vote strategically and do not always pick their first choice, but can opt for the second or third to prevent their least-preferred option from winning.

It is going to take a number of experiments to answer the question of whether contemporary polling can be considered scientific and whether this method is indeed the best one. Our method is just the beginning of such experiments. You should therefore take these results with a pinch of salt, even if they turn out to be true. The method is still work in progress and will continue to be so in a number of forthcoming elections.

Calling the low-sample states

Knowing that our sample is likely to be small we have focused our entire attention (and money) towards the swing states. Predicting them correctly gives us a good chance of guessing the entire outcome, expecting that the traditional Red and Blue states will vote as they always do anyway. This is why for about 20 states we did not even try to get a larger sample, knowing they will not be that hard to guess (e.g. ND, SD, WY, MT, ID, AK and some Southern Red states like OK, AR, MS are obviously going to Trump, just like OR, HI, CT, RI, DE, VT, or ME will go to Clinton) (for CA, NY, IL and TX we got enough votes to make our own predictions, given that these are populous states so they obviously caught a few more voters). Have we had more resources to spend (read: money), we would have gotten enough data to predict all the states. But it makes no difference whatsoever to our final prediction.

For the aforementioned states where we had a low number of survey respondents we simply took the average of several polls, including FiveThirtyEight’s polls only data, YouGov, Huff Post, PollyVote and the aggregate polls at Real Clear Politics. Some may overlap, but the point of this task was simply to call the Red and Blue states accordingly. Which we did, and this aligns with everyone else.

Predicting the Swing States

For the following swing states we relied solely upon our own prediction survey: PA, VA, NH, NC, FL, OH, IA, MI, WI, CO, and NM. We also had enough data to make viable predictions for much of the Southern states (GA, SC, AL, TN, KY, LA) and others like UT, AZ, KS, MO, MA, MD, DC, WA, MN (this is in addition to the ones mentioned above: CA, NY, IL, TX). The non-swing states are all aligned in their traditional Red-Blue pattern (predictions for these showed in the graphs below). For Nevada and Maine we unfortunately did not have enough voters of our own to make a prediction so we used the average polling data as well. This means that the results reported for Nevada and Maine are not our own, but an average of others’ predictions. For Maine we gave 3 electoral votes to Clinton, and 1 to Trump (from Maine’s second district). Altogether we used our model to make predictions for 30 states (29+DC), and for 21 of them we used the polling averages.  Again, the point was to correctly predict the swings in order to estimate who wins it.

final-result-clinton-states-1

final-result-trump-states-1

Our results differ considerably from the majority of the forecaster crowd for the following states: we give North Carolina and Florida to Trump (most label them as toss-ups), and we also give New Hampshire and, shockingly, Pennsylvania (!) to Trump. Not a single prediction model gives Pennsylvania to Trump, and only PEC gives Hillary a run for her money in New Hampshire. We also give Ohio and Iowa to Trump, while predicting that Hillary will take victories in Virginia, Michigan, Wisconsin, Colorado and New Mexico.

final-result-swing-states-1

Now if we’re being a bit more careful with this we would also say that Florida, North Carolina, Pennsylvania and New Hampshire are all within the margin of error, and are therefore – too close to call! In other words, we predict that these will be the crucial battlegrounds where the election will be decided. So far we believe Trump has an edge in each of these.

However, if Trump manages to lose either Florida or Pennsylvania, he will lose the election. Which is why our prediction is quite bold in its statement. It seems that Trump is on much shakier ground than Hillary, however if he indeed does manage to keep an edge in all these four states, as our model predicts he will, Trump will win the election.

Election 2016: What the others are saying?

As you may or may not know, we have been tracking a considerable amount of benchmark predictions over the past month or so, to get a feel of where the race is going. We tracked everything, from polling aggregation sites to a number of poll and non-poll based models, to the prediction markets, and even the GJP superforecasters.  The trends are summarized in the following graph:

trends_election2016

It includes the average across all polling aggregation sites that we used (Nate Silver’s FiveThirtyEight, New York Times’ the Upshot, the Daily Kos, the Princeton Election Consortium, Real Clear Politics average of polls, Huff Post, and PollyVote) – these are the farthest two lines showing a considerable difference in predicted probabilities of winning from both candidates. Even after the Clinton email investigation announcement by FBI director James Comey they adjusted downwards by only a slight margin. The second and third pair are prediction markets (dark blue and light red) and superforecasters (light blue and orange) which both seem to exhibit a similar trend – they started to move closer together after the email probe 9 days ago, however after Comey’s second announcement this weekend relieving Clinton of any responsibility, they have adjusted back to high numbers for Hillary and low numbers for Trump (the markets give her an >80% chance, whereas the superforecasters give her a 75% chance). The final two pair (graphs with little dots within them) show the estimated total electoral votes from the polls and from the models that we’ve used (the Cook Political Report, the Rothenberg and Gonzales Political Report, and Sabato’s Crystal Ball). They show a slight convergence but also point to a strong Hillary victory.

Individually this is what each of our benchmarks are saying (read about the specifics here). Data is from November 7th 2016.

1. Polling aggregators – a clear victory for Hillary Clinton. Only FiveThirtyEight has been cautious, the others are mostly confident that Hillary should win a landslide (notice the electoral college predictions).

table1_0811

2. Models – I’ll put both types of models here, the political analyst-based and the pure non-poll based models. They are both saying the same thing – a clear victory for Hillary. Even if all the toss-ups go to Trump.

table2_0811 table3_0811

3. Prediction markets – they have rebounded to again point to a clear victory for Clinton. After being skeptical last week, upon the news from this weekend the markets reacted quickly and on Monday changed their trends back to support Hillary.

table4_0811

4. Superforecasters – the final group exhibited the same behavioral pattern as the markets – from being bearish on Hillary just a few days ago, only to restore to their priors as of yesterday. Interestingly, by state they picture Florida and North Carolina to be a coin toss, while giving Ohio to Trump.

table5_0811

On a state by state level we only look at the swing states, given that every single pollster/predictor gives all the blue states to Clinton and the red states to Trump. There is no disagreement there. Even among the swing states all the benchmarks are saying that PA, VA, NH and CO are going to Clinton, and that IA is going to Trump. As for the rest, most of them also give NV to Clinton (RPC gives it to Trump), FL and NC are in most cases toss-ups, while OH is in most cases going to Trump.

chances-table_0811

The first table shows the probabilities of winning per each state for each of the selected benchmarks, whereas the second table is simply calling the states as per the EV map each of the models/predictors has produced on their site. The numbers are thus electoral votes per state. The color shows who they were assigned to.

Election2016: A comparison of predictions

NOTE: This text is being regularly updated. Current data is from October 30th 2016. 

In addition to our central prediction method (the BASON survey described in the previous blog) we will use a series of benchmarks for comparison compiled from a multitude of available polls, predictions, models, markets, etc. whose predictions we intend to update regularly (every five days).

hillaryvstrump

Polling aggregation sites

These benchmarks are separated into several categories. The first includes sites that use a particular polling aggregation mechanism. Namely, Nate Silver’s FiveThirtyEight, New York Times’ the Upshot, the Daily Kos, the Princeton Election Consortium, Real Clear Politics average of polls, Huff Post, and PollyVote.

For each site we track the probability of winning for each candidate, their final electoral vote projection, and their projected vote share (if any). For some of these sites we have only partial data meaning that we do not always feature all three categories. In some cases we also force a conclusion. For example the Upshot does not deliver a final definitive prediction on the electoral votes, instead listing many of the leaning tossup states. We sum up the electoral votes based on even a small lean towards one or the other candidate. The specific methodology for each of these can be found on their respective websites, with each of them employing a commendable effort in the election prediction game (except RCP which is just a simple average of polls).

We will compare each of these with our BASON survey based on all three criteria (where applicable). We will provide a forecast for all three outcomes; the chances of winning, the electoral college vote, and the final vote share. Our final comparison will also be on a state-by-state basis. This comparison will be made on election day featuring, where applicable, the precision of each prediction made by the benchmarks we use.

Models

There are two kinds of election prediction models we look at. The first group are political-analyst based models done by three reputable non-partisan newsletters/websites analyzing US elections: the Cook Political Report, the Rothenberg and Gonzales Political Report, and Sabato’s Crystal Ball from the University of Virginia. Each has built a strong reputation in correctly predicting a number of electoral outcomes in the past, and each is based on a coherent and sensible political analysis of elections. For each of these we report the tossup seats given that the reports themselves do not provide numerical information on a state level in order to determine where the state is leading towards. The reason these are a category for themselves is because they only report potential electoral college votes and do not give out probabilities nor predictions for total vote share.

The second group of models are pure models based on a number of political, economic, socio-demographic etc data. These are all conveniently assembled by the website PollyVote. This website does its own forecast aggregation (which is why we have placed it in the first group), however it also assembles a whole number of different prediction methods, including prediction markets (the Iowa Electronic Markets to be specific), poll aggregators (similar to above), citizen forecasts (asked every once in a while by selected pollsters), and even their own expert panel. What we are interested in are the so-called index models based on some form of an economic indicator, important campaign issue, candidate attribute etc. In addition to these there are a number of econometric models which also use a series of economic and socio-demographic factors to predict the election result. It is worth saying that most of these don’t use any polling data.

The downside is that the pure models only predict the final outcome of who the winner will be (and by what margin) but they don’t offer a state-by-state prediction of electoral college votes.

Prediction markets

The next obvious step are the prediction markets. Prediction markets were also historically shown to be even better than the polls in predicting the outcome (at least in the US – they failed considerably for Brexit, giving, on average, an 80% probability for Remain on referendum day). Their success if often attributed to the fact that they use real money so that people actually “put their money where their mouth is”, meaning they are more likely to make better predictions. We use a total of nine such markets: Iowa Electronic Markets (IEM), PredictIt, PredictWise, Betfair, the CNN prediction market, Pivit, Hypermind, Ig, and iPredict. Each market is given a weight based on the volume of trading, so that we can calculate and compare one single prediction from all the markets. The prediction markets, unlike the regular polls, don’t produce estimates of the total vote share, neither do they produce state-by-state predictions (at least not for all states). They instead offer probabilities that a given outcome will occur, so the comparison with the BASON survey will be done purely on the basis of the probability distributions of an outcome.

Superforcasters

And finally we will compare our method against the Superforcaster crowd of the Good Judgement Project. Superforecasters are a colloquial term for participants in Phillip Tetlock’s Good Judgement Project (GJP). The GJP was a part of a wider forecasting tournament organized by the US government agency IARPA following the intelligence community fiasco regarding the WMDs in Iraq. The government wanted to find whether or not there exists a more formidable way of making predictions which would improve decision-making, particularly in foreign policy. The GJP crowd (all volunteers, regular people, seldom experts) significantly outperformed everyone else several years in a row. Hence the title – superforecasters (there’s a number of other interesting facts about them – read more here, or buy the book). However superforecatsers are only a subset of more than 5000 forecasters who participate in the GJP. Given that we cannot really calculate and average out the performance of the top predictors within that crowd, we have to take the collective consensus forecast of all the forecasters in the GJP.

Finally, similar to the prediction markets, the GJP project doesn’t ask its participants to predict the actual voting percentage, nor does it ask the prediction of the total distribution of electoral college votes, it only asks them to gauge the probability of an event occurring. However it does ask predictions for some swing states, so we will at least report these for later comparison. They are doing the current election questions in collaboration with Washington Post’s Monkey Cage blog.


Powered by Oraclum Intelligence Systems Ltd.

Who will be the next US President? Forecasting #Election2016

Over the next month Oraclum I.S. will be engaged in forecasting the outcome of the 2016 US Presidential elections. It has been a hectic electoral year, culminating in the unexpected nomination of Donald Trump from the Republican Party and the unexpectedly difficult path to nomination for the Democratic candidate Hillary Clinton. So far, according to the polls, Hillary Clinton is in the lead, however the extent of her advantage over Trump has had its ups and downs during the summer.

However, the problem is that not everyone trusts the polls anymore. Which is suprising to a certain extent given that in the US, unlike the rest of Europe, pollsters and particularly polling aggregation sites (like FiveThirtyEight) have on aggregate been quite accurate in their predictions thus far, at least for the Presidential elections (although the polls themselves are not a prediction tool, they are simply representations of preferences in a given point in time). Still, one cannot escape the overall feeling that pollsters are losing their reputation, as they are often being accused of complacency, sampling errors, and even deliberate manipulations.

There are legitimate reasons for this however. With the rise of online polls, proper sampling can be extremely difficult. Online polls are based on self-selection of the respondents, making them non-random and hence biased towards a particular voter group (the young, the better educated, the urban population, etc.), despite the efforts of those behind these polls to adjust them for various socio-demographic biases. On the other hand, the potential sample for traditional telephone (live interview) polls is in sharp decline, making them less and less reliable. Telephone interviews are usually done during the day biasing the results towards stay-at-home moms, retirees, and the unemployed, while most people, for some reason, do not respond to mobile phone surveys as eagerly as they once did to landline surveys. With all this uncertainty it is hard to gauge which poll(ster) should we trust and to judge the quality of different prediction methods.

Is it possible to have a more accurate prediction by asking people how confident they are that their preferred choice will win?

However, what if the answer to ‘what is the best prediction method’ lies in asking people not only who they will vote for, but also who they think will win (as ‘citizen forecasters’) and more importantly, how they feel about who other people think will win? Sounds convoluted? It is actually quite simple.

There are a number of scientific methods out there that aim to uncover how people form opinions and make choices. Elections are just one of the many choices people make. When deciding who to vote for, people usually succumb to their standard ideological or otherwise embedded preferences. However, they also carry an internal signal which tells them how much chance their preferred choice has. In other words, they think about how other people will vote. This is why, as game theory teaches us, people tend to vote strategically and do not always pick their first choice, but opt for the second or third, only to prevent their least preferred option from winning.

When pollsters make surveys they are only interested in figuring out the present state of the people’s ideological preferences. They have no idea on why someone made the choice they made. And if the polling results are close, the standard saying is: “the undecided will decide the election”. What if we could figure out how the undecided will vote, even if we do not know their ideological preferences?

One such method, focused on uncovering how people think about elections, is the Bayesian Adjusted Social Network (BASON) Survey. The BASON method is first and foremost an Internet poll. It uses the social networks between friends on Facebook and followers and followees on Twitter to conduct a survey among them. The survey asks the participants to express: 1) their vote preference (e.g. Trump or Clinton); 2) how much do they think their preferred candidate will get (in percentages); and 3) how they think other people will estimate that Trump or Clinton will win.

Let’s clarify the logic behind this. Each individual holds some prior knowledge as to what he or she thinks the final outcome will be. This knowledge can be based on current polls, or drawn from the information held by their friends and people they find more informed about politics. Based on this it is possible to draw upon the wisdom of crowds where one searches for informed individuals thus bypassing the necessity of having to compile a representative sample.

However, what if the crowd is systematically biased? For example, many in the UK believed that the 2015 election would yield a hung parliament – even Murr’s (2016) citizen forecasters  (although in relative terms the citizen forecaster model was the most precise). In other words, information from the polls is creating a distorted perception of reality which is returned back to the crowd biasing their internal perception. To overcome this, we need to see how much individuals within the crowd are diverging from the opinion polls, but also from their internal networks of friends.

Depending on how well they estimate the prediction possibilities of their preferred choices (compared to what the polls are saying), the BASON formulates their predictive power and gives a higher weight to the better predictors (e.g. if the polls are predicting a 52%-48% outcome in a given state, a person estimating that one candidate will get, say, 90% is given an insignificant weight). Group predictions can be completely wrong of course, as closed groups tend to suffer from confirmation bias. On the aggregate however, there is a way to get the most out of people’s individual opinions, no matter how internally biased they are. The Internet makes all of them easily accessible for these kinds of experiments, even if the sampling is non-random.

So, over the coming month Oraclum will be conducting the survey across the United States. The survey will start on October 10th and will run up until Election Day (November 8th) when we will provide our final forecast. Our forecasts will show the electoral votes, the total predicted percentages, and the probability distributions for two main candidates, all presented on the map of US states. They will also show the distribution of preferences for the friends of each user (so that the user could see how his or her social network is behaving and who they, as a group, are voting for), and the aggregate predictions the survey respondents will be giving.

What went wrong and what went right with our latest prediction?

The results of the Croatian election came as a surprise to many. All the pollsters (in total only 4, in addition to our BASON survey) gave the SDP-led coalition an advantage from 3 to 7 seats over HDZ. In fact, the election result for the first two parties was the exact opposite of what the polls were saying – it was SDP that should have gotten 59 seats, and HDZ 54 seats. The reason why this hasn’t happened is the crucial question on the minds of domestic political pundits, pollsters, some voters, and most political enthusiasts.

It is difficult to pin down the answer to this question on a specific factor. There are a number of hypotheses being discussed on social networks, with low turnout being the most popular one. According to this hypothesis the SDP coalition lost more voters due to low turnout than did HDZ (105,000 compared to 75,000 votes). The third party, MOST, lost even more due to low turnout (they got 180,000, down from 300,000 last year), however still not enough for them to fall below the 5% electoral threshold. On the other hand, the anti-establishment Živi zid got 20,000 more votes than last time (when they got 95,000), which were distributed all across the country – just enough for them to pass the 5% threshold in 8 out of 10 electoral districts and hence get 8 seats in parliament. The second popular hypothesis is that turnout had nothing to do with it since it was evenly distributed across the electoral districts (in other words, it fell equally across the board). Rather the result was led by the swing from the voters of MOST to the HDZ, and from SDP to Živi zid (so across a similar ideological spectrum), as depicted by the graphs below.

It is possible that both hypotheses are true. The ill-conceived campaign by the SDP leader and former PM Zoran Milanović who flirted with nationalism could have turned some left-wing SDP voters either towards the hard-left Živi zid, or towards abstinence. In a similar fashion, the dissatisfied MOST voters did not go to either SDP nor the alternative third option party Živi zid, but also abstained from voting or to a lesser extent supported HDZ. Neither of these hypotheses is possible to prove without a direct analysis of voter preferences with specific questions like: “who did you vote for on last year’s elections, and who did you vote for now?”

sdpvszz_engmostvshdz_eng

None of this however answers the question of why the polls got it so wrong (again). Even the exit polls, which are usually quite precise, suggested a tie between the two top parties, with the final result being a +5 advantage for HDZ. This is still within the 5% margin of error, however most voters don’t react too kindly to the pollsters citing their success within the error margin. Citing the margin of error is important precisely in order to tell the voters that the results of the polls should be taken with a pinch of salt. Polls may suggest an outcome, they may be very good in showing a clear trend, but to be perfectly precise, even on election day, is impossible. After all, if the polls were perfectly precise, if polling was an exact science, there would be no point in having elections at all. Nonetheless, we won’t touch upon why other pollsters failed. We will only analyze the hits and misses of our own BASON survey.

table_surveys

Our BASON survey

In our survey, apart from asking our respondents who they will vote for, we also asked them to give us a prediction of how much each party would get (in percentages) in each electoral district. And while our method has built-in mechanisms to control for group bias for each survey respondent, the problem was obviously the collective illusion of all respondents that the SDP-led coalition would win. An illusion that was further enhanced by other polls. After all, the perception that this election will be a shoe-in for SDP has been perpetuated since May when there was first speculation that the HDZ-MOST government was going to fall. This certainly affected the expectations of the survey respondents which firmly believed in a relative victory of the SDP. In other words, the method in its current form failed to break the collective bias of its respondents.

In order to recognize the potential problem, in our post-election analysis of the data we looked at the individual predictions of the voters for each party separately. For each group of voters there was an obvious overestimation of their preferred party’s chances, and an underestimation for all others, especially those ideologically distant. The interesting thing is that by looking solely at HDZ voters on the level of the entire country, on average they produced expectations that HDZ would get 65 seats, while SDP 53. A clear overestimation, but an interesting one nonetheless. This doesn’t make them good predictors (last year they gave their party an advantage of 15 seats, while the result was a tie), but it does suggest that they had a felling, even if it was overly optimistic, that their party would defeat SDP. On the other hand, those same respondents underestimated their own party in the four electoral districts around Zagreb (three of which are swing districts, and one is an SDP stronghold). In these districts their estimates of HDZ’s vote share where lower than its actual vote share. So even the optimistic HDZ voters in these districts did not expect such a strong result.

This could imply that we would need to assemble a better sample for our survey. In other words, we should aim to have around the same number of right-wing and left-wing voters (about a third of each, plus a third of all other voters), in order to more accurately capture the aggregate feelings of our respondents, even if they suffer from confirmation bias. If the survey had the same number of HDZ and SDP voters this would have most likely increased HDZ’s seat projection in our prediction.

As far as the smaller parties are concerned, the method proved its resilience and gave incredibly accurate results. We could again notice the overestimation of one’s own party and the underestimation of others, but taken together and adjusted for the biases, the predictions for the smaller parties were excellent. The BASON survey got a correct prediction on all the smaller parties within their marginal seats, where each outcome came within the highest or the second highest probability implied by the Monte Carlo simulation. This is shown on the graph below where the red line depicts the result of this election, while the black line represents the results of last year’s election. Within we have the probability distribution of the seats for each party generated by the simulation from our prediction (a longer line implies a higher probability).

predikcija_promasaji-i-pogoci

Finally, we show the relative change in votes for HDZ, the People’s coalition, and for MOST on the map of Croatian cities and municipalities. It is clear how the SDP coalition lost the majority of their votes in their strongholds (15% in Zagreb, almost 20% in Rijeka and Istria). This is highly likely due to their party leader’s attempt to move too far to the right during the campaign, forcing the party to lose its traditional left-wing supporters. The only increase came in the small cities and municipalities outside of Zagreb which testifies to the positive influence of including the centrist farmer’s party HSS in their coalition (although the increase was rather small; the dark red colour in the map represents any increase in votes).  The HDZ, on the other hand, increased its number of votes in Zagreb and across Istria by roughly a few thousand votes, while also experiencing a decline of support in its traditional strongholds – Slavonia, Lika, and Dalmatia. Their decline was albeit smaller than that of the SDP in its strongholds. The third party MOST lost more than 50% of its votes in Zagreb, Rijeka and their surroundings, and even more in Lika and Mid-Dalmatia. The reason why they were able to keep their 5 seats from Dalmatia (2 in the 9th district, and 3 in the 10th) was due to increased support in other parts of Dalmatia, particularly in the South.

sdp_pad

hdz_pad mostpad

Final prediction of the Croatian 2016 general election

This week we were focused on predicting the outcome of the forthcoming Croatian general election. We did the same thing last year and were not expecting to do Croatian elections again, at least until the local elections in 2017. However, the new government – formed after almost 2 months of post-election negotiations – survived in office for only 6 months. Last year there was no relative winner as the two main coalitions, led by the conservative HDZ and the social-democrat SDP both came tied with 56 seats (76 needed to form government). The biggest surprise was a centrist party MOST which grabbed 19 seats making it the kingmaker party, with a number of smaller parties entering Parliament as well. The new election is to be held this Sunday, and so we ran our unique BASON Survey  (Bayesian Adjusted Social Network Survey) over the past week to feel the vibe of the voters and make our election prediction.

A little bit about the survey before we share its results. The BASON Survey is an experimental polling method with an aim to give a precise prediction of elections without worrying about the representativness of our sample. It rests upon our newly-designed Facebook survey, where we use a variety of Bayesian updating methodologies to filter out internal biases in order to offer the most precise prediction. In essence we ask our participants to express their preference of who they will vote for, and more importantly, how much do they think their preferred choice will get (in percentages). Depending on how well they estimate the prediction possibilities of their preferred choices we formulate their predictive power and give higher weight to the better predictors. (We used the same thing for the Brexit referendum, see here).

The results

The final results of our BASON survey predict a close relative victory of the People’s Coalition (Narodna koalicija) led by SDP with 59 to 61 seats, just above HDZ with 52 to 54 seats. MOST would remain the third biggest party with a predicted 12 to 14 seats, while the biggest surprise of the election could be the anti-establishment Živi zid (Human blockage) with a potential of up to 8 seats (last time they had a single seat). Koalicija za premijera led by the Mayor of Zagreb Milan Bandić is estimated to get between 1 and 3 seats, while the new party Pametno (Smart) is estimated to get up to 2 seats. The two regional parties will remain within their usual seat allocations: IDS is predicted to get 3 seats, and HDSSB between 1 and 2 seats.

sabor-polukrug_final

The general conclusion is that the result will be similar to the one from last year’s elections, with two important differences: the SDP-led coalition will get the largest number of seats (although still about 16 seats short of a majority), and last year’s big surprise MOST will cease to be the only kingmaker party – they will be joined by the anti-establishment Živi zid. This suggests another long round of negotiations to form a government. Furthermore, it is interesting to note the divergence of MOST’s political capital compared to the last election where they received a whopping 19 seats. We are again likely to get around 20 seats from voters positioning themselves against the establishment SDP and HDZ (not counting the regional parties), but this time in slightly different ratios. Živi zid will, according to our results, manage to take about 5 to 8 seats from MOST which came out bruised from their short-lived coalition with HDZ. However, a clear-cut conclusion as to who took the votes from whom is beyond the scope of our method.

izborne-jedinice_final

Distribution of seats – minimum and maximum number of seats for each party

The novelty of this survey is the inclusion of so-called marginal seats (seats that can go to two or more parties at a given district – shown as half a circle in the map above). The following figure shows the probability distributions for each party in terms of total seats it could get. The People’s coalition has a probability of gaining between 55 and 64 seats, with the greatest probability between 59 and 61 seats. HDZ could get between 49 and 58, with the greatest probability between 52 and 54. The seats for MOST vary between 10 and 16, with the greatest chance between 12 and 14. Živi zid could get between 3 and 10, but most likely between 5 and 8. The coalition behind Milan Bandić could get anything between 0 and 5 seats, with 1 to 3 the most likely. Pametno could get anything between 0 and 3 seats, IDS is almost certain to get 3 seats, while HDSSB has equal chances of getting 1 and 2 seats.

sabor-vjerojatnosti-final
The x-axis measures the number of seats for each party. Each party has a distribution of seats with the widest bar denoting the highest probability.

Comparing our survey to the others

When it comes to the relative precision of our survey, it is interesting to see how it compares to the other surveys done for these elections. We compare it to the surveys done for three national televisions, each done on the level of electoral district, as ours. The following table shows the total number of seats for all three surveys, including ours, however without marginal seats. The considerable differences from all three is that the survey done by IPSOS gives the People’s coalition slightly less that the other three, while it simultaneously overestimates the chances of the Bandić coalition, which according to all three surveys vary from 0 to 7. The potential number of seats for MOST and HDZ are more or less equally predicted across all surveys. So are the chances of Živi zid which is estimated to be the biggest surprise of the election, even though they have a marginal seat in almost every electoral district, meaning they could easily under-perform as well as over-perform.

survey-comparison

In conclusion, it is interesting to note that the results of our online survey are similar to the ones done by traditional surveys, despite a non-representative sample and a very biased pool of participants. This kind of a novel predictive online survey is therefore robust to the biggest problem of surveys in general – assembling a representative sample. In further iterations we will test the survey again on other elections (first on the forthcoming US elections in November) whilst improving its methodology.

Analysis of the Brexit referendum results and our predictions

On yesterday’s historic referendum, Britain has voted Leave. It was decided by a small margin, 51.9% to 48.1% in favour of Leave, with turnout at a high 72.2% (highest since the 1990s). The outcome dealt a decisive blow against PM David Cameron who announced his resignation in the morning. The markets have had a strong negative reaction, with the FTSE plummeting, and the pound sharply declining to its 30-year low against the dollar. It was an outcome the markets failed to anticipate (or were hoping to avoid), which explains the investors’ abrupt reactions.

How did we do with our predictions?

Even though our prediction of the most likely outcome was a narrow victory for Remain (50.5 to 49.5), our model correctly anticipated that Leave has almost the same probability of winning. We gave the Leave option a 47.7% chance, admittedly more than any other model, expressing clearly that our prediction was nothing short of a coin toss.

As could be seen from our probability distribution graph below, the highest probability for the exact result of 49.5% for Leave (the one we decided to go with) was 7.53%, while the probability for the actual outcome of 51.9% for Leave was a close 6.91%, according to the model. This is a painfully small difference that comes down to pure luck in the end. Or as we said – a coin toss.

Graph1_distribution

Turns out – the coin fell on the other side. Nevertheless, we stayed within our margin of error and can honestly say that we came really close (off by 2.4%; see the graph below). We knew that the last few days have been hectic and that the Remain campaign was catching up (high turnout suggests so), but it was obviously not enough to overturn the result. Leave started to lead two weeks before the referendum, and just as our model was showing an increasing chance of Leave over the weekend, a new flock of polls switched some voters’ opinions towards a likely Remain victory by Wednesday. In addition to the trend switch in our model we also failed to receive a larger sample, which proved to be decisive in the end.

Our results in greater detail are available in the graph below. It represents the comparison of our predictions to the actual results for the UK as a whole, and for each region (in other words, the calibration of the model). It shows that most of our predictions fall within the 3% confidence interval, and almost all of them (except Northern Ireland) fall within the 5% confidence interval. The conclusion is that we have a well calibrated model.

Prediction-graph

This is even more impressive given our very small overall sample size (N=350). However even with such a small sample we were able to come really close to the actual prediction, beating a significant amount of other prediction models. Obviously the small sample size induced larger errors when it came down to certain regions (e.g. Northern Ireland or Yorkshire and Humberside), but it was remarkable how well the model performed even with so few survey respondents. Even if it did eventually predict the wrong outcome.

This was a model in its experimental phase (it still is), and the entire process is a learning curve for us. We will adapt and adjust, attempting to make our prediction method arguably the best one out there. It certainly has the potential to do that.

How did the benchmarks do?

It appears that the simplest model turned out to be the best one. The Adjusted polling average (APA), taking only the value of the polls two weeks prior to the referendum gave Leave 51% and Remain close 48.9%. This doesn’t mean individual pollsters did good, but that pollsters as a group did good (remember, polls are not predictions, they are merely representations of preferences at a given point in time). The problem with the individual pollsters was still a lot of uncertainty, such as double digits for undecided voters, even the day before the referendum. This is hardly their fault of course, but it tells us that looking at pollsters as a group is somewhat better than looking at a single individual pollster, no matter when they publish their results.

However, the Poll of polls (taking only the six last ones) was off, as it was 52:48 in favour of Remain (they’ve updated that yesterday just after I published the post, so I didn’t have time to change it). And the expert forecasting models from Number Cruncher Politics and Elections Etc both failed by 4% and 5% respectively.

Most surprisingly, the prediction markets and the betting markets have all failed significantly! As have the superforecasters. It turns out that putting your money where your mouth is still is not enough for good predictions. At least not when it comes to Britain. Prediction markets in some cases were giving an over 80% chance to Remain at the day of the referendum. In this case ours was the only model predicting a much more uncertain outcome.

Brexit: Our final prediction

Our final prediction is a close victory for Remain. According to our BAFS method Remain is expected to receive a vote share of 50.5%, giving it a 52.3% chance of winning.

Graph1_distribution

explanation

Our prediction produces a probability distribution shown on the graph above (see explanation on the right), presenting a range of likely scenarios for the given vote shares. Over the past week we have consistently been providing estimates of the final vote share and the likelihood of each outcome. Daily changes and close results simply reflect the high levels of uncertainty and ambiguity following the EU referendum. However our prediction survey (the BAFS) has noticed a slight change of trend in favour of Remain in the past two days.

This is why our final prediction gives a slight edge towards Remain, and predicts a vote share of 50.5% for Remain, and 49.5% for Leave (the graph below represents the vote share of Leave, denoted as ‘votes for Brexit’ – a higher expected vote share for Brexit decreases the probability of Remain as the final outcome). The probabilities for both outcomes are also quite close, standing at 52.3% for Remain, and 47.7% for Leave. This means that 52% of the time when polling is so close and when the people themselves expect and predict a very close result, the Remain outcome would win. 48% of the time it wouldn’t.

Vote share for Leave (votes for Brexit). The grey area describes the average error. As the sample size grew, the average error decreased.

A timeline of probabilities for both outcomes since the start of our survey.

Why such low probabilities?

Due to a relatively high margin of error (± 5.3%). However given that this is not a standard survey with a representative sample, the error term does not mean much in this case (there is a whole debate about the controversy behind the margin of error – read it here).

Nevertheless, why is the error so high? Primarily because of very high levels of uncertainty among the actual polls, as well as among the predictions our respondents gave us. Also our sample size was relatively low (more on that below). If the error was around 1%, then the probabilities would have been much higher in favour of Remain (above 70%). This is closer to what the prediction markets and the superforecasters are saying.

But this means the prediction is as good as a coin toss?

Indeed. As it stands, the race is nothing short of a coin toss.

The problem in predicting such close outcomes is the measure of relative success of the prediction method. Usually being correct within a 3% margin is considered to be quite precise. In this case nothing short of a 1% margin will be permissible, which is essentially ridiculous and extremely difficult to guessestimate.

Having said that, we do hope our prediction method will be correct within its margin of error, but more importantly that it has correctly predicted the final outcome.

How does the method work?

The BAFS method (Bayesian Adjusted Facebook Survey) is a prediction method based on its own unique poll where we ask the people not only to express their preferences, but also who they think will win and how they feel about who other people think will win. This makes it different than regular polls which are simply an expression of voter preferences at a given point in time.

The obvious difference between standard polling and our method was noticeable during our initial predictions where we had a very small sample (around a 100 respondents) which was obviously biased towards one option (it gave Remain a 66% vote share), but we were still able to produce very reliable and realistic forecasts (see the graph below, the first results pointed to a slight victory for Remain, even with very high margins of error – initially over 10%). The later variations in our predictions were small even as the sample size increased threefold.

We follow here the logic of Murr’s (2011, 2015, 2016) citizen forecaster models where even a small sample within each constituency (21 average respondents per constituency for group forecasts) is enough to provide viable estimates of the final outcome across the constituencies.

The BAFS method, similar to the citizen forecaster model, is therefore relatively robust to sample size, as well as the self-selection problem (all of our respondents voluntarily participated in the survey). Both of these issues undermine the quality of standard polling, but in this case it was shown to have little or no effect. The BAFS method, utilizing the wisdom of crowds approach (group level forecasting), benefited from a diverse, decentralized, and independent group of respondents (see Surowiecki, 2004) which gave us very realistic estimates of the final outcome. This implies that our prediction is likely to be quite close to the actual outcome on 23rd June.

How do we compare to other methods?

As we announced last month, in addition to our central prediction method we will use a series of benchmarks for comparison with our BAFS method. In the following tables we have summarized the relevant methods. For more about each method please read here. (Note: We have decided to introduce two new methods, from Number Cruncher Politics and from Elections Etc., both of which have proven track records in previous elections).

Graph4_polls

Graph5_probabilities

* For the adjusted polling average, the regular polling average, and for the forecasting polls we have factored in the undecided voters as well.

As it stands, we tend to be quite close to the predictions for the vote share (polls are slightly in favour of Leave, while other prediction methods are slightly in favour of Remain), but we tend to be a bit far from the probability estimates (the reasons for which are described above – if our error was lower, our probabilities would have also been around 70:30 in favour of Remain).

Finally, here is how the map of the UK is supposed to look like if our predictions are correct:

Regions-highres

And here is the table:

Graph6_by regions

Our Brexit survey has started!

We have started with our Brexit survey. I invite all of my UK readers to give it a go. You will be helping us test our new BAFS prediction method. In other words you will be helping us make a better prediction for the upcoming UK EU referendum. As you may or may not know, the polls are showing the country is split. At this point, a week before the referendum the uncertainty regarding the potential outcome is sky-high. With our survey, which will be running until the final day before the referendum, we hope to reduce some of this uncertainty by utilizing our unique BAFS prediction method and forecasting the exact percentage each of the two options will get.

The trick with our survey, as opposed to all others, is that we make our prediction by asking the people not only who they will vote for, but who they think will win, and how they think others will estimate who will win. For further clarification, read more here.

This basically means that we are not worried about the non-representatives of our sample, nor of the self-selection problem the survey is facing. Neither of these will bias the prediction. We hope to have our first results within a day or two, and will keep updating them every day until the day before the referendum.

Also, after you vote, you can see how your friends voted (on aggregate, not individually), and how popular/influential you are within your network – but only if you share the survey directly through the app. So don’t forget to share, either on Facebook or Twitter. Here’s the link to the survey itself.