There is only one Democratic primary candidate that outperforms Trump on Twitter

Analysis of Twitter hashtags of candidates for US elections done by Prospectus Research in collaboration with Oraclum Intelligence Systems uncovered a host of interesting results.

Judging solely by online activity over the past two weeks, only a single candidate in the Democratic primary race outperformed Trump on Twitter – Tulsi Gabbard.

Prospectus Research performed an in-depth text analysis of all tweets and retweets of official candidate hashtags from 21st to the 30th of October. This included recognizing the behavioral pattern of each tweet; what are people talking about when referring to each candidate, what are the key emotional cues, what are the moral concerns, and the personalities of the supporter base for each candidate.

The analysis only included official campaign hashtags to gauge the actual support for a candidate, hence eliminating potential negative comments whilst using a separate method to eliminate the impact of bots and fake accounts. Official hashtags used were for example, #tulsi2020 or #kamalaharris2020 or #Trump2020, rather than just #Gabbard, #KamalaHarris or #Trump. The official campaign tags are much more likely to be used by actual supporters or people who have something constructive to say about the campaign, rather than smear campaigns or hate messages.

Gabbard’s official campaign hashtag, #Tulsi2020, had more than 12m retweets during the 9 observed days, Trump’s campaign hashtag, #Trump2020 around 8m, while #JoeBiden (the official Biden campaign hashtag) came third, slightly short of 6m. All other primary candidates were not even close to these numbers. It is also interesting to note that, comparing their relative position in the polls to their Twitter performance last week, it seems that only Gabbard and Yang were overperforming their relative positions. This means they were being more active on social media than their polling numbers rank them. Joe Biden was about the same, whereas all others, including Warren and Sanders were severely underperforming.

Nevertheless, even with Tulsi outpreforming Trump this week, compared to online activity of Trump supporters the Democrats are collectively taking a heavy beating on Twitter.

Emotional cues, moral concerns, and personalities of voters

Observing total retweets was just the first step. We then dived deeper into the data and uncovered some of the key moral concerns addressed by voters when discussing their candidates. First we looked at the average markers for the five moral domains*. The dominant moral domain was the ingroup loyalty.

The moral concern for ingroup got a huge burst on the 27th, the day that Gabbard’s retweet storm started, triggered by allegations from Clinton. Purity followed a day later, presumably in response to the ongoing discussion, however it also coincided with Yang’s retweet bump.

Both in-group Loyalty and Purity are what is called binding moral foundations that create group cohesion and are usually more prominent in conservative voters. Loyalty especially translates into strong feelings of attachment and obligation to the group we identify with. Overarching group to identify with being the nation, whilst it can also signal loyalty to the party or the candidate. People high on this moral foundation tend to approve actions that bring cohesion, advantage, benefits and well being to the group even if those actions are costly. In a sense, in the context of partisanship, this is the most important moral foundation and research has indicated how Loyalty not only predicts partisan strength but also predicts voting intentions. Loyalty along with Authority and Sanctity or Purity are more prevalent in Conservative voters, and this finding can signal that either the conservative voting body was engaged in this tweet bump or that Gabbard’s supporters tend to to hold a distinct moral profile within the Democrats party. More precisely, as a recent study has shown, people whose identities are fused with a group and have a deep visceral feeling of oneness with a group, be that a nation or their party, equally support binding foundations (Loyalty, Authority, Purity) as do the conservatives. Meaning that it is possible that Gabbard has, through her own actions and character but also through recent accusations that posed a threat to the ingroup, gained a very powerful and dedicated base of followers that will be engaged to support and defend her online and also be motivated to vote on the upcoming elections.

Next came the social concerns that were key for the campaign. Tulsi Gabbard’s supporters spoke a lot about friends and family, most of all the candidates, that is in lieu with in group loyalty domain. On the opposite end Kamala Harris’ and Elizabeth Warren’s campaigns or supporters were not at all interested in friends. Given that family and friends are often key transmitters of trustworthy information via word of mouth a successful campaign should start placing a greater emphasis on these dimensions.

Moral concerns by hashtag show that Trump and Gabbard are the most concerned with ingroup morality. None of the candidates besides Tulsi Gabbard are significantly pro ingroup. Identity is a well-established psychological predictor of voting behavior, but this interaction works for all parties and for independent voters. Given Trump’s begrudging acceptance by Republicans, who are often offended by his disregard to the military (which is a key identity marker for American moderates and conservatives), and Gabbard’s status in the Armed forces, her ability to engage him in the general election looks overwhelmingly positive. She is the only person coming close among her other primary contenders in this category.

When it comes to personalities of the analyzed postings markers for agreeableness and conscientiousness are both extremely low across all candidates followers.

In terms of Neuroticism, Kamala Harris’ and Pete Buttigieg’s campaign followers are the most neurotic, while Biden and Warren’s are the least.

As for Openness, a typical characteristic of political liberals, it is interesting to note that Trump’s campaign is low in openness (thus possibly attracting a lot of conservatives), but no lower than Tulsi Gabbard’s. However, Biden and Buttigieg are the lowest in openness, meaning that they attract the biggest pool of conservatives for the Democrats. The Sanders supporters are, by this category the most progressive and liberal. That is, persons high Openness apart from holding more liberal and progressive attitudes have broad interests and prefer novelty over convention.

Most of the supporters for different campaigns are indistinguishable when you look at their personalities. Trump supporters look nearly identical statistically to Sanders’ supporters. The only clear difference is that Tulsi Gabbard’s supporters appear to be far more conscientious than any other group. Consciousness again, is related to orderliness and thoroughness and preference for structure. It has also been related to conservative attitudes and traditional religiosity for instance.

What does it all mean for the campaign?

These results are particularly worrying for all Democratic party candidates who want to take on Trump in 2020 – none, except for Gabbard, has the social media prowess to seriously compete with him online. A recent article by the New York Times confirms this intuition suggesting that the Democrats are seriously lagging behind Trump on social media and online in general. While the Trump campaign is spending massively to raise supporters online, the Democratic candidates are struggling to adapt to the new political landscape.

Gabbard’s Twitter overperformance shouldn’t be surprising given that on the 27^th of October she was called out, albeit indirectly, by Hillary Clinton accusing her of being groomed by the Russians (which was later corrected as she was referring to the Republicans, not the Russians) to run as a third-party candidate and thus undermine the party’s nominee to benefit Trump in 2020. Her row with Clinton garnered much attention, both in the media and especially on social media, as exemplified by what we see in our data.

One explanation behind such dramatic overperformance of Gabbard on Twitter is that this is indeed a covert operation by online trolls or even foreign entities which target social media accounts of candidates polling at lower numbers in order to infiltrate and radicalize their supporter base and potentially disrupt the frontrunners. Even if this is not true in the case of Gabbard it still presents a potential threat to the party which seems to have no effective digital strategy to combat Trump or his supporters.

In general it is very hard to explain where exactly this is all coming from, as also noted by FactBase:

“Tulsi Gabbard has a social media presence that doesn’t correlate with any data outside social media. She’s seen a huge increase, statistically, in followers and subscribers, but those increases don’t match up with any similar-size news event or in her polling visibility.”

The vast online activity from last week has yet to translate into polling numbers for Gabbard. She is still polling at around 2% nationally, with very low probability of upsetting the race. However, if online activity is any indication of actual voting behavior, as we’re led to believe by the hypothesis being thrown around for the 2016 election, Gabbard could be the only candidate with wide enough online support to challenge Trump on social media.

This type of analysis could be done on a weekly basis tracking the performance of each candidate and figuring out which messages resonate and which do not among supporters. Clearly Gabbard’s messages in the wake of Clinton’s accusations touched a positive nerve among supporters online. More importantly they resonated among moderates and swing voters (and perhaps even leaning Republicans), the key demographics that need to be courted in order to win an election.

——————–

The analysis was conducted in collaboration between Prospectus Research and Oraclum Intelligence Systems.
Prospectus Solutions AS is a Norwegian based AI and simulation design company that has developed new platforms around its Multi-Agent AI technology. One of its key current projects is the VOSA system (Virtual Online Society Analytics) which creates digital twins of real world social networks to allow for market and message testing using their Multi-Agent AI architectures at scale. More at www.prospectussolutions.com . Earlier work by Prospectus has been used to predict religious extremism, Trump, the Catalonian Referendum, and global social stability.
Oraclum is a data science and market research company that uses the power of social networks and machine learning to predict election outcomes, market movements, product demand, and consumer behaviour. In 2016 we have successfully predicted both Brexit and Trump. Our work includes doing survey experiments, data science modelling, and complex network and social network surveys.

Authors: Justin Lane (PhD Cognitive Anthropology, University of Oxford), Igor Mikloušić (PhD Personality Psychology, University of Zagreb), Vuk Vuković (PhD Political Economy, University of Oxford)

______________________________________________________________________

* Moral foundations theory is a theoretical framework created by Jonathan Haidt, Rav Iyer, Sena Koleva, Craig Joseph and Jesse Graham. They drew from the rich history of morality research within both psychology and anthropology in order to identify the full scope of human moral landscape and the reasons behind cross cultural differences and similarities in morality. Their theory proposes the existence of five universal, innate and evolved psychological modules that make us value certain traits as virtues and view certain behaviors as morally commendable or reprehensible.
– The care module, that expanded from our kin attachment system, makes us concerned with the wellbeing of others and responsive to signs of distress and harm. It is represented through the virtues of nurturance, kindness, empathy, and compassion.
– The fairness module, evolved as both a means to avoid exploitation and to enable reciprocally altruistic relationships, makes us sensitive to inequality, non-proportional compensations and cheating. It is manifested through virtues such as justice and righteousness.
– The loyalty module, evolved through forming and maintaining strong coalitions, binds us to a group. It is best represented through the virtues such as patriotism, self-sacrifice for the group heightened feeling of group loyalty and sensitivity to betrayal.
– The authority module, evolved through a long history of hierarchical social structures, manifests itself through the respect and desire for social structure and authority, valuing leadership or followership, hierarchy, traditionalist and contempt for either illegitimate authority or disrespect for authority, societal rules and/or roles.
– The purity module, evolved on top of our disgust and pathogen avoidance modules, is represented by promoting behaviors that suppress our biological desires and preserve of our minds and bodies from harmful ideas or pathogens. It is represented through values such as chastity, purity, temperance.
And although the foundations are thought to be universal how much emphasis a person will put on certain foundation, or which foundation will be central for understanding a moral issue will depend on the environment and culture as well as personality and temperament of the person. Theory gained prominence through findings which demonstrated that liberals (or in their case Democrats) and conservatives (or Republicans) differ in the weight they put on each of these foundations. Individual oriented moral foundations tend to be more represented with liberals (Care/Fairness) whilst group oriented moral concerns are more salient in the conservatives, although conservatives seem to value individual oriented moral concerns as well. For more information visit yourmorals.org

Bias in approval ratings

This post is part of the Oraclum White Paper 09/2018, published on our website. Oraclum White Papers are analytical reports on Oraclum’s predictions and prediction methods. They are designed to be informative, provide an in-depth statistical analysis of a given issue, call for a proposal to action, or introduce a unique solution based on one of Oraclum’s products.

Trump’s approval ratings have the same problem as his pre-election polls – they are biased!

Since the beginning of his presidency Donald Trump has been experiencing the lowest recorded presidential approval ratings in US history. According to FiveThirtyEight the aggregate numbers for March, after a year and two months in office, are at around 41-42%, which is lower than any US president since WWII. Usually presidential approval ratings are to some extent correlated with the probability of re-election for the second term, but even more importantly less popular Presidents tend to drag their parties down in midterm elections (see Figure 1). Notice, however, that no matter how popular they were; only two presidents, Bush in his first term, and Clinton in his second, have helped their parties gain House seats in the midterm elections. All the others have seen their parties lose seats, but the size of the loss was inversely proportional to the president’s popularity immediately prior to the midterm election. In other words, a more popular president helped his party lose less House seats.

Although at first glance this might sound concerning or reassuring (depending on whether you’re a Republican or a Democrat), bear in mind that Trump has had a stern record in defying both polls and historical political trends. He remains a very divisive president, just as much he was a divisive presidential candidate; however he still managed to carry the national victory, while exercising a very strong coattail effect with only 46.1% of the final vote share. His approval ratings therefore need to be taken with a pinch of salt, and should certainly not be examined at face value. The reason is similar as to why his polling numbers were wrong in 2016 – an increasing number of non-respondents.

Non-response bias in polls

Pollsters in many countries have been subject to a lot of bad press over the past few years. One of the main reasons was their failure to accurately grasp voter preferences in election times. The most prominent ones were the big misses in three consecutive UK elections, the 2015 and 2017 generals and the 2016 Brexit referendum, and of course the 2016 Trump victory in the US.

One reason for this is the rapidly decreasing number of response rates for traditional telephone polls. A response rate is the number of people who agree to give information in a survey divided by the total number of people called. According to Pew Research Center, a prominent pollster, and Harvard Business Review response rates have declined from 36% in 1997 to as little as 9% in 2016. This means that in 1997 in order to get say 900 people in a survey you had to call about 2500 people. In 2016 in order to get the same sample size, you needed to call 10,000 people. Random selection is crucial here (because the sample mean in random samples is very close to the population mean) and pollsters spend a lot of money and a lot of effort to achieve randomness even among those 9% who did respond. But can this be truly random is an entirely different question. Such low response rates are almost certainly making the polls subject to non-response bias. This type of bias significantly reduces the accuracy of any telephone poll, making it more likely to favor one particular candidate because they only capture the opinion of particular groups, and not the entire population. Online polls on the other hand suffer from self-selection problems and are by definition non-random and hence biased towards particular voter groups (younger, urban populations, usually also better educated).

Following the above example, assume that after calling about 10,000 people and only getting 900 (correctly stratified and supposedly randomized) respondents, the results were the following: 450 for Clinton, 400 for Trump, and 50 undecided (assuming, for simplicity, no other candidates). This would yield the poll saying that Clinton is at 50%, Trump at 44.4%, and that 5.5% are undecided, and it would conclude that because the sampling was random, the average of responses for each candidate in the sample is likely to be very close to the average in the population.

But it’s not. The low response rate suggests that some of those who do intend to vote simply did not want to express their preferences. Among all those 9000 non-respondents the majority are surely people who dislike politics and hence will not even bother to vote (turnout in the US is usually between 50 and 60%, meaning that almost half of the eligible voters simply don’t care about politics). However, among the rest there are certainly people who will in fact vote, some of which will probably support Trump, but are unwilling to say this to the interviewee directly. Why people do this is still unknown. There are two plausible expiations of why a potential Trump supporter would refuse to give an answer to a poll: 1) they are embarrassed or afraid to say they support Trump to a live phone interviewer, or 2) they distrust the pollsters and view them in the same context as the “fake news” media. There could be a number of other reasons, but one thing is sure – voters have started to avoid expressing their opinions in surveys. And this is posing a serious problem to the industry, and hence to anyone who depends on information from survey research.

Before offering potential remedies, how can we be so sure that the non-respondents in polls are Shy Trump voters? Why shouldn’t they potentially be Shy Hillary voters?

Shy Trump voters

There are several reasons suggesting that non-respondents in polls are in fact Trump rather than Hillary voters.

The first one is the recent finding that Trump’s approval ratings tend to be higher in Interactive Voice Response (IVR) or online polls as opposed to telephone polls, i.e. when polls are not being done by live human interviewers, but when people are talking to machines, or are just filling out an online poll. The difference is as large as 10 percentage points (48.7% Trump support in IVR surveys versus 38.2% in Internet surveys), which is a huge difference, much larger than the usual margin of error.

Furthermore, some pollsters survey the entire adult US population, while others focus only on those who are likely to vote (more on this below). In terms of election polls this can make a big difference. Some people might dislike a candidate so much that they will give them a negative rating, but they have no intention of voting at all (perhaps they are fed up with politics). On the other hand anyone rating a political candidate highly will surely vote for them. This implies that the responses from the entire adult population will be less accurate than responses from likely voters. This won’t make much of a difference in general market research surveys for products or services, but it will make a difference in political polling.

Finally and most importantly, our own polling during the 2016 election uncovered a systematic anti-Trump bias within the 30 states for which we ran our BASON survey.

Figure 2 compares the success of our method (x-axis) with the success of the polling average (y-axis) for the difference between the predicted and actual vote share for Donald Trump. For the polling average any dots beyond the horizontal line overestimate Trump, while any dots under the horizontal line underestimate him. For our model the overestimation is to the right of the vertical line, and the underestimation is to the left of it.

It is clear that our model under and overestimates Trump to a relatively equal extent for all states, being most precise in the most important swing states (PA, FL, NC, VA, CO, etc.). On the other hand the polls consistently underestimate Trump in almost every state. The only outlier where they overestimated Trump by almost 6%, was – DC. This implies that the polls systematically and significantly underestimated Donald Trump.

Figure 2: Oraclum’s BASON Survey vs. polling average for Trump

Looking at the same numbers for Hillary Clinton we can see that the polls were relatively good in estimating her chances. For most states they fall within a 2% margin of error, where for about 10 states the polling average was spot on. Our method once again over and underestimated Clinton to an equal extent, being the most precise where it mattered the most.

Figure 3: Oraclum’s BASON Survey vs. polling average for Clinton

Taking all this into account, the key to understating the underestimation of Trump by the pollsters was in the undecided voters. Therefore the hypothesis of a ‘Shy Trump’ voter could be true – many Trump voters simply did not want to identify themselves as such in the polls. Or they really were undecided until the very last minute, making the final decision in the polling both itself.

Finally, let’s examine this systematic bias a bit further by comparing the calibration of the BASON Survey versus the polling average (calibration is the difference between prediction and actual results). The following graph shows the difference between predictions (y-axis) and the actual results (x-axis) for our method (blue dots) and the polling average (orange dots). A good prediction should be close to having a slope of 1, which is exactly what our method proved to be (a slope of 1.1). The polling averages on the other hand experienced a flatter slope of 0.77 which confirms a systematic underestimation of Trump even in states which Clinton easily won.

Figure 4: Calibration of the BASON Survey vs. polling average

So how is Trump doing right now?

If we look at things on a state-by-state level, Gallup has this data for the entire past year. They show that Trump is unpopular countrywide and that he is still underperforming his electoral result in almost all states. In the red states his net approval is still positive, but in the swing states, including all which he had won in 2016 his current approval is worse than his electoral result (see Table 1, column “Diff from 2016”).

However there are a few important caveats here. First, the data reports averages for the entire last year, from January 20^th to December 30^th 2017, so it does not account for the recent upward change in trend. Second, for the whole country on average, using a year-long sample of over 170,000 people, the approval rating was 38%, and disapproval was 56% (this is accounting for different state size). This is a bit lower than what the current polling average for March gives him, which is between 41 and 42%. If we account for this evenly across all states it suggests that Trump is doing slightly better in the swing states that he won, however he is still underperforming in almost all of them by at least 3 to 4 percentage points (instead of 6 to 8 p.p.).

The third caveat is concerning Gallup’s methodology. Gallup is one of those pollsters that uses telephone interviews and calls a representative sample of all over-18 Americans to see if they approve or disapprove of the President. The first issue here, as emphasized previously, is that not all of these people eventually vote. When looking at Rasmussen polls which take into account only likely voters, Trump’s approval ratings are much higher – around 46-47% in March. The second issue is that the Gallup polls are done using live telephone interviews which make them more subject to anti-Trump bias. Rasmussen uses an automated polling methodology (IVR) where respondents give their opinions to an automated machine, making them more likely to be truthful.

Table 1: Trump 2017 approval ratings, 2016 pre-election polls, and 2016 results

Finally, the fair comparison in this case would not be Trump’s election result versus his approval ratings, but rather his 2016 pre-election polls versus his 2017 approval ratings (the final column in Table 1). According to these his performance in 2017 was not too far off from his pre-election polling. In fact, for a few key swing states, like Ohio or Pennsylvania, or the surprises he pulled in Michigan and Wisconsin, he is very much in the same position he was in 2016 before the election. He is underperforming in Florida, North Carolina, Georgia, and Iowa (of the states that he’d won), however when taking into account that his nation-wide trend has improved and is now at 41-42% instead of 38% that Gallup reported for 2017, Trump is very likely not doing any worse than he was in 2016.

Bearing in mind that the current polls are still underestimating Trump, the current face values of his approval ratings will not be too informative of the actual state of his popular support. His approval rating is certainly low, but so was his general election vote share, yet he still managed to scrape a victory in almost every key swing state.

What does this suggest about the midterm House and Senate races? The recent election results do offer a glimpse of hope to the Democrats as they imply that Trump’s coattail effect has waned for his fellow Republicans down the ballot. Given that he himself is not running and that opinion on politicians is at an all time low in the US, there shouldn’t be any coattail effect this time, and the House races will probably repeat the historical trend when low approval ratings of a President suggest a House net loss for his party. However, when designing pre-election prediction models on specific House and Senate races, the Democrats would be wise not to count too much on the current Trump approval ratings. The suggestion would be to either avoid placing a high emphasis on the approval ratings, as they tend to overestimate the chances of the Democratic party’s candidates, or use an alternative method that can be much better in correctly and precisely estimating the Trump approval rating.

Oraclum’s BASON Survey – the only poll to successfully solve the sampling bias problem

Oraclum’s BASON Survey is just that type of method. It is proven to yield much more accurate estimates of election outcomes when taking into account that people distrust polls and have a tendency to be less truthful. The BASON Survey asks people what they think who will win, and how they feel about who other people think will win.

It is based on a wisdom of crowds (WoC) approach to polling, accompanied by a network analysis of survey participants and their friendship networks in order to eliminate groupthink bias (see the Box for further explanation). By doing so it is able to generate much more accurate results than regular polls which struggle to find the right sampling methodology in times when response rates are at their historical lows.

By asking people to express their opinions on what others in their neighborhoods or states would do, we avoid the issue of respondents not truthfully reporting their opinions. After all, the information we seek to find out is making a prediction who will win, not who you will vote for, and it’s about thinking what other people would do. The BASON Survey leaves people well within their own comfort zones and gives them a chance to think and without pressure express their opinion on a subject.

The way we ask the questions also gives people further incentive to think about the questions and self-correct their own answers. This delayed judgment has been proven by behavioral scientists to improve accuracy of people’s own forecasts. By asking our questions this way we deliberately sacrifice large samples for accuracy. It is important to stress out that the BASON Survey does not use any private information of its respondents, and has no way of knowing who they are. We base our predictions purely on what people tell us.

The BASON Survey has been tested on a number of elections and market research problems, and has yielded incredible accuracy every time. It is the single best prediction tool available on the market, guaranteed to correctly identify what voters (or customers) want and why, without invading anyone’s privacy.

Read more about the BASON Survey here, or in the White Paper.

UK 2017 GE results: resurgence of two-party politics

The 2017 UK General Election produced another upset for election forecasters and most pollsters (kudos to YouGov for being the only big gun to predict a hung parliament with their new prediction model!). As we have surveyed in our previous text, a vast majority of the poll-based and model-based forecasters, just like with Brexit and the 2015 General Election, got it wrong. They were all estimating a comfortable victory for the Tories. And they would have been right (the Tories got 2.5m more votes than two years ago – see Table), had it not been for Labour’s major upset and an increase of over 3.5m votes since 2015.

Looks like the polls and forecasters underestimated Corbyn’s pulling effect on young voters. The young (under 25) registered in large numbers for this election (more than a million of u-25), and turnout in constituencies with younger voters rose significantly, which most likely benefited Labour.

Let’s look at the numbers:

Both Labour and the Conservatives had a huge upsurge in vote. For Labour this was, by number of votes, the best result since Blair in 1997 (13.5m votes). Never since have they had more than 11m votes. For Conservatives it was the best result since Thatcher’s 1987 election (13.7m – and almost the same actually). So who did they take their votes from?

UKIP, most certainly. UKIP had a disastrous night, which is hardly surprising. Their 3.2m votes were most likely split between Tories and Labour about 65:35. This explains the 2.1m more votes for Tories. As for Labour, they most likely got a huge chunk of the 1.5m new voters (large turnout in young, newly-registered voter constituencies), plus almost all Green Party voters. It’s hard to exactly pin down what happened in Scotland. Some are saying that SNP voters switched to the Tories, while others are claiming that they were switching to Labour, which helped the Tories take first place in several constituencies. Both might be true, as the SNP votes probably also split between the two parties, but a bit more in favor of the Tories.

Overall, a fascinating result where the two main parties got more votes than in the past 20 years. After two elections which saw the upsurge of third parties (first the LibDems in 2010 and then UKIP and SNP in 2015), this election re-established the two-party dominance in UK politics. Realignment of the political spectrum a la France? Not quite yet.

Pollster and forecaster performance

Finally, a few more words on failed pre-election predictions.

In the last week before the elections, only two pollsters were giving Labour 40% – Survation (bolded, as they had the most accurate poll), and Qriously, which was the only one predicting a Labour victory. SurveyMonkey is another one worth mentioning as they too predicted a much narrower lead (at 4%) than most other pollsters who were anchoring Labour down at 35-36%, not believing that large turnout among young voters would prove decisive.

The major forecasters (even more are surveyed here) were way off. In the last week before the vote the consensus was that the election will deliver a comfortable majority for May. Only the above praised YouGov was sceptical and went against the current. The final result wasn’t as close as they predicted, but it was really accurate. Even for LibDems which were dully underestimated by the rest. YouGov’s prediction was only slightly disturbed by the results in Scotland, where Conservatives managed to take some seats away from SNP.

UK general election poll tracker (NOT our prediction)

We are NOT predicting the 2017 UK general election. Unfortunately. They came too soon given their specifics (one must do polling in a number of toss-up constituencies to make sure to deliver an accurate prediction).

But we did keep track of what all other polls were saying. What seemed like a Conservative landslide only last month got much tighter over the past several weeks. According to the overall polling trends (shown below), Labour started its ascent in May, about the same time the Conservatives started losing ground. In the past week however, the trends started to diverge again, suggesting that the Conservatives might get the majority they thought was so easily attainable back in April.

As mentioned in the previous post, we used our ranking of UK pollsters to combine our own adjusted polling average. The idea behind it is rather simple. We create a joint weight of timing (the more recent the poll, the greater the weight), sample size (the greater the sample size, the greater the weight), whether the poll was done online or via telephone, and our ranking for each pollster. This allows us to calculate the final weighted average across all polls in a given time frame. We use this weighted average to adjust each pollster’s numbers in order to get a more accurate estimation of the final result. We take this over the last two week period and over the entire sample (from 18th of April, when the election was called). Bear in mind that this is hardly a prediction. It is just a weighting method based on past performance of pollsters. It need not mean anything in terms of how they will perform at this election. The star performers from before may flop, and the flops may turn out great.

Nevertheless, it is still fun to look at what the adjusted polling average is saying.

It is giving Conservatives a 9 point lead over Labour. This should translate into an overall majority for them easily. Notice that the difference was 13 points when looking at the entire sample of the past two months. But it is more realistic to look at only the most recent numbers.

Comparing this to other forecasting models, Elections etc (compiles an average from betting markets and models) gives the Conservatives a 10 point lead, while Electoral calculus predicts a similar 9 point margin. YouGov is the only one that seems to expect a much more narrow race. Given that they rank no.1 in our rankings this does not seem like such an impossible outcome. On the other hand YouGov did miss both Brexit and the previous general election when they called a tie. We’ll see how it goes for them this time. Interestingly, the Adjusted polling average is diverging from the rest of the group in its prediction for the Lib Dems. It seems to be underestimating them (compared to others) by a 2-3 point margin.

Vote shares don’t mean much in first-past-the-post electoral systems, so it makes more sense to look at seat projections. The adjusted polling average can’t really do that, but we can take a look at what the others are saying:

Let’s start with the similarities. There is a clear consensus that UKIP won’t get a single seat. This is hardly a surprise. The one-issue party dissolved after its one issue (leaving the EU) got realized. The SNP is very likely to repeat its excellent result from 2015, and will probably end up with around 46 seats. Now the differences. Notice a big difference in seat projections for the Lib Dems. While Lord Ashcroft and Electoral Calculus are giving them only 4 seats, YouGov has them at 12. Britian elects and Elections etc give them 9 seats. They will probably end up somewhere in between the two extremes.

What about the main parties? Everyone except YouGov is estimating a clear Conservative majority of between 134 to 150 seats. This would make it an even better result for them than in 2015, and it would give Theresa May a strong mandate. YouGov however, is estimating that the race will be much tighter. They are the only ones swimming against the current and are suggesting a Conservative majority of only 33 seats. This would necessitate a post-election coalition in order to form a government, and it would imply a disaster for May and her decision to call the election.

Given that none of these got it right for Brexit, it is difficult to impose any certainty on these predictions. But overall they seem to suggest that the Conservatives will manage to pull off a comfortable victory. Every other outcome will be yet another upset for the pollsters (well, except for YouGov).

Our NEW rankings of UK pollsters

Last year, before the Brexit referendum, we offered the first unbiased ranking of UK pollsters. This was our effort to vindicate some in the polling business during a time of widespread public anger over their failure to correctly predict the outcome of the 2015 UK general election (despite the fact that making predictions is not really the job of pollsters). The Brexit referendum was another big miss, but the public outcry against them was lower this time, most likely because people already lost faith in the polls and were expecting them to go wrong, while on the other they were too busy either gloating or sobbing following the referendum results.

The situation in the US didn’t help. When virtually everyone was predicting a clear Hilary victory (everyone except us, of course!), Trump’s victory undermined faith even in the best in the business. And while there was certainly something that went wrong in the sampling methods of the pollsters, the same error that translated into various prediction models, we will not go into too much detail as to what that was.

Instead we offered our own methodology to track how good (or bad) the polls can be, and in order to offer a score of pollsters that can be used to adjust their current numbers before the 2017 general elections. We will use these numbers to construct our adjusted polling average[1], the benchmark we usually use to compare ourselves against (e.g. see the benchmark for Brexit, or the same benchmark for the US). As you might see the adjusted polling average was wrong in both cases (unlike our predictions), which goes to show that even when adjusting for the usual polling bias, the cases of Brexit and Trump were very specific. There was a systematic bias and underestimation of both Brexit and Trump in the polls. This is why all the poll-based forecasts went wrong.

Anyway, here is the ranking list of 15 selected UK pollsters. Before making any judgements, please do read the text below explaining our methodology. We accept all suggestions and criticism.

Source of data: UK Polling Report and Wikipedia. All calculations (and mistakes) are our own.

How do we produce our rankings?

The rankings are based on past performance of pollsters for four earlier elections, the 2016 Brexit referendum, the 2015 general election, the 2014 Scottish referendum, and the 2010 general elections. In total we observed over 500 polls from 15 pollsters (not all of which participated in all three elections). We realize the sample could have been bigger by including local and previous general elections, however given that many pollsters from 10 years ago don’t produce polls anymore, and given that local elections are quite specific, we focus only on these four national elections thus far. We admit that the sample should be bigger and will think about including the local polling outcomes, adjusted for their type. There is also the issue of methodological standard of each pollster which we don’t take into account, as we are only interested in the relative performance each pollster had in the previous elections.

Given that almost all the pollsters failed to predict the outcome of the 2015 general election and Brexit, we look at the performance between pollsters as well, in order to avoid penalizing them too much for these failures. If no one saw it coming, they are all equally excused, to a certain extent. If however a few did predict correctly, the penalization against all others is more significant. We therefore jointly adjust the within accuracy (the accuracy of an individual pollster with respect to the final outcome) and the between accuracy (the accuracy of an individual pollster with respect to the accuracy of the group).

Within accuracy

To calculate the precision of pollsters in earlier elections we again have to assign weights for timing and sample size, in the same way as earlier described (older polls are less important, greater sample size is more important). Both of these factors are then summed up into the total weight for a given poll across all pollsters. We then take each individual pollster and calculate its weighted average (this is the sum of the product of all its polls and their sample and timing weights, divided by the sum of all weights – see footnote [2]). By doing so we can calculate the average error each pollster made in a given election. This is done for all three elections in our sample allowing us to calculate their within accuracy for each election. We calculate the average error for an individual pollster as the simple deviation between the weighted average polling result and the actual result for the margin between the first two parties in the elections (e.g. Conservatives and Labour)[3]. Or in plain English, how well they guessed the difference between the winner and the runner-up.

Between accuracy

After determining our within index, we estimate the accuracy between pollsters (by how much they beat each other) and sum them both into a single accuracy index. To do this we first calculate the average error for all pollsters during a single election. We then simply subtract the joint error from each individual error. This represents our between index: the greater the value, the better the pollster did against all others (note: the value can be negative).

Joint within-between ranking

To get our joint within-between index we simply sum up the two, thereby lowering the penalization across all pollsters if and when all of them missed. In this case those who missed less than others get a higher value improving their overall performance and ranking them higher on the scale.

We repeat the same procedure across all three elections and produce two final measures of accuracy. The first is the final weighting index (which we use for the ranking itself), and the second is the precision index. The difference between the two is that the precision index does not factor in the number of elections, whereas the final index does. The precision index is thus the simple average of the within-between indices, while the final index is the sum of all three divided by the total number of elections we observed regardless of how many of them the pollster participated in. The two are the same if a pollster participated in all four elections, but they differ if the pollster participated in less than four elections.

For example, consider Lord Ashcroft’s polls. We only have the data on his predictions for the 2015 general election, where he had a relativelly high precision score compared to the rest of the group (6.91 – see [3] to understand what this number means), but given that this was the only election that we have data for, his overall score is rather low (but note that his precision is higher). Pollsters that operated across all three elections give us a possibility to measure their consistency, a luxury we do not have for single-election players.

To conclude, the numbers reported under the final weighting index column represent the ranking weight that can be used to adjust the polling figures of each pollster in order to make viable predictions from them. Combined with the timing and sample size weights, it helps us calculate the final weighted average of all polls. When we do predictions this is how we calculate our strongest benchmark, the adjusted polling average.

[1] We formulate a joint weight of timing (the more recent the poll, the greater the weight), sample size (the greater the sample size, the greater the weight), whether the poll was done online or via telephone, and the ranking for each poll, allowing us to calculate the final weighted average across all polls in a given time frame.
[2] Calculated as ∑x_iw_i/ ∑w_i, where x_i is an individual poll and w_i the corresponding weight. w_i is calculated as the sum of three weights, for timing (using an adjusted exponential decay formula, decreasing from 4 to 0, where half-life is defined by t_1/2 = τ ln(2) ), for sample size (N/1000), and the ranking weight (described in the text).
[3] Define x_i as the difference between total predicted vote share of party A (v_A) and party B (v_B) for pollster i, and y as the difference between the actual vote share of the two parties. Assume A was the winner, and B was the runner-up. The within accuracy of pollster i (z_i) is then defined simply as z_i = |x_i – y|. The closer the value of z_i is to 0, the more accurate the pollster. From this we calculate the within index as I_w = (10 – z_i).

France: post-election analysis

Once again, we have correctly predicted an outcome of a general election. As we said on Saturday:

“The overall likelihood of a Le Pen victory is below 1%. Unless there is a huge Black Swan event on Sunday, Emmanuel Macron should be the next President of France.”

Emmanuel Macron is indeed the next President of France. He won by a, rather unexpected, landslide of 65.8% to 34.2%.

Source: Starus – File:2017 French presidential election – Second round – Majority vote (Metropolitan France, communes).svg, CC BY-SA 4.0, Link

Most pollsters were saying that the result will be around 60% to 40% for Macron, although the result was tightening at one point in the days prior to the debate on May 3rd. It seems that Macron’s persuasive performance in the debate was the decisive turning point of the election, as the polls started to again show a diverging trend between the two candidates. The average of post-debate polls in the last two days of the campaign was 62.3% to 37.7% for Macron. Once again a very good result for the pollsters.

Interestingly however our survey was projecting a much tighter result. We predicted that Macron was going to get 56% to Le Pen’s 44%. Our survey captured the distrust of our participants towards the projections of the pollsters. In our many interactions with our participants we could hear both sides believing that the pollsters were overestimating Macron due to a greater media bias towards him. People were highly suspicious of the polls and we had to defuse a large bias bubble. Even when we look at our sample of only Macron voters they collectively predicted a result of 59.5% to 41.5%. The Le Pen voters on the other hand collectively predicted her to get just above 50%. This is normal. Core partisan supporters are always biased in favor of their candidate. Our method works to as a way of eliminating the biases of core supporters and from that, in addition to a few glitches, formulates an optimal prediction of how the people feel what is likely to happen.

What happened in this election is that the people of France honestly believed that Le Pen has a chance of winning. Even the Macron supporters were expressing genuine concern that she might pull it off. After the debate they were more optimistic on his chances but still his numbers on our survey were not going over 60%.

Our survey therefore captured the sentiment of the voters really well. It turns out that their concerns were exaggerated, but after Brexit and Trump the French voters were obviously overly cautious in their expectations. In the end it’s worth noting that according to the exit polls, 43% of Macron voters said that they voted for him in order to stop Le Pen from winning power. This surge in anti-Le Pen supporters from genuine fear of her victory is what ultimately proved decisive in this election.

France: our final prediction

Our survey for the French elections is now officially complete. We would first and foremost like to thank everyone who participated and made our prediction possible. We analyzed the first round results in our previous post this week, and concluded that although this election seems like a landslide victory for Macron, things won’t go down that easy. The polls consistently gave him a 20 point lead in front of Le Pen, an advantage that was slightly narrowing before the debate on May 3rd, but after the debate and what was apparently a resounding victory for Macron, his chances went up again.

The final results of our prediction survey suggest a slightly tighter outcome than what the polls are currently suggesting. As shown in the figure below, we have Macron at 55.9%, and Le Pen at 44.1%. Our survey was at all times a bit more conservative and has, unlike all the other polls, consistently estimated a narrower gap between the candidates. There is a regional variation as well. We predict Macron to take all of the Western and central regions as well as Paris (Ile-de-France) and the Auvergne-Rhône-Alpes region. Le Pen will most likely win in Provence-Alpes-Côte d\’Azur and in Corse, and will probably come on top in the North-West.

However, just like the polls, our survey has never brought Macron’s victory into question. The issue was only the margin of victory and how precise our prediction will be compared to the average of the polls.

The blue and yellow zones around the lines show the average errors for each candidate’s predicted vote share. At this point the errors are around 4 percentage points in each direction. This suggest that the overall likelihood of a Le Pen victory is below 1%. Unless there is a huge Black Swan event on Sunday, Emmanuel Macron should be the next President of France.

Who will be the next President of France?

From the team that successfully predicted the victory of Donald Trump comes a new prediction survey for the upcoming second round of the French presidential elections. Vuk Vukovic, Oraclum’s director and Ilona Lahdelma, his colleague from Oxford with in-depth knowledge of french politics, discuss the results of the first round, its implications for the second, the precision of French pollsters, and how the BASON survey can again offer the best prediction for the French elections.

See the survey and its current results:

The first round of the French Presidential elections, surprisingly, went exactly as expected. The outcome suggested by the pollsters was that Macron and Le Pen would enter the second round, separated by a small margin of around 1-2%.

The uncertainty regarding the outcome, however, was much greater. A week before the election headlines everywhere were suggesting a four-way tie with each of the candidates likely to enter the second round.

The first round results did indeed show a highly divided France. On one hand there is the France of the winners of globalisation: the urban clusters of Paris, Bordeaux, Lyon, Rennes and Strasbourg with educated, internationally-oriented citizens, working in the fields of finance, business, or technology, with clearly pro-Europe and pro trade views. On the other hand, the de-industrialized North and the South-East coping with immigration and job losses have been areas of increasing support for the National Front for the past decade. Interestingly, Macron and Le Pen did not manage to penetrate the citadels of the Left and Catholic France. Mélenchon did well in the traditionally leftist South-West and Fillon among the traditionally Catholic voters of the Mid-West and South-East areas of France. However, in the greater Paris area it was interesting to see that Macron managed to get votes both from the traditionally right-leaning affluent Western areas and the poorer Eastern and Southern suburbs, demonstrating that his “not left but not right-wing” policies do manage to cross traditional partisanships. This suggests that what matters in these elections are class, income, and a person’s view of the current economic system, rather than traditional party linkages.

How the pollsters did?

Table 1 shows the weighted average bias of all French pollsters and the results are laudable, given that the average of polls was correct within a single percentage point for all the candidates. As it shows the polls may have only slightly overestimated Le Pen and Hamon, and have underestimated, again only marginally, Macron, Fillon, and Mélenchon. In an election as close as this one was, this is remarkable precision. Particularly since they have correctly indentified the trend which has carried on until election day – the trend suggested a marginal rise for Macron over Le Pen since mid-April, as well as a large increase in support for Mélenchon after the debates (and a consequential decline for Hamon as he lost the support of Left voters), and finally a gradual stabilization for Fillon since his scandals. Even in the last few weeks the polls got the trends right – both Fillon and Mélenchon were closing the gap behind the two front-runners.

What does their first-round precision tell us about the outcome in the second round? Prior to the first round all polls suggested that any candidate that went against Le Pen in the second would secure a 15 to 20 percentage point margin of victory against her. The consensus was that, just like 15 years ago, the ‘front républicain’ would rally against Le Pen in the second and produce an overwhelming victory for the opponent. The combinations of how the voters will re-align have already begun. Many are estimating slightly higher abstention rates, particularly among Fillon and Mélenchon voters. Hamon voters are more or less all projected to go to Macron, Fillon voters will split in three-ways (about 40% for Macron, one third for Le Pen, and one third will abstain), whereas Mélenchon voters are most likely to abstain from the second round (or cast a blank vote). Adding this up, it seems like this is Macron’s race to lose.

The current average of all polls after the second round stands at 60.3% for Macron to 39.7% for Le Pen (with a slight narrowing of the trend in the last few days – see the figure). Given the success of first-round polling one would assume that making a second-round prediction will not be too difficult. However the slight narrowing of the trend in the past few days could imply an interesting race even as we enter the final week of the campaign.

The factor that can play against Macron is the abstention rate that promises to be record high in these elections. Already the first round had less votes (abstention of 22%) than five years ago (abstention 20 %), with 2.6 % of cast votes being blank. The proportion of blank votes has also risen from the first round of the previous presidential elections (1.9%). Blank voting was especially strong in the middle parts of France where both Le Pen and Macron did well in the first round and in the overseas domains where Mélenchon was the winner.

This illustrates two things: first, both Macron and Le Pen have more potential votes in the middle parts of France than what they have mobilized so far, and second, areas that voted for Mélenchon in the first round are prone to abstention, suggesting that there will not be a clear overflow of votes from Mélenchon to Macron. Mobilizing these voters could very well be a determining factor in this election. Le Pen’s votes come from the lower income strata that usually correlates more with higher abstention. Macron’s more educated and urban voters come from the upper income strata, but Le Pen’s voters are more committed than Macron’s supporters. Both candidates need Mélenchon’s votes and how these votes will be cast is a mystery not only because of the candidate’s ambiguity regarding his own stance, but also because it is left to be seen how Macron and Le Pen manage to cajole these voters.

What does our survey say about the race, and what makes it so special?

Oraclum’s Bayesian Adjusted Social Network (BASON) Survey is an Internet poll. It uses the social networks between friends on Facebook to conduct a survey among them. The survey asks the participants to express: 1) their vote preference (e.g. Le Pen or Macron); 2) how much they think their preferred choice will get (in percentages); and 3) how likely they think other people will estimate that either Le Pen or Macron will win. This is essentially a wisdom of crowds (citizen forecaster) concept, the idea of which is to incorporate wider influences from peer groups that shape an individual’s perception over who is more likely to win in his or her local area.

In addition, we adjust our participants’ forecasts for groupthink – how likely they are to be biased due to their friendship networks or their source of information. In many cases information from likeminded friends or media can create a distorted perception of reality (colloquially a “bubble”) which is returned back to the crowd biasing their internal perception. To overcome this, we need to see how much individuals within the crowd are diverging from the opinion polls, but also from their internal networks of friends (from their bubbles). This is why we use social networks to capture the groupthink influence of friends, and adjust the initial predictions given by each participant. The end result is a very precise prediction that guessed, within a single percentage point, all the major US swing states in favour of Trump, as well as the Brexit referendum.

Our current survey is available at the following app: https://france2017.oraclum.co.uk/. We invite all the French readers to give it a try and tell us what they think who will be the next president of France. We will publish our final results the day before the elections, on May 6th, but you may check out our current results on the website.

—–

Vuk Vukovic is a DPhil student in Politics at the University of Oxford and the Director and co-founder of Oraclum Intelligence Systems (together with Dr Dejan Vinkovic and Prof Dr Mile Sikic).

Ilona Lahdelma is a DPhil student in Politics at the University of Oxford specializing in European politics and political behaviour.

Comparing Brexit and Trump

What the voters found important in the Brexit referendum and the US elections?

After the two most turbulent political events of 2016, the UK Brexit referendum and the victory of Donald Trump in the US Presidential elections, one cannot help to make comparisons between the two. Both attracted the same type of voter support based on several cleavages: urban vs rural, young vs old, more vs less educated, and particularly in the US white vs non-white.

However little attention has been given to specific issues the voters found important in both elections. Pollsters had this data, as asking for voter sentiment is a standard part of their surveys, but have failed to include it in their projections of the final result. Let us take a quick look into what the most important issues were in the Brexit and US election campaigns and how this could have been used to anticipate if not voter choices, then at least the trend of support for each candidate/option.

Hindsight is a wonderful thing, but saying ex-post that both of these outcomes were predictable is not doing a favor to anyone. However we feel that in future elections polls should pay more attention to voter mood, not just their answer who to vote for. Which is why the following hindsight analysis can be useful.

Brexit mood

Several months before the Brexit referendum the three most important issues to UK voters were immigration, the NHS (an issue that always has high salience in Britain), and Britain’s relationship with the EU, according to Opinium. The recovering economy, the most important issue since 2008, was now in a distant fourth place, on par with inequality and housing. At about the same time, when asked who is to be blamed for Britain’s biggest current problems, a poll by IPSOS found that the top two answers were immigrants coming to work for lower wages (32%), and rules and regulations imposed by the EU (28%). The decisions of the Labour and Conservative governments (Labour deficit-spending policies and Conservative austerity policies) came third and fourth.

As the referendum date was approaching the volume of voters which placed an emphasis on things like immigration was expanding. In June 2016 YouGov reported that the issue of immigrants coming into the UK became the number one determinant of voter choice in the upcoming referendum. A total of 33% of the voters put immigration as the most important issue helping them make their decision, whereas 28% said they were concerned with the impact Brexit would have on the UK economy. 12% of the voters said they care about Britain’s ability to make its own laws, and 11% expressed concerns about the impact of the referendum result on public services (like the NHS). It is easy to see how the Leave campaign successfully picked up the voters’ sentiment and tailored their messages precisely on immigration and the NHS. Nigel Farage got a lot attention with his ‘breaking point’ posters featuring a picture of refugees crossing the border into Europe, while arguably the biggest success of the campaign was the big red bus in which Boris Johnson was riding around featuring a huge sign saying that Britain is sending £350m to the EU which should be taken back to fund the NHS instead (both showed below).

Trump mood

In the US elections the story was similar. The economy was decreasingly important in the voters’ final choices. According to Gallup, in the 2012 elections 70% of the voters emphasized an economic issue over a non-economic one. By 2016, this has completely shifted as only 30% of voters in October 2016 recognized an economic issue (like the economy in general, unemployment, or the budget deficit) as the most important one. 70% chose a non-economic issue. The most important one was dissatisfaction with the government, followed by race relations, immigration, and terrorism and national security. Not only that, but according to PewResearch immigration has almost doubled in its salience since the 2012 election, when only 40% of the voters raked it very important, to 70% of voters raking it very important in 2016.

Naturally, the salience of an issue differed with electoral support of a particular voter. PewResearch reports that Trump supporters all found the economy, terrorism, immigration, trade policy, and Supreme Court nominations as more important than Clinton voters. Clinton supporters on the other hand placed a much higher emphasis on treatment of minorities, education, health care, and the environment. It is difficult to say which of these prevailed on a national level given that Clinton did get more votes than Trump, so it would have been more useful to look at each of these issues on a state-by-state level, particularly in the key swing states like Pennsylvania, Florida, Ohio, or North Carolina (each won by Trump), and the surprise states like Michigan and Wisconsin.

SurveyMonkey offers an insight as to how the electoral map would look depending on which issue the voters found most important. If only those who ranked the economy as the most important issue would vote, Trump would have won in a landslide (winning in all but a few states like CA, NY, MA and MD). For those ranking immigration first he would have won every state except for CA. On the other hand if only voters who found foreign policy or the environment to be important voted, they would have given Hillary a landslide, as she would win every state. When looking at those who ranked health care as very important, the result would be a tie.

The Trump campaign tailored its messages around the very key issues aimed at galvanizing their supporters – the economy (“Make America Great Again”, losing jobs because of trade with China and Mexico), immigration (the wall, terrorism, bashing Mexicans, also job losses), and particularly the voters’ dissatisfaction with the government (“drain the swamp”) which was delivered as a key message against Clinton (“lock her up”) – a personification of a typical career Washington politician. Her email scandals helped deliver that message as well.

Finally, it was interesting to see that about an equal amount of voters, 33%, said that they are voting for Clinton or Trump as a protest vote (voting against Trump or Clinton). Trump supporters emphasized his appeal as an outsider who will bring change and his policy positions on immigration and the economy. Clinton supporters valued her experience, her policy issues and her personality. Only 5% of voters would vote for one or the other just because they were Democrat or Republican.

Post-election analysis: have the polls underestimated Trump?

In the previous post we discussed the precision of our model’s result, particularly compared to predictions made by other forecasters and pollsters. On a state-by-state level our BASON method was arguably the most precise, having missed only 3 states: Michigan, Wisconsin and New Hampshire. However, for Wisconsin we did not have enough survey respondents, so we had to use an average of polls, meaning that our method actually only missed Michigan (where we gave Hillary a 0.5 point advantage) and New Hampshire (where we gave Trump a 1 point advantage). Our biggest success however was the correct predictions of Pennsylvania, Florida and North Carolina – the three key swing states that carried this election to Trump.

In this post we will examine our results a bit further and try to find out whether the polls had a problem of underestimating Trump which led to the systematic bias in their predictions.

Let’s start by first taking a look at how close our method was for our predicted states compared to the ones we did not predict. Keep in mind that we had enough data to predict only 30 states (29+DC) while for 21 of them we had to use the polling average (I explain how we made the average in this post). Most of these 21 were traditional Red and Blue states so there was no problem with the polls there (except obviously for Wisconsin which was the big surprise of the election).

The first graph looks at the popular vote the winner got in each state (x-axis) compared with the difference between actual results and our prediction (y-axis), all in percentage points. We separated the 30 states that we predicted (orange dots) versus the 21 states where we used the polling average (green diamonds).

Two immediate conclusions arise: (1) whenever we used the polling average it is clear that the polls underestimated the winner’s performance. In fact there wasn’t a single case in which the polling average gave more to the winner than what he or she actually got in the 21 traditional Red and Blue states. (2) Our model overall was within a 6% margin of error, however the most important swing states (FL, PA, NC, VA) were all within a 1% margin of error (including some others like AZ, MO, NY, TX, KS). Ohio was the only important state where we underestimated the scope of Trump’s victory, even though we correctly called it in his favor. This was the only key race the model suggested would be closer than it actually was. Nevertheless, our success for predicting FL, PA and NC was key to us making the correct prediction overall, as no pollster or forecaster was giving all three to Trump.

The ‘Shy Trump’ voters?

Let’s examine the underestimation of the polling averages a bit further, and only look at the 30 states for which we did make our prediction. We want to see how the polls performed with respect to each candidate, to see which one was more underestimated.

The second graph therefore compares the success of our method (x-axis) with the success of the polling average (y-axis) for the difference between the predicted and actual vote share for Donald Trump. In other words, for the polling average any dots beyond the horizontal line overestimate Trump, while any dots under the horizontal line underestimate him. For our model the overestimation is to the right of the vertical line, and the underestimation is to the left of it.

You can see that our model under and overestimates Trump to a relatively equal extent for all states, being most precise in the most important swing states. On the other hand the polls consistently underestimate Trump in almost every state. The only outlier where they overestimated Trump by almost 6%, was – DC. This implies that the polls systematically and significantly underestimated Donald Trump.

Looking at the same numbers for Hillary Clinton we can see that the polls were relatively good in estimating her chances. For most states they fall within a 2% margin of error, where for about 10 states the polling average was spot on. Our method once again over and underestimated Clinton to an equal extent, being the most precise where it mattered the most.

Taking all this into account, the key to understating the underestimation of Trump by the pollsters was in the undecided voters. In other words the hypothesis of a ‘Shy Trump’ voter could be true – many Trump voters simply did not want to identify themselves as such in the polls, most likely due to their mistrust of the pollsters, or any other equally likely reason. Or they really were undecided until the very last minute, making the final decision in the polling both itself.

Calibration

Finally, let’s examine this systematic bias a bit further by comparing the calibration of our model versus the polling average (calibration is the difference between prediction and actual results). The following graph shows the difference between predictions (y-axis) and the actual results (x-axis) for our method (blue dots) and the polling average (orange dots). A good prediction should be close to having a slope of 1, which is exactly what our method proved to be (a slope of 1.1). The polling averages on the other hand experienced a flatter slope of 0.77 which confirms a systematic underestimation of Trump even in states which Clinton easily won.

How large is this systematic bias? The following graph can answer this question. It compares the errors of the pollsters for Trump versus the errors of the pollsters for Clinton (errors being the difference between prediction and actual results). A good poll is expected to have a clear linear trend going through the origin, given that the overestimation of one candidate in a given state implies an underestimation of the other (adjusted for the noise from third party candidates). The polling average does exhibit this expected linear trend, however it has a big offset of around 4.3%. This is roughly the size of their systematic bias which led them to underestimate Trump’s chances.

Our method on the other hand has a much lower offset of less than 1%, which was yet another important reason of why our method did not make the same mistake of underestimating Trump.