In predicting elections, hindsight can be as telling as foresight

Mark Kayser and Arndt Leininger reveal insights from their forecasting model after German elections.

Five months ago, in May, we published a forecast for the German federal election, held on September 24. Now that the election is behind us we can evaluate how our model fared. More importantly, since ours was a structural forecast we can also offer a substantive interpretation of the election result, which eludes poll aggregation models. As opposed to polling, structural forecasting models use large volumes of historical data to make predictions. These predictions are based on the actual electoral behaviour of citizens, as opposed to surveys, which show hypothetical behaviour. Forecasting thus shows how average candidates would perform in average campaigns. After the election, we can use this information to tell which parties effectively campaigned and which did not.

Our so-called Länder Model used recent German state (Länder) parliament election results, as well as state and federal elections going back to 1961, weighting state elections held close to national elections more heavily. Calculations were aggregated into a final forecast, taking into account, among other things, differences in voter turnout.

So how did the numbers stack up? The conservative Christian Democratic Union/Christian Social Union (CDU/CSU) received 32.9% of the vote (our forecast: 35.1%), the Social Democratic Party (SPD) 20.5% (26.1%), the socialist Left party (Die Linke) 9.2% (9.2%) , the Greens (Bündnis 90/Die Grünen) 8.9% (10.5%), the economic liberal Free Democrats (FDP), 10.7% (8.7%). The far-right Alternative for Germany (AfD) received 12.6% of the vote. Because our forecast was based on an extrapolation of past elections we were not able to forecast the AfD, which only began competing in federal elections in 2013. This party is subsumed in the category Others* in our forecast. The Others (excluding the AfD) received 5.1% of the vote. Our forecast for Others (including the AfD) was 10.5%. This means we missed the actual result by an average of 3.1 percentage points. This is not bad for a forecast five months out. But it is worse than the forecast of our colleagues at the German forecasting group zweitstimme.org, who combine a structural model with vote intention pools using a dynamic Bayesian approach. They missed the actual result by an average of only 1.4 percentage points. However, they also incorporated vote intention up to two days prior to the election in their forecast. The final pre-election polls by the major German polling companies were off by an average of 1.1 percentage points.

Forecasts always come with a good degree of uncertainty. The above figure shows our point estimates along with the uncertainty surrounding them visualised as confidence bands. While we got the vote share for The Left party exactly right, we missed the other parties' vote shares by quite a bit. In particular, the SPD. The Social Democrats' result was even outside our 95% credible intervals. While we are naturally a bit disappointed about the magnitude of our miss, we are not surprised that our forecast was less accurate than final pre-election polls or the zweitstimme.org model, given the difference in time leads. Our forecast was made five months ahead of the election, zweitstimme.org’s forecast is updated daily with the last update being just before the election..

Moreover, structural models such as ours, based on theory and estimated on historical data, serve an important function other than simply as long-range forecasts. Our model sets an expectation against which outcomes can be compared. Structural models essentially predict how an average candidate with an average campaign and opposition would fare in the predicted election, given the expected conditions on the day of the vote. This allows us to compare the parties' results against their electoral potential as we saw it when they went into the campaign. Vote intention polls and models based on them like zweitstimme.org’s do not offer this possibility.

The CDU and CSU saw losses of 7.4 and 1.2 percentage points, respectively. However, their results in 2013 were unusually good. In 2009 their combined vote share was only 33.8% (with the AfD not yet in existence). If one follows our forecast, the CDU/CSU fell short of their electoral potential by 2.2 percentage points. This may well reflect losses to the AfD and a lack of mobilisation, but our forecast suggests that their losses are less dramatic if one applies the right standard. In contrast, the SPD's losses were dramatic by any standard. They underperformed their 2013 result by 5.2 percentage points and our forecast by 5.6 percentage points. Given their losses in the past two grand coalitions, their decision post-election to go into opposition, abandoning the previous ‘grand coalition’ with the CDU/CSU, seems understandable.  Less sanguinely, however, our forecast also suggests that they should not be too optimistic about how many votes they will be able to regain.

The Greens’ result most strongly highlights the difference between vote intention polls and structural models. Relative to polling just before the election, the Greens are perceived to have done rather well. However, our forecast suggests that they did not meet their full electoral potential. As we will see later, they underperformed relative to our model, where our model thought they should be strongest. We successfully predicted the re-entry of the FDP into the German parliament. Their over performance vis-à-vis our forecast seems to be a mirror image of the CDU/CSU's underperformance. Whether this has to do with the party's heavily personalised campaign is anyone's guess. Turning to those that did not make the cut in 2013, the AfD and Others, with a combined vote share of 17.7%, highlights the shortcomings of models estimated on historical data such as ours. Parties with a short electoral history are simply hard to predict with a statistical model of past elections. Last but not least, our forecast was spot-on for the Left party, despite systematic forecast errors at the state level. The party performed stronger in the west and weaker in the east than we thought it should. This is also reflected in losses for the party in the east and gains in the west with respect to their 2013 result.

This leads us to our state level forecasts. The second figure plots our forecast (on the x-axis) against a party's actual result (on the y-axis) for each of Germany’s 16 states.  A distinct pattern emerges that may shed some light on one source of error in our forecast.  The states in which our forecast deviated the most from the actual results are predominantly those in which we had the highest forecast.  This is the case for the CDU/CSU, SPD, the Left party and the Greens.  Losses were greatest in each party’s strongholds which may suggest an actual process in the election or question our use of a linear model.  Less surprisingly, we also see that we systematically under-predicted the performance of the Others category.  This is obviously due to the surge of the AfD in the 2017 election and an acknowledged shortcoming of our model, whichuses historical data and could not separately estimate results for a new party.

But learning from our forecast is one benefit of a post-mortem. For the run-up to the 2017 election, we lent our structural forecast to the the political data analysis group Signal & Rauschen who, similar to zweitstimme.org, assembled a hybrid model by combining it with polling data. Because the weighting of the polls increased as the election approached, however, both forecasts converged toward the polls, which themselves over-predicted vote-shares for the larger parties.  We will likely run our own hybrid model next time but look for ways to improve the polling component of the forecast as well as the structural component.

Note: Arndt Leininger, one of the authors of the forecasting model discussed here consulted for Signal & Rauschen.

For the original model, see:

• Leininger, Arndt and Mark A. Kayser. 2017. Eine Länderbasierte Prognose zur Bundestagswahl 2017.  With Arndt Leininger.  Politische Vierteljahresscrift 58(3).
• Kayser, Mark A. and Arndt Leininger. 20017. A Länder-based Forecast of the 2017 German Bundestag Election.  With Arndt Leininger.  PS: Political Science & Politics 50(3):689-692.