Nate Silver’s Value Added and Systematic Forecast Errors

Nate Silver had a very good night on November 6th.  He forecast that President Obama would win 313 electoral votes and was within 5.7% of the 332 electoral votes received by the President.  He proved to be more accurate than many individual pollsters.  Silver bases his projections on state poll averages, national polls, “state fundamentals” and trends in polls.  Anyone could have forecast the election outcome in at least 35 states, so the best way to understand Silver’s contribution is to focus on states where the election was expected to be close.  Both state poll averages and Silver’s projections underestimated President Obama’s strength in swing states, but Silver’s 538 model had substantially less systematic bias than state poll averages.

Nate Silver has brought statistical analysis of elections to the masses (or at least to the readers of the New York Times).  His blog makes it clear that when an average of polls prior to the election shows a candidate has a 55-45 advantage, it doesn’t mean that she will lose 45% of the time.  It means instead that she will almost certainly win the election.  Poll averages improve the precision of forecasts because errors due to sampling variation are decreased.  More importantly, the 538 model incorporates “state fundamentals” and poll trends to improve the precision of  forecasts. 

Consider the following comparison of state poll averages, Silver’s projections and election outcomes in 11 states that were expected to close on election day:

State State Poll Avg. Silver’s Projection Election Outcome
Colorado DEM:  1.9 DEM:  2.5 DEM:  4.7
Florida REP:    0.7 D/R:    0.0 DEM:  0.9
Iowa DEM:  2.6 DEM:  3.2 DEM:  5.6
Michigan DEM:  4.7 DEM:  7.1 DEM:  9.5
Nevada DEM:  3.6 DEM:  4.5 DEM:  6.6
New Hampshire DEM:  2.6 DEM:  3.5 DEM:  5.8
North Carolina REP:    1.9 REP:    1.7 REP:    2.2
Ohio DEM:  3.0 DEM:  3.6 DEM:  1.9
Pennsylvania DEM:  4.6 DEM:  5.9 DEM:  5.2
Virginia DEM:  1.3 DEM:  2.0 DEM:  3.0
Wisconsin DEM:  4.3 DEM:  5.5 DEM:  6.7

Notice that in every one of these states Silver’s 538 model predicted that the President would outperform the polls.  Although this infuriated many conservatives Silver was correct, on average.  The President’s vote share in the states listed above was 2.0 percentage points higher than forecasted by poll averages.  The President even out-performed Silver’s forecast by receiving 1.1 percent more of the votes cast than predicted by the 538 model.  This occurred despite the fact that Mitt Romney exceeded Silver’s expectations in North Carolina, Ohio and Virginia. 

The standard deviation of the forecast error was 1.7% for state poll averages and 1.4% for Silver’s model (about 18% lower) in the eleven swing states listed above.  Thus in the states where the election was contested Silver’s simulations were slightly more accurate than a simple average of state polls.  However, the most important contribution of the 538 model in 2012 was that it substantially reduced the systematic underestimate of President Obama’s vote share from 2.0% to 1.1%.

The systematic gap between vote totals and state polls has little to do with sampling variation and more to do with mis-estimation of voter enthusiasm and turnout.   While averages of state polls provide more efficient forecasts than individual polls in a given election there will be systematic errors across states because of the difficulty in forecasting voter turnout and assessing voter enthusiasm.  Based on the past four presidential elections the systematic poll gap favors Democrats in some elections and Republicans in others and is likely to be similar in magnitude to the 2.0% difference observed in 2012.  If the 2.0% systematic gap in poll forecasts had favored Mitt Romney on Tuesday he would have received 266 electoral votes and lost the presidency by less than 5,000 votes in New Hampshire.

Nate Silver simulations are valuable when they reduce the magnitude of the systematic gap between state poll averages and election outcomes as they did in 2012.  His work correctly identified that polls were underestimating President Obama’s support even though the 538 model also contained systematic bias.  This source of forecast error can’t be reduced by taking more polls but can be mitigated somewhat by supplementing poll averages with additional information.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: