To Punt or not to Punt: Policy Advice based on Observational Data

Rocky Long, second year head football coach of the San Diego State Aztecs, is facing a problem that is similar, in some respects, to what voters face.  Coach Long may choose to never punt the football once the Aztecs cross midfield, regardless of distance on fourth downs.  He is listening to advice suggesting that the strategies of many previous coaches were wrong.  Coach Long says “We had one of our professors in our business school … go over a system we are thinking about using.  We’ll have a chart come game time that will determine what we do in different situations.”

Voters must decide whether the policies of another Obama Administration will help turn the economy around and contribute to more job creation, or if the Romney campaign proposals will be more successful in achieving these goals.  Football and economic policy advice are both derived from observational, not experimental, data.  It is difficult to know what might happen if alternative policies were to be enacted.  Analysts look to history, in similar circumstances with similar policy options, and hope this provides useful guidance.

Coach Long’s decision apparently follows the strategy of high school coach Kevin Kelley of Pulaski Academy in Little Rock, Arkansas who has been remarkably successful.  But college football is not high school football.  The strategy also appears similar to advice given by Berkeley macroeconomist David Romer, who several years ago found that NFL teams kick surprisingly often on fourth down.

Romer concluded that teams are not pursuing strategies that would maximize their chance of winning the game.  Romer may be correct, but we should be cautious because his study is based purely on observational data.  It is possible that the real world problem is more complicated than the model used to analyze the data.  Whenever economics (or other social science) professors explain that agents motivated by self-interest are making choices not in their self-interest, the professors may have mis-specified their model or may misunderstand the real-world problem that people are attempting to solve.

Romer’s paper is interesting, well written, and well executed.  All of the criticisms raised here are ones that Romer acknowledges, but they aren’t enough to sway his opinion and policy advice.  The data Romer analyzed indicated that NFL teams rarely try for a first down on fourth down.  The primary question is why?  Romer’s explanation is myopia.  An alternative explanation is that a failed fourth down attempt will shift momentum in the game.

The key problem with observational data is that it is difficult to calculate the expected outcome from counterfactual decisions and policies.  Romer argues that teams are not optimizing because he believes they would be successful fairly often on fourth down and two yards to go.  He concludes this despite rarely seeing teams attempting this play.  So how does Romer “guesstimate” the likelihood of success on fourth down and two?  He looks at outcomes of third down plays with two yard to go in the first quarter of games.  He uses first quarter plays because once enough game time has elapsed and the score is uneven, both teams will adjust their game strategies.  He assumes that third down strategies and outcomes are very similar to what would happen on fourth down plays (if they actually were to occur) because he doesn’t have enough data on fourth down plays.  Even if he observed more fourth down plays, they would not be a random or representative sample of teams and/or game situations.  Romer’s assumptions are made out of necessity, not because they are realistic or accurate.

It is also important to note that Romer’s empirical model does not allow for momentum.  The value of having possession of the ball with first down and ten yards to go on a given yard line (say midfield) in his empirical model is completely independent of how one reached that position on the field.  In the language of dynamic programming, the state variables for the optimization problem include down and yardage, but not the sequence of plays leading up to that point.  He imposes this assumption, as good applied economists often do, to make the problem tractable.  He recognizes that momentum could matter and makes some supplemental calculations to show that teams don’t perform much differently immediately after very bad plays (fumbles, interceptions, blocked kicks, and long kickoff and punt returns by the opponent) and just after very good plays (touchdowns) than they do after typical plays.  This is, at best, a half-hearted attempt to determine whether there is momentum in NFL games.  The model simply doesn’t take the idea of momentum shifts seriously, but avoiding these shifts seems to be a key reason why coaches kick field goals rather than go for it on fourth down.

Rocky Long, at San Diego State, may follow the advice of economics and business school professors and forsake the punt.  But he should do so understanding how the policy advice was determined.  It is extremely difficult for economics professors to evaluate counterfactual policies- whether it is a forecast of what will happen if teams ran and passed on fourth down or a “guesstimate” of the 2012 unemployment rate had there been no stimulus package in 2009. 

Asking economics professors, the Congressional Budget Office or other forecasters to evaluate alternative policies and predict what might happen over the next decade also has limited value.  Many of these same professionals either didn’t forecast the recession or underestimated its severity.  Government economists and advisors didn’t know how deep the downturn was until a year or two later when the data came in.  Mis-estimation of the recession in the midst of the downturn is the explanation given for the woefully inaccurate prediction that the stimulus would keep the unemployment rate below 8%.

It is unwise to rely too heavily on economists as authorities on counterfactual policies.  Economists can’t easily determine what would have happened had there been no stimulus, or how the economy might perform if taxpayers earning more than $200,000 were to face higher marginal tax rates.  In fact they struggle to measure output and employment in real-time.  Predictions about hypothetical economic policies are as fraught with error as predictions about fourth down decisions that have rarely been tried in the past.

Alabama and Stanford Provided the Most Valuable Talent to the NFL in the 2012 Draft

Many NFL teams use the relative valuation of draft selections established by the Dallas Cowboys to evaluate possible trades for “moving up” or “moving down” in the draft.  According to the Cowboys’ chart, the top pick in the draft is worth more than 5 times as much as the first pick in the second round (the 33rd pick) and ten times as much as the first pick in the 3rd round (the 65th selection).  The value of players declines exponentially because there are more substitutes for less skilled players and the NFL imposes a minimum salary schedule.

These values can be used to determine which positions are expected to provide the NFL with the most valuable talent in the just completed 2012 Draft.  The following chart shows that quarterbacks, wide receivers, defensive ends and cornerbacks, despite representing less than one third of a team’s positions, are expected to provide almost half of the value in the draft.  This is a clear indication that the NFL has become a pass first league.

Position Share of Draft Value Share of Players Selected



Wide Receiver



Defensive End






Defensive Tackle



Offensive Tackle



Running Back



Offensive Guard



Outside Linebacker






Inside Linebacker



Tight End



Offensive Center



All Other Positions



The chart also shows that quarterbacks are selected much earlier in the draft (their share of draft value is more than three times their share of players selected) than other positions.  Safeties, inside linebackers, tight ends and centers are selected later, on average, than other positions.

The draft values used by NFL teams also can determine which college teams provided the NFL with the most valuable talent in expected value terms) in the 2012 draft.  Alabama had the most players selected (8 out of 253) and are expected to be the source of almost 10% of the value of the 2012 draft.

Position Share of Draft Value Share of Players Selected












South Carolina






Oklahoma State






Notre Dame



Boise State



College football players were drafted out of 105 different colleges and universities.  The ten schools listed above accounted for 19% of the players and 51% of the value of the talent in the draft.  Only twenty schools accounted for 32.4% of the players and 71.5% of the value in the draft.

The pattern of players selected in the draft emphasizes the high value that NFL General Managers place on both passing offense and defense.  It is difficult to forecast the value of professional football players based on their performance in college and the NFL combine.  Many of the quarterbacks selected in the 2012 draft will underperform relative to expectations, while others will outperform quarterbacks selected ahead of them.  An NFL  team, such as the Miami Dolphins, is willing to take a big chance in the draft on a quarterback because the right player can elevate a team’s performance and profitability for a decade.

Using the Draft to Determine which Universities Provide the Most Talent to the NFL

The NFL Draft, which begins tonight, is televised monopsony (the market power obtained by being the single buyer of a commodity).  The best college football players are selected by a team that will own their exclusive bargaining rights.  If an agreement is not reached the player can’t return to play college football or play for another NFL team that year.  The draft provides an advantage to teams in negotiations with players who are expected to be superstars.  The draft also reveals the expected relative value of players entering the NFL.  A draft pick is a valuable traded asset because it confers monopsony power to the team that owns the pick.  Higher picks are worth more because they represent the rights to exclusively bargain with the very best players.  If players could negotiate with any team in the NFL, draft picks would be worthless.

The Dallas Cowboys established a relative valuation of draft selections, in order to better evaluate possible trades for “moving up” or “moving down” in the draft.  According to the Cowboys’ chart , the top pick in the draft is worth more than five times as much as the first pick in the second round (the 33rd pick) and ten times as much as the first pick in the 3rd round (the 65th selection).  The value of players declines exponentially because there are more substitutes for less skilled players and the NFL imposes a minimum salary schedule.  The Indianapolis Colts and Washington Redskins expect to earn substantial profit from future superstars Oliver Luck and Robert Griffin III, who will be paid much less than they would earn as free agents, because of the salary cap.  It is this expected future profit from top selections that makes high draft picks so valuable.

The draft valuations established by the Cowboys can be used to determine which universities have supplied the NFL with the most valuable talent since the current seven round draft system began in 1994.  In the charts below the top pick is normalized to have a value of 100.  Each college team/year is evaluated by the sum of the value of the players drafted in that year and the next three years.  For example, the value of a team’s players in 2000 equals the value of players drafted from that university between April 2000 and April 2003.  In other words the player quality indexed is summed across four consecutive recruiting /draft classes.

Miami, USC, Ohio State and Florida State have provided the most valuable talent to the NFL since 1994.  The most valuable group of players on campus at the same time was the University of Miami football team in the fall of 2000 (with the first players drafted in April 2001).  Miami had a total player quality index in excess of 650.

There are substantial differences in the sources of talent by year and major conference.  The following chart shows the parity in talent in the Southeastern Conference, where five different teams have had the most valuable talent since 1994.

Florida State and the University of Miami had the most valuable football talent in the Atlantic Coast Conference until 2005.  Since then there has been parity in talent among the top four teams.

Texas and Oklahoma have been the dominant teams in the Big 12 since 1994.  The quality of the football talent at Oklahoma surpassed Texas in 2007.

The PAC-12 had the most unequal distribution of football talent of any major conference in the past 18 years.  The University of Southern California had dominant player talent relative to the other schools in their conference.

Ohio State has been the dominant team in the Big Ten since 1994, followed by Penn State.  The talent at Notre Dame is far below the talent at other major football powers in the Big Ten.

Major college football programs have provided the NFL with valuable talent for decades.  The universities illustrated in the charts above serve as the “minor league” for professional football.  Many of the players drafted from these universities earned substantial salaries from the NFL.  Many of their teammates, however, helped their universities generate millions of dollars in football revenue, but never played football professionally.

%d bloggers like this: