Reinhart and Rogoff's Second Response To Critique Of Their Research

At 2:00 a.m. on Wednesday, Carmen Reinhart and Kenneth Rogoff of Harvard emailed a second, fuller response to the new paper finding several errors in their research on debt and growth rates.

The full response is printed below. The gist of it is that, while they admit they made a mistake with their Excel spreadsheet, they defend their debt weighting and say they have corrected for data omissions in later research. They say the new critique doesn't change their ultimate finding: That higher debt is associated with slower growth.

Response to Herndon, Ash and Pollin by Carmen Reinhart and Kenneth Rogoff, April 17, 2012

We are grateful to Herndon et al. for the careful attention to our original Growth in a Time of Debt AER paper and for pointing out an important correction to Figure 2 of that paper. It is sobering that such an error slipped into one of our papers despite our best efforts to be consistently careful. We will redouble our efforts to avoid such errors in the future. We do not, however, believe this regrettable slip affects in any significant way the central message of the paper or that in our subsequent work. But first let us consider the specific points raised by Herndon Ash and Pollin (HAP) in their comment that we were sent yesterday.

The authors point out that there are three problems with our 1945-2009 averages and the paper itself: (i) a coding error that causes the first five countries in the alphabet to be omitted in forming averages for the 1946-2009 period in one figure, (ii) "selective exclusion" of 1946-1950 for New Zealand, and (iii) "unconventional weighting of summary statistics"--the implication being that these omissions are intentionally used to bias the results. They argue that the interaction of three problems magnifies their effects and leads to completely different conclusions, especially when they choose a different weighting scheme.

On the first point, we reiterate that Herndon, Ash and Pollin accurately point out the coding error that omits several countries from the averages in figure 2. Full stop. HAP are on point. The authors show our accidental omission has a fairly marginal effect on the 0-90% buckets in figure 2. However, it leads to a notable change in the average growth rate for the over 90% debt group. The median growth rate we report is the right order of magnitude.

Our interpretation of the errant data point in figure 2 was fortunately tempered somewhat by the parallel weight given to the median GDP growth rate for the various levels of debt in our discussion, an issue HAP selectively ignore. To quote from our opening paragraph: "median growth rates for countries with public debt over roughly 90 percent of GDP are about one percent lower than otherwise; There is also Table 1 which immediately follows figure 2, and does not have the same issues. Table 1 gives all the individual country estimates for all the buckets and over a much longer time period than figure 2, and of course figures importantly in our analysis. We are fortunate that we chose to present our results in several different ways, including means, medians, and individual country averages, in no small part as standard robustness checks. Nevertheless, the mistake in figure 2 resulting from the coding error is a significant lapse on our part.

HAP go on to note some other missing debt data points, which they describe as "selective omissions". This charge, which permeates through their paper, is one we object to in the strongest terms. The "gaps" are explained by the fact there were still gaps in our public data debt set at the time of this paper, a data set no one else had ever been able to construct before and which we now have filled in much more completely. Many readers of our work have followed this evolution on our data website. For example, at the time the 2010 AER paper was written, there were gaps in the French data for the 1970s that we only filled in later. Other data, including data for New Zealand for the years around WWII, had just been incorporated and we had not vetted the comparability and quality data with data for the more recent period. In effect, HAP only knew we had these data as we sent them the file we had used at that time.

We have no issues with to Herndon, Ash and Pollin for bringing attention to any data question regarding our work. In this regard, we note that we have long since fully integrated the New Zealand data back to the early 1800s, once we were able to process it. Every major high debt episode for advanced countries since 1800 and the underlying data is included in our 2012
Journal of Economic Perspectives paper co-authored with Vincent Reinhart.

But surely the authors do not mean to insinuate that we manipulated the data to exaggerate our results. To what purpose would we "manipulate" the average growth rate for debt above 90% and show an average of -0.1% when in the same AER paper we report the median for 1946-2009 at 1.6%, and over the longer sample 1790-2009, we report and average of 1.7% and a median of 1.9%? (see table below) Why, for that matter, would we provide all the data that we have gathered and used in our research over the years documenting in detail multiple sources to the public domain?

This brings us to the core conceptual issue, which Herndon, Ash and Pollin argue greatly biases our results. They argue that we use an "unconventional weighting of summary statistics." In particular, for each bucket, we take average growth rates for each country and then take an average of the result. This seems perfectly natural to us, and hardly unconventional. We do not want to excessively weight Greece, for example, which has debt over 90% for 19 years in the 1946-2009 sample. The post-war Advanced Economy experience would quickly reduce to the experiences of Greece and Japan. Our approach has been followed in many other settings where one does not want to overly weight a small number of countries that may have their own peculiarities. Our approach is quite clear from table 1, which also gives the averages for each individual country.

Our 2012 Journal of Economic Perspectives paper, based a much longer time period (1800-2011 versus the 1946-2009 Herndon et al focus on), gives episode-by-episode data for each country, including growth rates and number of years, so the results are quite transparent. The problems with weighting long episodes much more heavily than short episodes, as Herndon et al. suggest, become much more apparent in the longer time series (an earlier version of which was also used in Table 1 of our original AER paper) As we noted in our initial comment yesterday upon just receiving the paper, our JEP paper anticipates most of the aggregation debate and diffuses it by using a case-study approach. That is where this literature is now expanding.

So do where does this leave matters on debt and growth? Do Herndon et al. get dramatically different results on the relatively short post war sample they focus on? Not really. They, too, find lower growth associated with periods when debt is over 90% (they find 0-30 debt/GDP , 4.2% growth; 30-60, 3.1 %; 60-90, 3.2%,; over 90, 2.2%. Put differently, growth at high debt levels is a little more than half of the growth rate at the lowest levels of debt. They ignore the fact that these results are close to what we get in our Table 1 of our AER paper they critique, and not far from the median results in Figure 2 despite its coding error. And they are not very different from what we report in our 2012 Journal of Economic Perspectives paper with Vincent Reinhart--where the average is 2.4% for high debt versus 3.5% for below 90% . [Table omitted, but you can see it here .]

There is also the question of whether these growth effects can be economically large. Here it is very misleading to think of 1% growth differences without recognizing that the typical high debt episode lasts well over a decade (23 years on average in the full sample.)

It is utterly misleading to speak of a 1% growth differential that lasts 10-25 years as small. If a country grows at 1% below trend for 23 years, output will be roughly 25% below trend at the end of the period, with massive cumulative effects.

Looking to the reaction to this comment in blogosphere, we note that this is not the first time our academic work is seen pandering to a political view. What is quite remarkable is that this claim has spanned polar opposites! This time, we are charged with misconstruing analysis to support austerity. Only a few months ago, our findings on slow recoveries from financial crises was accused as providing a rationale for the deep recession and weak economy the Obama administration has faced since 2007.

Herndon, Ash and Pollin have written a useful paper, finding a significant mistake in one of our figures, and helped reconcile why one result is out of line with all the other results in our original paper as well as ones presented in our later research, not to mention those they present in their helpful comment. Clearly more research is needed on debt and growth and we welcome all efforts, it is very exciting area. We now have debt data for a larger number of countries than the original sample and long time periods that allows this research to press forward.

Correction: An earlier version of this story incorrectly identified Carmen Reinhart as being with the University of Maryland. She was at the time she wrote Growth In A Time Of Debt, but is now with Harvard.