Is There a Debt/GDP Threshold at 90 Percent? (Part 2)

Is There a Debt/GDP Threshold at 90 Percent? (Part 2)

As mentioned in my prior post, I have posted an Excel spreadsheet which is an extension of one included in a zip file posted by Herndon, Ash and Pollin (hereafter called HAP). I will use data from that spreadsheet to look at the HAP criticisms of the Reinhart and Rogoff (hereafter called RR) paper described previously.

A good starting point is the now infamous Excel spreadsheet shown in my prior post and repeated below:

After noting the coding error by which 5 rows were excluded, the first question that occurred to me was "where's the beef?". The number of countries on which the 90 percent threshold is based is a mere seven (Belgium having been left out by RR). The HAP critique points out that RR is using only 71 data points (110 after Belgium and 14 other excluded data points are added). Since there were relatively few data points, my first inclination was to try to look at that data so see how it was distributed. The following table shows the 71 data points used by RR, the 25 data points for Belgium, and the other 14 excluded data points (marked by ^):

        REAL GDP GROWTH FOR DEBT OF 90 PERCENT OF GDP OR ABOVE FOR 1946-2009

                                            New                    Aus-
Belgium* Greece   Italy Ireland   Japan Zealand      UK      US  tralia* Canada* Year
------- ------- ------- ------- ------- ------- ------- ------- ------- ------- -----
                                            7.7^   -2.5   -10.9    -3.6^   -1.0^ 1946
   15.2                                    11.9^   -1.3    -0.9     2.5^    4.4^ 1947
                                           -9.9^    2.9     4.4     6.4^    1.8^ 1948
                                           10.8^    3.3    -0.5     6.6^    2.2^ 1949
                                                    3.2             6.9^    7.4^ 1950
                                           -7.6     2.7                          1951
                                                    0.1                          1952
                                                    3.8                          1953
                                                    4.1                          1954
                                                    3.5                          1955
                                                    0.9                          1956
                                                    1.7                          1957
                                                    0.3                          1958
                                                    4.3                          1959
                                                    5.3                          1960
                                                    2.3                          1961
                                                    1.1                          1962
                                                    4.3                          1963
                                                    5.5                          1964
                                                                                 1965
-------------------------(no cases from 1965 to 1982)--------------------------------
                                                                                 1982
                           -0.7                                                  1983
    2.1                     3.2                                                  1984
    1.8                     1.9                                                  1985
    1.9                     0.4                                                  1986
    2.4                     3.6                                                  1987
    4.6                     3.0                                                  1988
    3.6                     5.6                                                  1989
    3.1                                                                          1990
    1.8     3.1                                                                  1991
    1.3     0.7                                                                  1992
   -0.7    -1.6    -0.9                                                          1993
    3.3     2.0     2.2                                                          1994
    4.3     2.1     2.8                                                          1995
    0.9     2.4     1.1                                                          1996
    3.7     3.6     1.9                                                          1997
    1.7     3.4     1.4                                                          1998
    3.4     3.4     1.5            -0.1                                          1999
    3.8     4.5     3.7             2.9                                          2000
    0.8     4.2     1.8             0.2                                          2001
    1.5     3.4                     0.3                                          2002
    1.0     5.6                     1.4                                          2003
    2.8     4.9                     2.7                                          2004
    2.2     2.9                     1.9                                          2005
            4.5                     2.0                                          2006
            4.0                     2.3                                          2007
    1.0     2.9                    -0.7                                          2008
   -3.2    -0.8    -5.1            -5.4                                          2009

* column excluded by RR because of Excel error
^ data excluded by RR

Note that the 1951 data point for New Zealand is the only data point used by RR for that country. The data points from 1946 to 1949 for New Zealand are excluded. Tab A of the aforementioned spreadsheet shows the following numbers for New Zealand from 1946 to 1955:

           DEBT/GDP AND REAL GDP GROWTH FOR NEW ZEALAND: 1946-1955

Year                1946  1947  1948  1949  1950  1951  1952  1953  1954  1955
------------------ ----- ----- ----- ----- ----- ----- ----- ----- ----- -----
Debt/GDP(%)        134.0 120.4 117.2 111.5  87.7  91.8  85.8  79.3  75.3  73.8
Real GDP Growth(%)   7.7^ 11.9^ -9.9^ 10.8^ 14.7  -7.6   4.3   3.4  13.8   1.9

^ data excluded by RR

As can be seen, New Zealand had real GDP growth of 14.7% the year before and 4.3% the year after the -7.6% growth in 1951. Hence, if RR had used a debt threshold of 85% of GDP, the average rate of growth would have been 3.8 percent. On the other hand, if RR had used a debt threshold of 95% of GDP, New Zealand would have had no data points. Both of these calculations assume that RR continues to exclude 1946 through 1949. If these are included, the averages become 4.6 and 5.1 percent, respectively. Only by using a threshold of 90% of GDP and excluding 1946 through 1949 did RR come up with -7.6 percent. A cursory inspection of the data, as I have done here, would have shown the -7.6 figure to be an unrepresentative outlier, requiring some sort of corrective action.

That cursory inspection also turns up the real GDP growth of -10.9% for the U.S. in 1946. This was the year after World War II ended so it's no surprise that real GDP dropped that year. If that year had been excluded, the average real GDP growth for years that the U.S. was above the 90% threshold would have been +1.0 percent instead of -2.0 percent.

One might argue that Belgium's real GDP growth of 15.2% in 1947 was also an outlier. Excluding that year, however, just drops the average real GDP growth for years that Belgium was above the 90% threshold from 2.6% to 2.0%. This is because the effect of an outlier is much greater in a small country sample when the weighting is done per country as was done by RR.

The following table shows the effect on the average real GDP growth for all years above the 90% threshold of various corrections and changes in calculation method:

 AVERAGE REAL GDP GROWTH FOR DEBT OF 90 PERCENT OF GDP OR ABOVE FOR VARIOUS SCENARIOS

                                            New                    Aus-
Belgium* Greece   Italy Ireland   Japan Zealand      UK      US  tralia* Canada*  TOTAL  Scenario
------- ------- ------- ------- ------- ------- ------- ------- ------- ------- -------  --------
            2.9     1.0     2.4     0.7    -7.6     2.4    -2.0                   -0.02  RR (Reinhart and Rogoff)
    2.6     2.9     1.0     2.4     0.7    -7.6     2.4    -2.0                     0.3  + fix Excel Error
    2.6     2.9     1.0     2.4     0.7     2.6     2.4    -2.0                     1.6  + include 1946-1950 for New Zealand
    2.6     2.9     1.0     2.4     0.7     2.6     2.4    -2.0     3.8     3.0     1.9  + include 1946-1950 for Australia & Canada

   64.2    55.3    10.3    17.1     7.6    12.9    45.6    -8.0    18.9    14.8   238.6  Total of all country-years
   25.0    19.0    10.0     7.0    11.0     5.0    19.0     4.0     5.0     5.0   110.0  Count of country-years
------- ------- ------- ------- ------- ------- ------- ------- ------- ------- -------  ----------------------
    2.6     2.9     1.0     2.4     0.7     2.6     2.4    -2.0     3.8     3.0     2.2  Country-year weighting, all data

            2.9     1.0     2.4     0.7             2.4    -2.0                     1.2  RR minus New Zealand
            2.9     1.0     2.4     0.7             2.4     1.0                     1.7  RR minus New Zealand and 1946 for U.S.
    2.6     2.9     1.0     2.4     0.7             2.4                             2.0  RR minus countries with no cases after 1951
    2.0     2.9     1.0     2.4     0.7             2.9                             2.0  RR for 1952-2009 instead of 1946-2009
    2.0     2.9     1.0     2.4     0.7                                             1.8  RR for 1980-2009 instead of 1946-2009

* column excluded by RR because of Excel error
^ data excluded by RR

The first line shows the slightly negative figure calculated by RR (Reinhart and Rogoff). The next 4 results are in response to various corrections suggested by HAP. The second line shows an improvement of 0.3% when the infamous Excel error is fixed. The biggest improvement of 1.3% occurs in the third line when the missing years of 1946 to 1949 are included for New Zealand. Another improvement of 0.3% occurs when the missing years of 1946 to 1950 are included for Australia and Canada. Finally, another improvement of 0.3% occurs if the data is weighted by country-year instead of by country as suggested by HAP. This gives a real GDP growth of 2.2 percent given by HAP in the Abstract on page 1 of their critique.

The final 5 lines show that, even if one insists on weighing the data by country instead of country-year, any reasonable attempt to minimize the effect of outliers will bring the result much closer to HAP's value of 2.2% than RR's value of -0.02%. The first of these shows that simply excluding the outlier of New Zealand will cause an improvement of 1.2 percent. The second shows that another improvement of 0.5% occurs if one excludes the outlier of the U.S. in 1946.

A more consistent way of dealing with the small samples of debt right after World War II would be to exclude all countries with no case after 1951. This would raise the real GDP growth to 2.0 percent, nearly as high as the HAP value of 2.2 percent. The same result would occur if the starting year of the data was moved from 1946 up to 1952. The final line shows that, if the study were to simply focus on the "modern era" after 1980, the result would be a nearly-as-high 1.8 percent. As can be seen, any of the fixes beyond merely fixing the Excel error would bring the calculation much closer to HAP's value of 2.2 percent than RR's value of -0.02 percent.

What is the lesson to be learned from all of this?

As mentioned here and here, the Reinhart and Rogoff paper was not peer-reviewed. However, the latter article makes the following interesting comment:

So the answer is to only accept peer-reviewed work as economic knowledge, right? Nope. That would be a) too limiting, and b) wouldn't advance the epistemological cause as much as you think. Peers have their own sets of biases, particularly as gate keepers.

I do think that peer-review does have a role to play but I can see that it's not the only answer. However, it does seem that we could at least clearly label what is peer-reviewed and what is not. I don't recall the absence of peer-review having been mentioned once during the public debate of the past three years. Only once the Excel error was found did we hear, "Oh yeah, that paper was never peer-reviewed".

There is an additional step that I think would help a great deal. For any paper to be taken seriously, the authors should have to give their sources AND show their work. In their online appendix, RR stated:

We took great pains to provide the data in as accessible form as possible, including especially meticulous source documentation in the spreadsheets, far more than one sees normally posted with journal papers. So we are simply stunned when bloggers and irresponsible commentators say we have not shared our debt data. Open access to our data has been central to our whole project.

I believe that RR is referring to the links to data that they have posted here. However, it seemed that HAP had to go through a great deal of effort to recreate RR's numbers. It was only by requesting the original Excel spreadsheet that they were able to start recreating the numbers. Speaking of the spreadsheet, I was unable to find a copy of the spreadsheet anywhere on the web even now. I manually typed in the numbers that appeared in the one graphics that I saw on the web and added it as tab C in my spreadsheet. I then tried to reproduce the numbers in the spreadsheet using RR's sources but was unable to find all of the data. Fortunately, HAP posted the zip file with their spreadsheet and I was able to replicate the numbers with that.

Why don't economists just post their original spreadsheets? There may be many reasons. Some may be willing to put up with peer-review when required but not want every pointy-headed number-cruncher with too much time on their hands to be going through their work. I suspect that those number-crunchers would catch a number of errors or questionable methodology that peer-review would not catch. In addition, some economists might feel that publishing their spreadsheets would reveal trade secrets and/or help other economists to compete in their area of study. Hence, I think that it's largely up to those who consume the studies to demand that the work be released. Of course, economists would still be free to release studies without peer-review and showing precisely how they arrived at their conclusions. But if those studies were simply ignored or treated as interesting ideas until they can be verified, I suspect that many of those economists would be willing to "show their work".

Note: There is a discussion of this post at this link.

Is There a Debt/GDP Threshold at 90 Percent? (Part 1)

In January of 2010, Harvard economists Carmen M. Reinhart and Kenneth S. Rogoff released an NBER working paper titled "Growth in a Time of Debt". The paper's abstract states:

Our analysis is based on new data on forty-four countries spanning about two hundred years. The dataset incorporates over 3,700 annual observations covering a wide range of political systems, institutions, exchange rate arrangements, and historic circumstances. Our main findings are: First, the relationship between government debt and real GDP growth is weak for debt/GDP ratios below a threshold of 90 percent of GDP. Above 90 percent, median growth rates fall by one percent, and average growth falls considerably more.

This 90 percent threshold was cited by a number of public figures. For example, Paul Ryan stated the following on page 80 of the House Fiscal Year 2013 Budget Resolution titled "The Path to Prosperity: A Blueprint for American Renewal".

Even if high debt did not cause a crisis, the nation would be in for a long and grinding period of economic decline. A well-known study completed by economists Ken Rogoff and Carmen Reinhart confirms this common-sense conclusion. The study found conclusive empirical evidence that gross debt (meaning all debt that a government owes, including debt held in government trust funds) exceeding 90 percent of the economy has a significant negative effect on economic growth.

As another example, a Washington Post editorial stated the following:

The CBPP analysis assumes steady economic growth and no war. If that’s even slightly off, debt-to-GDP could keep rising — and stick dangerously near the 90 percent mark that economists regard as a threat to sustainable economic growth.

A number of other positive references to the 90 percent threshold are mentioned in this article. However, there were some criticisms of this 90 percent threshold as in this article from the Economic Policy Institute.

In any case, a very interesting thing happened in April of 2013, over three years after the Reinhart and Rogoff paper was released. The event is described in Planet Money podcast titled "Episode 452: How Much Should We Trust Economics?". The following is from a summary of the show:

Reinhart and Rogoff looked at what had happened in many different countries over many years. And they found a what looked like a clear debt threshold: 90 percent. Average growth was much, much slower in countries with debt-to-gdp ratios over 90 percent.

The paper got a lot of coverage in the press. Politicians cited it in the U.S. and Europe.

Then, this week, a 28-year-old grad student and his professors published a startling finding: Reinhart and Rogoff [hereafter called RR] had made a simple Excel error in one part of their study. The authors of the new critique also questioned other elements of the study and argued that, in fact, there is no debt threshold.

The new critique was published by Herndon, Ash and Pollin (hereafter called HAP) and is titled "Does High Public Debt Consistently Stifle Economic Growth? A Critique of Reinhart and Rogoff". The error that was most widely reported was the simple Excel error, shown in the following image:

As seen, the formula for cell L51 is given as AVERAGE(L30:L44), the cells within the blue box. In fact, the formula should have been AVERAGE(L30:L49). This mistake served to omit the data for Denmark, Canada, Belgium, Austria, and Australia. In the case of column L, only the Belgium omission mattered since the other four omitted countries did not have data for that column. However, the same mistake was made in columns I, J, and K which did contain data for those other four countries.

However, the HAP critique pointed out two other issues. One was data exclusion which it described as follows:

More significant are RR's data exclusions with three other countries: Australia (1946-1950), New Zealand (1946-1949), and Canada (1946-1950). The exclusions for New Zealand are of particular significance. This is because all four of the excluded years were in the highest, 90 percent and above, public debt/GDP category. Real GDP growth rates in those years were 7.7, 11.9, -9.9, and 10.8 percent. After the exclusion of these years, New Zealand contributes only one year to the highest public debt/GDP category, 1951, with a real GDP growth rate of -7.6 percent. The exclusion of the missing years is alone responsible for a reduction of -0.3 percentage points of estimated real GDP growth in the highest public debt/GDP category. Further, RR's unconventional weighting method that we describe below amplifies the effect of the exclusion of years for New Zealand so that it has a very large effect on the RR results.

The other was the unconventional weighting method mentioned at the end of the above excerpt. As an example, Great Britain had 19 postwar years (1946 to 1964) when it's debt to GDP was above 90 percent and the average growth in real GDP over this period was 2.4 percent. According to RR, however, New Zealand had just one year above 90 percent (1951) and the growth in real GDP for that year was -7.6 percent. RR give the -7.6 percent the same weight as Great Britain's 2.4 percent despite the fact that the former is for one year and the latter is for 19 years. HAP, on the other hand, gives the latter 19 times as much weight as the former.

In an April 25th New York Times editorial, RR responded to these criticisms, saying the following:

Last week, three economists at the University of Massachusetts, Amherst, released a paper criticizing our findings. They correctly identified a spreadsheet coding error that led us to miscalculate the growth rates of highly indebted countries since World War II. But they also accused us of “serious errors” stemming from “selective exclusion” of relevant data and “unconventional weighting” of statistics — charges that we vehemently dispute. (In an online-only appendix accompanying this essay, we explain the methodological and technical issues that are in dispute.)

In an April 29th New York Times editorial, Pollin and Ash responded, saying:

(Ms. Reinhart and Mr. Rogoff have substantial disagreements with us about the proper selection and weighting of data. They elaborated on these points in their Op-Ed appendix. We have presented all our data, calculations and methodological arguments on the Web site of the Political Economy Research Institute at the University of Massachusetts, Amherst, where we teach.)

The given Web site points to a page for the HAP critique which points to a zip file containing "Data and code files upon which the results are based". This zip file contains an Excel spreadsheet named RR.xls which contains all of the data on which the HAP critique is based, including the original RR data. I added three tabs labeled A, B, and C to the beginning of this spreadsheet on which I used this data to examine the HAP criticisms. I have posted that spreadsheet at http://www.econdataus.com/RR2.xls and will refer to it in the following post.

Is There a Debt/GDP Threshold at 90 Percent? (Part 2)