As mentioned in my prior post, I have posted an Excel spreadsheet which is an extension of one included in a zip file posted by Herndon, Ash and Pollin (hereafter called HAP). I will use data from that spreadsheet to look at the HAP criticisms of the Reinhart and Rogoff (hereafter called RR) paper described previously.
A good starting point is the now infamous Excel spreadsheet shown in my prior post and repeated below:
After noting the coding error by which 5 rows were excluded, the first question that occurred to me was "where's the beef?". The number of countries on which the 90 percent threshold is based is a mere seven (Belgium having been left out by RR). The HAP critique points out that RR is using only 71 data points (110 after Belgium and 14 other excluded data points are added). Since there were relatively few data points, my first inclination was to try to look at that data so see how it was distributed. The following table shows the 71 data points used by RR, the 25 data points for Belgium, and the other 14 excluded data points (marked by ^):
REAL GDP GROWTH FOR DEBT OF 90 PERCENT OF GDP OR ABOVE FOR 1946-2009
New Aus-
Belgium* Greece Italy Ireland Japan Zealand UK US tralia* Canada* Year
------- ------- ------- ------- ------- ------- ------- ------- ------- ------- -----
7.7^ -2.5 -10.9 -3.6^ -1.0^ 1946
15.2 11.9^ -1.3 -0.9 2.5^ 4.4^ 1947
-9.9^ 2.9 4.4 6.4^ 1.8^ 1948
10.8^ 3.3 -0.5 6.6^ 2.2^ 1949
3.2 6.9^ 7.4^ 1950
-7.6 2.7 1951
0.1 1952
3.8 1953
4.1 1954
3.5 1955
0.9 1956
1.7 1957
0.3 1958
4.3 1959
5.3 1960
2.3 1961
1.1 1962
4.3 1963
5.5 1964
1965
-------------------------(no cases from 1965 to 1982)--------------------------------
1982
-0.7 1983
2.1 3.2 1984
1.8 1.9 1985
1.9 0.4 1986
2.4 3.6 1987
4.6 3.0 1988
3.6 5.6 1989
3.1 1990
1.8 3.1 1991
1.3 0.7 1992
-0.7 -1.6 -0.9 1993
3.3 2.0 2.2 1994
4.3 2.1 2.8 1995
0.9 2.4 1.1 1996
3.7 3.6 1.9 1997
1.7 3.4 1.4 1998
3.4 3.4 1.5 -0.1 1999
3.8 4.5 3.7 2.9 2000
0.8 4.2 1.8 0.2 2001
1.5 3.4 0.3 2002
1.0 5.6 1.4 2003
2.8 4.9 2.7 2004
2.2 2.9 1.9 2005
4.5 2.0 2006
4.0 2.3 2007
1.0 2.9 -0.7 2008
-3.2 -0.8 -5.1 -5.4 2009
* column excluded by RR because of Excel error
^ data excluded by RR
Note that the 1951 data point for New Zealand is the only data point used by RR for that country. The data points from 1946 to 1949 for New Zealand are excluded. Tab A of the aforementioned spreadsheet shows the following numbers for New Zealand from 1946 to 1955:
DEBT/GDP AND REAL GDP GROWTH FOR NEW ZEALAND: 1946-1955
Year 1946 1947 1948 1949 1950 1951 1952 1953 1954 1955
------------------ ----- ----- ----- ----- ----- ----- ----- ----- ----- -----
Debt/GDP(%) 134.0 120.4 117.2 111.5 87.7 91.8 85.8 79.3 75.3 73.8
Real GDP Growth(%) 7.7^ 11.9^ -9.9^ 10.8^ 14.7 -7.6 4.3 3.4 13.8 1.9
^ data excluded by RR
As can be seen, New Zealand had real GDP growth of 14.7% the year before and 4.3% the year after the -7.6% growth in 1951. Hence, if RR had used a debt threshold of 85% of GDP, the average rate of growth would have been 3.8 percent. On the other hand, if RR had used a debt threshold of 95% of GDP, New Zealand would have had no data points. Both of these calculations assume that RR continues to exclude 1946 through 1949. If these are included, the averages become 4.6 and 5.1 percent, respectively. Only by using a threshold of 90% of GDP and excluding 1946 through 1949 did RR come up with -7.6 percent. A cursory inspection of the data, as I have done here, would have shown the -7.6 figure to be an unrepresentative outlier, requiring some sort of corrective action.
That cursory inspection also turns up the real GDP growth of -10.9% for the U.S. in 1946. This was the year after World War II ended so it's no surprise that real GDP dropped that year. If that year had been excluded, the average real GDP growth for years that the U.S. was above the 90% threshold would have been +1.0 percent instead of -2.0 percent.
One might argue that Belgium's real GDP growth of 15.2% in 1947 was also an outlier. Excluding that year, however, just drops the average real GDP growth for years that Belgium was above the 90% threshold from 2.6% to 2.0%. This is because the effect of an outlier is much greater in a small country sample when the weighting is done per country as was done by RR.
The following table shows the effect on the average real GDP growth for all years above the 90% threshold of various corrections and changes in calculation method:
AVERAGE REAL GDP GROWTH FOR DEBT OF 90 PERCENT OF GDP OR ABOVE FOR VARIOUS SCENARIOS
New Aus-
Belgium* Greece Italy Ireland Japan Zealand UK US tralia* Canada* TOTAL Scenario
------- ------- ------- ------- ------- ------- ------- ------- ------- ------- ------- --------
2.9 1.0 2.4 0.7 -7.6 2.4 -2.0 -0.02 RR (Reinhart and Rogoff)
2.6 2.9 1.0 2.4 0.7 -7.6 2.4 -2.0 0.3 + fix Excel Error
2.6 2.9 1.0 2.4 0.7 2.6 2.4 -2.0 1.6 + include 1946-1950 for New Zealand
2.6 2.9 1.0 2.4 0.7 2.6 2.4 -2.0 3.8 3.0 1.9 + include 1946-1950 for Australia & Canada
64.2 55.3 10.3 17.1 7.6 12.9 45.6 -8.0 18.9 14.8 238.6 Total of all country-years
25.0 19.0 10.0 7.0 11.0 5.0 19.0 4.0 5.0 5.0 110.0 Count of country-years
------- ------- ------- ------- ------- ------- ------- ------- ------- ------- ------- ----------------------
2.6 2.9 1.0 2.4 0.7 2.6 2.4 -2.0 3.8 3.0 2.2 Country-year weighting, all data
2.9 1.0 2.4 0.7 2.4 -2.0 1.2 RR minus New Zealand
2.9 1.0 2.4 0.7 2.4 1.0 1.7 RR minus New Zealand and 1946 for U.S.
2.6 2.9 1.0 2.4 0.7 2.4 2.0 RR minus countries with no cases after 1951
2.0 2.9 1.0 2.4 0.7 2.9 2.0 RR for 1952-2009 instead of 1946-2009
2.0 2.9 1.0 2.4 0.7 1.8 RR for 1980-2009 instead of 1946-2009
* column excluded by RR because of Excel error
^ data excluded by RR
The first line shows the slightly negative figure calculated by RR (Reinhart and Rogoff). The next 4 results are in response to various corrections suggested by HAP. The second line shows an improvement of 0.3% when the infamous Excel error is fixed. The biggest improvement of 1.3% occurs in the third line when the missing years of 1946 to 1949 are included for New Zealand. Another improvement of 0.3% occurs when the missing years of 1946 to 1950 are included for Australia and Canada. Finally, another improvement of 0.3% occurs if the data is weighted by country-year instead of by country as suggested by HAP. This gives a real GDP growth of 2.2 percent given by HAP in the Abstract on page 1 of their critique.
The final 5 lines show that, even if one insists on weighing the data by country instead of country-year, any reasonable attempt to minimize the effect of outliers will bring the result much closer to HAP's value of 2.2% than RR's value of -0.02%. The first of these shows that simply excluding the outlier of New Zealand will cause an improvement of 1.2 percent. The second shows that another improvement of 0.5% occurs if one excludes the outlier of the U.S. in 1946.
A more consistent way of dealing with the small samples of debt right after World War II would be to exclude all countries with no case after 1951. This would raise the real GDP growth to 2.0 percent, nearly as high as the HAP value of 2.2 percent. The same result would occur if the starting year of the data was moved from 1946 up to 1952. The final line shows that, if the study were to simply focus on the "modern era" after 1980, the result would be a nearly-as-high 1.8 percent. As can be seen, any of the fixes beyond merely fixing the Excel error would bring the calculation much closer to HAP's value of 2.2 percent than RR's value of -0.02 percent.
What is the lesson to be learned from all of this?
As mentioned here and here, the Reinhart and Rogoff paper was not peer-reviewed. However, the latter article makes the following interesting comment:
So the answer is to only accept peer-reviewed work as economic knowledge, right? Nope. That would be a) too limiting, and b) wouldn't advance the epistemological cause as much as you think. Peers have their own sets of biases, particularly as gate keepers.
I do think that peer-review does have a role to play but I can see that it's not the only answer. However, it does seem that we could at least clearly label what is peer-reviewed and what is not. I don't recall the absence of peer-review having been mentioned once during the public debate of the past three years. Only once the Excel error was found did we hear, "Oh yeah, that paper was never peer-reviewed".
There is an additional step that I think would help a great deal. For any paper to be taken seriously, the authors should have to give their sources AND show their work. In their online appendix, RR stated:
We took great pains to provide the data in as accessible form as possible, including especially meticulous source documentation in the spreadsheets, far more than one sees normally posted with journal papers. So we are simply stunned when bloggers and irresponsible commentators say we have not shared our debt data. Open access to our data has been central to our whole project.
I believe that RR is referring to the links to data that they have posted here. However, it seemed that HAP had to go through a great deal of effort to recreate RR's numbers. It was only by requesting the original Excel spreadsheet that they were able to start recreating the numbers. Speaking of the spreadsheet, I was unable to find a copy of the spreadsheet anywhere on the web even now. I manually typed in the numbers that appeared in the one graphics that I saw on the web and added it as tab C in my spreadsheet. I then tried to reproduce the numbers in the spreadsheet using RR's sources but was unable to find all of the data. Fortunately, HAP posted the zip file with their spreadsheet and I was able to replicate the numbers with that.
Why don't economists just post their original spreadsheets? There may be many reasons. Some may be willing to put up with peer-review when required but not want every pointy-headed number-cruncher with too much time on their hands to be going through their work. I suspect that those number-crunchers would catch a number of errors or questionable methodology that peer-review would not catch. In addition, some economists might feel that publishing their spreadsheets would reveal trade secrets and/or help other economists to compete in their area of study. Hence, I think that it's largely up to those who consume the studies to demand that the work be released. Of course, economists would still be free to release studies without peer-review and showing precisely how they arrived at their conclusions. But if those studies were simply ignored or treated as interesting ideas until they can be verified, I suspect that many of those economists would be willing to "show their work".
Note: There is a discussion of this post at this link.
No comments:
Post a Comment