Structural Breaks, Biased Estimations, and Forecast Errors in a GDP Series of Canada versus the United States.
IntroductionIn a joint attempt to obtain a high level of comparability in collecting business statistics for Canada, Mexico and the United States (U.S.), the Department of Commerce, Bureau of Economic Analysis (BE A) switched the reporting of gross domestic product (GDP) and other national accounts from the Standard Industrial Classification (SIC) system to the North American Industry Classification System (NAICS) in 1997. The NAICS allows for the identification of 1170 industries compared to the 1004 found in the SIC system. A detailed comparison of the SIC and the NAICS can be found in Issue Papers 1 through 6 of the Economic Classification Policy Committee (1993a, 1993b, 1993c, 1993d, 1993e, and 1993f). We suspected the change resulted in a structural break in the GDP series of the U.S. and Canada in 1997. A structural break implies that parameter values governing the data-generating process have changed, which has an important policy implication in macroeconomics modelling. For example, when researchers employ GDP time series in their macroeconomic models, the unidentified structural break may result in previously unsuspected problems such as inaccuracy of forecasts. (1)
A vital question remains: What would the forecast errors be if a researcher did not include the 1997 structural break in the GDP time series analysis? This study attempts to address this question by using a GDP series from Canada and the United States. Researchers working with the GDP measure in academia, industry, and in government as policy makers may benefit from this knowledge because forecasts are likely to be relatively more accurate when the models used in the data analysis are corrected for any possible structural break.
Furthermore, it is possible that unidentified in-sample breaks may result in biased estimates of parameters adversely affecting the model's out-of-sample forecasting performance (Gujarati 2009). To investigate this possibility, we analyzed both the poor performance of a forecast and any biased estimation in the presence of 1997 structural breaks in the U.S. and Canadian GDP series. Two alternative stability diagnostic criteria were used to confirm the structural break in the data. Prediction models were fit with and without break to explore any possible errors in the model specifications and in the forecasts. Overall, these exercises suggest that predictions from the break indicator models perform better than those models that ignore breaks.
Confirmation of a Structural Break
Biased estimation and forecast errors are two possible negative consequences if the 1997 structural break in the GDP series is not properly taken into account. When models are not corrected for structural breaks, the estimated parameters will be biased and unstable because an important variable has been omitted from the model. The omitted variable will result in a specification error (Gujarati 2009). Instability of parameters and weak power of prediction (larger forecasting errors) are two other problems that may arise when models are not corrected for structural breaks. One can apply stability criteria to assess the instability of coefficients and test for constancy, stability and robustness of the estimated relationship in a regression model. (2)
To determine a possible breakpoint in 1997 for the U.S. during the time period 1973-2014, Luitel and Mahar (2015) ran a regression of GDP in its natural logarithmic form versus time trend and applied a Chow test of structural break. In the current study, to confirm a break in the structure of the GDP series for Canada for the same time period (1973-2014), the 1997 breakpoint was tested in the current study using two alternative methods: a log likelihood ratio and Wald statistics. Both statistics indicated that the coefficients were not stable over time (Online Supplemental Appendix Tables 1, 2, 3 and 4). Therefore, the null hypothesis of no-break at the 1997 breakpoint was rejected. Table 1 reports the results for Canada and for the U.S. as a point of comparison. These results confirm the existence of a break in 1997 due to the switch in the data reporting system from the SIC system to the NAIC system of national accounts in North America.
Methodology and Data
To compare models with and without correction for structural break and to analyze forecasting errors in the presence of a structural break, three forecasting models were used that include a lagged dependent variable. The first model has a no-break indicator as follows:
Ln([GDP.sub.t]) = a + b Ln([GDP.sub.t-1]) + [u.sub.t] (1)
where GDP, is U.S. or Canadian GDP, and a and b are the regression parameters to be estimated. It is widely accepted in the econometric literature that indicator variables need to be embedded in the model to account for the effects of potential breaks (Harris and Sollis 2003). To account for this issue, the second regression model is as follows:
Ln([GDP.sub.t]) =a + b Ln([GDP.sub.t-1]) + [cD.sub.1l] + [u.sub.t] (2)
where [D.sub.lt] is the breakpoint indicator defined as [mathematical expression not reproducible]. The timing of the structural change is known and for this reason the dummy variable [D.sub.lt] was introduced to correct Model (1) that ignores breaks. In the third model, an autoregressive component AR (1) was added to correct for any residual serial correlation (Gujarati 2009). (3) For practical purposes, a correlogram graph is generally used to determine the number of lags to be included in the regression. One lag of GDP was included (Online Supplemental Appendix Tables 5 and 6). Our third model is as follows:
Ln([GDP.sub.t]) = a + b Ln([GDP.sub.t-1]) + [cD.sub.lt] + gAR(1) + [u.sub.t], (3)
As this study extended the research question raised by Luitel and Mahar (2015), the same data set was used herein. In addition, annual U.S. GDP data were gathered for the period from 1973 to 2015 from the U.S. Department of Commerce (2017), Bureau of Economic Analysis website. For the Canadian GDP, the data were obtained from Statistics Canada (2018), CANSIM Tables 379-8031, 370-0031, 380-0063, and 380-8063.
Empirical Analysis and Results
Table 2 shows the estimated results from the three different models introduced above. Both the Akaike Information Criterion (AIC) and the Schwarz Information Criterion (SIC) reported in the table indicate that models corrected for break perform better when including the omitted variable and when correcting for structural change in the time series. By including the dummy variable [D.sub.1t], there was an improvement in the model's performance confirmed by the increased SIC as the specification error of an omitted variable bias was reduced.
An objective of this study was to be able to recognize and select a superior model to perform better forecasting exercises. Based on the SIC, the model with a lower SIC will be more robust. The AR (1) models designed to include a break indicator and enough lags for GDP confirmed by the correlogram graph that justified one lag (Online Supplemental Appendix Tables 5 and 6). As seen in Table 2, AR (1) models had lower values of SIC for both Canada and the U.S. Thus, AR (1) models would be more robust than non-AR (1) models as confirmed by the SIC. Comparing results from all three models reported in Table 2, there is enough evidence to claim that the models with break indicators and lag corrections will have unbiased robust estimations that are more suitable for forecasting purposes. The next step is to empirically compare the forecasting power of all three models.
The estimated models enable the performance of a static forecast of GDP and the evaluation of outcomes. Static forecasting creates a one-step forward forecast of the dependent variable. Using the annual GDP series, dynamic forecasting that involves a multi-step ahead forecast of GDP would not be appropriate. Dynamic forecasting may be used if data are in the monthly or quarterly form (Tong and Lim 1980). Table 3 reports the point (one-step forward) forecast of the GDP for 2015 from the three models. For comparison purpose, Table 3 also reports the actual values of the GDP for 2015 as reported by Bureau of Economic Analysis (2017) and Statistics Canada (2018).
Looking at the actual values of GDP for Canada and the U.S. and the forecasted values from models with and without correction for structural break, one notices improvements in prediction precision. The values of the 2015 predictions from break indicator and AR (1) models are much closer to the actual Canadian and U.S. GDP. Different forecasting results between the no-break model and the corrected models indicate the necessity of discovering possible structural breaks in time series analysis. As reported in Table 3, both countries have the same pattern of forecasting precision and prediction power from the corrected models.
Figures 1 and 2 show more detail of the above analysis. Note that these figures are a snapshot of one-step-forward forecasts of GDPs from 2014 to 2015. Therefore, the legends reflect just 2014 and 2015 and not every year of the study period. In these two figures, it is easy to compare the point-estimated forecast of GDP from all three models. One also notices the different prediction power and precision of the models from these figures. The line graph of AR (1) overlaps the break indicator line as they are both very close estimates. The line graph also shows that the gap between the no-break model and the actual GDP was increasing when the out-of-sample forecast was performed. Put differently, the possible failures of not including an in-sample-break indicator would adversely impact the model's out-of-sample forecasting performance. The Canadian GDP point forecast was an overestimate of the actual GDP regardless of the model specification. However, the U.S. GDP was underestimated by all three models. One possible explanation for such results could be that the magnitude of the impact of a macroeconomics shock (such as from the 2007-2009 financial crisis) experienced by different countries would likely differ. As such, it is likely that countries respond to shock differently and their recovery from shocks would also likely differ.
So far, we have provided evidence of the poor performance of the no-break model and evidence of superior performance of the corrected models. The statistical results of evaluating the forecasts will now show the performance improvement of the out-of-sample predictions. For forecast evaluation from the three models, Table 4 provides information on the computation of nine different forecast error statistics. Root mean squared error (RMSE) and mean absolute error (MAE) statistics reported in the first two rows of Table 4 are relative measures to compare forecasts of GDP. The smaller the error, the better the forecasting ability of a model. The corrected models have a smaller RMSE and MAE that indicate their supremacy against the no-break model. Mean absolute percentage error (MAPE) and Theil inequality coefficient are scale invariant statistics. Their absolute values show the best fit of different models as both are lower for corrected models. The bias, variance, and covariance proportions measure systematic and unsystematic forecasting errors. If a forecast is accurate, the bias and variance proportions need to be small and the covariance proportion needs to be big as all three measures add up to one (Pindyck and Rubinfeld 1998). The computed statistics reported in Table 4 show that predictions from the break indicator models outperform predictions from models that ignore breaks.
Summary and Conclusion
Within the discipline of economics, economists have long recognized the importance of forecasts (Christ 1975; Armstrong 1978; Klein and Moore 1983; Klein 1984; Diebold 1998). They regularly forecast macroeconomic variables, such as GDP, unemployment rate, and interest rate. The ultimate test of an economic theory, or an econometric model, rests on its ability to forecast the future accurately. In a recent study, Luitel and Mahar (2015) showed a structural break in the U.S. GDP time series when the data reporting system switched from the SIC system to the NAICS system. The current study tests whether there was a structural break in Canadian GDP in 1997 the same year when Canada also switched the data reporting system from the SIC to the NAICS system. As expected, there was a structural break in the Canadian GDP in 1997. Forecasting performance and biased estimation were then compared in the presence of structural breaks in the U.S. and Canadian GDP series by using no-break, break-indicator, and AR (1) models. The results show gains in forecasting precision when structural breaks are taken into account for out-of-sample forecasts. Researchers working with the GDP measure in academia, industry and government as policy makers may benefit from this knowledge because forecasts are likely to be relatively more accurate when the models used in the data analysis are corrected for any possible structural break.
Two research areas need further investigation. First, only GDP series were used in the current analysis. GDP is a highly aggregated series obtained from summing consumption, investment, government purchases and net exports. These components of GDP also have their own sub-components. For example, components of consumption include durable goods, nondurable goods and services. Components of investment include residential investment. It is unknown whether the structural break in GDP is an artifact of the data series of GDP alone or wheuier all components of GDP were affected by switching the data reporting system from the SIC to the NAICS system. An analysis of all components of GDP and their sub-components would be necessary to address this issue.
Second, in addition to reporting annual series, both BEA and Statistics Canada report the GDP series and its components on a quarterly basis. Because annual observations of the GDP series were used in the current analysis, dynamic forecasting could not be preformed that involves multi-step-ahead forecasting of GDP. Due to availability of quarterly data, dynamic forecasting would be desirable to address the question whether there would be any improvement in the forecasting precision of GDP if quarterly series were used rather than annual series. Hopefully, the background information provided in this paper will facilitate future work on these issues.
Acknowledgments We thank Lillian Kamal and session participants in the conference for their helpful comments and suggestions. We also thank one anonymous reviewer and the editor for their comments and suggestions.
References
Armstrong, J. S. (1978). Forecasting with econometric methods: Folklore versus fact. The Journal of Business, 5/(4), 549-564.
Christ, C. F. (1975). Judging the performance of econometric models of the U.S. economy. International Economic Review, 16(1), 54-74.
Clements, M. P.. & Hendry. D. F. (2002). A companion to economic forecasting (1st ed.). Wiley-Blackwell Publication.
Diebold, F. X. (1998). The past, present, and future of macroeconomic forecasting. Journal of Economic Perspectives, 12(2), 175-192.
Economic Classification Policy Committee. (1993a). Conceptual issues. Issue Paper No. 1, Bureau of Economic Analysis, Department of Commerce, Washington D. C. https://www.census.gov/eos/www/naics/history/docs/issuc_paper_l.pdf. Accessed June 2018.
Economic Classification Policy Committee. (1993b). Aggregation structure and hierarchies. Issue Paper No. 2, Bureau of Economic Analysis, Department of Commerce. Washington D.C. https://www.census.gov/eos/www/naics/history/docs/issue_paper_2.pdf. Accessed June 2018.
Economic Classification Policy Committee. (1993c). Collcctability of data. Issue Paper No. 3, Bureau of Economic Analysis, Department of Commerce, Washington D.C. https://www.census.gov/cos/www/naics/history/docs/issuc_paper 3.pdf. Accessed June 2018.
Economic Classification Policy Committee. (1993d). Criteria for determining industries. Issue Paper No. 4, Bureau of Economic Analysis. Department of Commerce. Washington D.C. https://www.census.gov/cos/www/naics/history/docs/issuc_papcr 4.pdf Accessed June 2018.
Economic Classification Policy Committee. (1993e). The impact of classification revisions on time series. Issue Paper No. 5, Bureau of Economic Analysis, Department of Commerce, Washington D.C. https://www.census.gov/cos/www/naics/history/docs/issue_paper_5.pdf. Accessed June 2018.
Economic Classification Policy Committee. (1993)- Services classifications. Issue Paper No. 6, Bureau of Economic Analysis, Department of Commerce. Washington D.C. https://www.ccnsus.gov/eos/www/naics/history/docs/issue_paper_6.pdf. Accessed June 2018.
EViews[R] 10 User's Guide II. (2017). IHS Global Inc., Irvine, CA (pp. 143-148 and 421-423).
Gujarati, D. N. (2009). Basic econometrics. Tata McGraw-Hill Education.
Harris, H. & Sollis, R. (2003). Applied Time Series Modeling and Forecasting. Wiley, West Essex.
Klein, L. R. (1984). The importance of the forecast. Journal of Forecasting, 3(1), 1-9.
Klein, P. A., & Moore, G. H. (1983). The leading indicator approach to economic forecasting - Retrospect and prospect. Journal of Forecasting, 2(2), 119-135.
Luitel, H. S., & Mahar, G. J. (2015). A short note on the application of chow test of structural break in U.S. GDP. International Business Research, 8(10), 112-116.
Pindyck, R. S., & Rubinfeld, D. L. (1998). Econometric models and economic forecasts (Vol. 4). Boston: Irwin/McGraw-Hill.
Statistics Canada. (2018). CANSIM, electronic database. Retrieved from https://www l50.statcan.gc.ca/tl/tbll/en/tv.action?pid=3610043401 (last accessed: December 2018).
Tong, H., & Lim, K. (1980). S. Threshold autoregression. limit cycles and cyclical data. Journal of the Royal Statistical Society. Series B (Methodological), Vol. 42, No. 3. pp. 245-292.
U.S. Department of Commerce (2017), Bureau of Economic Analysis' electronic database. Retrieved from https://www.bea.gOv/data/gdp/gross-domestic-product#gdp (last accessed: July 2017).
Publisher's Note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
The research was presented at the 85th International Atlantic Economic Conference. 14-17 March 2018, London. UK.
Electronic supplementary material The online version of this article (https://doi.org/10.1007/sll294-019-09731-w) contains supplementary material, which is available to authorized users.
[??] Afshin Amiraslany
[email protected]
Afshin Amiraslany (1)[iD] * Hari S. Luitel (2) * Gerry J. Mahar (2)
(1)Department of Finance and Management Science. University of Saskatchewan. Saskatoon. SK S7N 5A7, Canada
(2)School of Business and Economics. Algoma University. Sault Ste. Marie, Ontario P6A 2G4, Canada
(1)One of the main problems with economic forecasting is change in the legislation (Clements and Hendry 2002).
(2) Eviews[R] 10 was used to run built-in tests for model stability.
(3) It is based on the Durbin-Watson h statistics for lagged independent variables. However, in Eviews the Cochrane-Orcutt procedure (AR1) is a non-linear least squares (NLLS) estimator to overcome inconsistency in estimating beta (EViews[R] 10 User's Guide II. (2017), pp. 143-148).
https://doi.org/10.1007/s11294-019-09731-w
Table 1 No break hypothesis tests of GDP (1973-2014) (Null hypothesis:
no break at 1997 breakpoint)
United States (a)
Log likelihood ratio 62.74562 (Prob. Chi-Squarc 0.0000)
Wald Statistic 131.2758 (Prob. Chi-Square 0.0000)
Canada (b)
Log likelihood ratio 56.80098 (Prob. Chi-Square 0.0000)
Wald Statistic 110.8662 (Prob. Chi-Square 0.0000)
Sources: (b) Bureau of Economic Analysis (2017a): (b) Statistics Canada
(2018)
Notes: For complete chow breakpoint test and chow forecast test, see
Online Supplemental Appendix Tables 1 and 2
Table 2 Comparison of the models (1973-2014)
Models
Regressors No Break Break indicator
US (a) Canada (b) US (a)
Constant 0.3282 (***) 0.3720 (***) 0.3751 (***)
[LogGDP.sub.1-1] 0.9694 (***) 0.9523 (***) 0.9635 (***)
[D.sub.lt] 0.011
AIC -5.1230 -3.3101 -5.1030
SIC([dagger]) -5.394 -3.2257 -4.977
Regressors AR(1)
Canada (b) US (a) Canada (b)
Constant 0.5914 (***) 0.3694 (***) 0.6119 (***)
[LogGDP.sub.1-1] 0.9144 (***) 0.9642 (***) 0.9108 (***)
[D.sub.lt] 0.0639 0.010 0.0699
AIC -3.4396 -5.029 -3.3431
SIC([dagger]) -3.3129 -4.8203 -3,1319
Notes: (***) Indicates 99% significant, ([dagger]) Typically, AIC and
SIC should select the same model, but SIC has a harsher penalty for more
independent variables. (c) Sec Online Supplemental Appendix Tables 5
and 6 for the Log GDP Correlogram
Sources: (a) Bureau of Economic Analysis (2017); (b) Statistics Canada
(2018)
Table 3 Static forecast of log (GDP) (1973-2014)
United States (a)
Models No Break Break Indicator
Point forecast of GDP for 2015 9.7898 9.7907
Actual GDP Reported in 2015 2015 9.7951
Canada (b)
Models AR(1) No Break Break Indicator
Point forecast of GDP for 2015 9.7910 7.4275 7.4270
Actual GDP Reported in 2015 7.4155
Models AR(1)
Point forecast of GDP for 2015 7.4244
Actual GDP Reported in 2015
Sources: (a) Bureau of Economic Analysis (2017); (b) Statistics Canada
(2018)
Table 4 Log (GDP) forecast error statistics (1973-2014)
United States (a) Break Indicator
Models No Break
Test Statistics (c)
Root Mean Squared Error 0.025648 0.017537
Mean Absolute Error 0.021801 0.012785
Mean Absolute 0.251031 0.148310
Percentage Error
Theil Inequality Coefficient 0.001453 0.000993
Bias Proportion 0.001173 0.000000
Variance Proportion 0.005649 0.000156
Covariance Proportion 0.993177 0.999844
Theil U2 Coefficient 0.366794 0.243971
Symmetric MAPE 0.251048 0.148263
Canada (b)
Models AR(1) No Break Break Indicator
Test Statistics (c)
Root Mean Squared Error 0.017072 0.043463 0.039753
Mean Absolute Error 0.012318 0.022235 0.021528
Mean Absolute 0.141311 0.339291 0.331115
Percentage Error
Theil Inequality Coefficient 0.000963 0.003263 0.002985
Bias Proportion 0.002998 0.000025 0.000050
Variance Proportion 0.012245 0.000710 0.000456
Covariance Proportion 0.984757 0.999265 0.999494
Theil U2 Coefficient 0.245554 0.511994 0.464479
Symmetric MAPE 0.141289 0.340510 0.332018
Models AR(1)
Test Statistics (c)
Root Mean Squared Error 0.039587
Mean Absolute Error 0.021376
Mean Absolute 0.324027
Percentage Error
Theil Inequality Coefficient 0.002957
Bias Proportion 0.000481
Variance Proportion 0.002215
Covariance Proportion 0.997304
Theil U2 Coefficient 0.491553
Symmetric MAPE 0.324983
Sources: (a) Bureau of Economic Analysis (2017); (b) Statistics Canada
(2018)
Notes: (c) For a detailed definition of the forecast evaluation test
statistics sec Online Supplemental Appendix Table 7
 Printer friendly
 Cite/link
 Email
 Feedback
| |
| Title Annotation: | gross domestic product |
|---|---|
| Author: | Amiraslany, Afshin; Luitel, Hari S.; Mahar, Gerry J. |
| Publication: | International Advances in Economic Research |
| Article Type: | Report |
| Geographic Code: | 1CANA |
| Date: | May 1, 2019 |
| Words: | 3512 |
| Previous Article: | Tourism, Capital and Labor Inflows and Regional Development. |
| Next Article: | Trends in Risk-Based Thinking and Risk Management Methods in Czech Plastic Cluster Production Companies. |
| Topics: | |