Robustness test for Synthetic Control Method I am working on a basic Synthetic Control Method (SCM) analysis for establishing the causal effect of a change in bankruptcy legislation (treatment) on the level of entrepreneurship (the outcome variable) in a certain country (the treated unit). Robustness Tests for Quantitative Research The uncertainty researchers face in specifying their estimation models threa- tens the validity of their inferences. On the other hand, a test with fewer assumptions is more robust. But this is generally limited to assumptions that are both super duper important to your analysis (B is really bad), and might fail just by bad luck. A common exercise in empirical studies is a “robustness check”, where the researcher examines how certain “core” regression coefficient estimates behave when the regression specification is modified by adding or removing regressors. This page won't teach you how to run any specific test. These are often presented as things you will want to do alongside your main analysis to check whether the results are "robust.". At the same time, you also learn about a bevy of tests and additional analyses that you can run, called "robustness tests." We ran it because, in the context of the income analysis, homoskedasticity was unlikely to hold. No more running a test and then thinking "okay... it's significant... what now?" But then, what if, to our shock and horror, those assumptions aren't true? 7 Π= + − 0 0 1 01 0 10 ˆ 1 2 1 δ k m δ δ. 19 The main advantage of this methodology is that all variables enter as endogenous within a system of equations, which enables us to reveal the underlying causality among … What was the impact of quantitative easing on investment? We are worried whether our assumptions are true, and we've devised a test that is capable of checking either (1) whether that assumption is true, or (2) whether our results would change if the assumption WASN'T true.1. But the real world is messy, and in social science everything is related to everything else. robustness test econometrics 10 November, 2020 Leave a Comment Written by 355 0 obj > endobj Robustness tests were originally introduced to avoid problems in interlaboratory studies and to identify the potentially responsible factors [2]. Sure, you may have observed that the sun has risen in the East every day for several billion days in a row. A common exercise in empirical studies is a “robustness check”, where the researcher examines how certain “core” regression coefficient estimates behave when the regression specification is modified by adding or removing regressors. Testing restrictions on regression coefficients in linear models often requires correcting the conventional F-test for potential heteroskedasticity or autocorrelation amongst the disturbances, leading to so-called heteroskedasticity and autocorrelation robust test procedures. The following example adds two new regressors on education and age to the above model and calculates the corresponding (non-robust) F test using the anova function. So the real question isn't really whether the assumptions are literally true (they aren't), but rather whether the assumptions are close enough to true that we can work with them. In most cases there are actually multiple different tests you can run for any given assumption. The idea of robust regression is to weigh the observations differently based on how well behaved these observations are. As a robustness test and in order to deal with potential issues of endogeneity bias, we also employ a panel-VAR model to examine the relationship between bank management preferences and various banking sector characteristics. I would like to conduct some robustness checks in Stata (by using the method of Lu and White (2013) - Lu, Xun, and Halbert White. Do a Hausman. Let's imagine that we're interested in the effect of regime change on economic growth in a country. Robust standard errors: Autocorrelation: An identifiable relationship (positive or negative) exists between the values of the error in one period and the values of the error in another period. So you can never really be sure. Robust regression is an alternative to least squares regression when data is contaminated with outliers or influential observations and it can also be used for the purpose of detecting influential observations. 1 If you want to get formal about it, assumptions made in statistics or econometrics are very rarely strictly true. If the coefficients are plausible and robust, this is commonly interpreted as evidence of structural validity. A few reasons! For example, it's generally a good idea in an instrumental variables analysis to test whether your instrument strongly predicts your endogenous variable, even if you have no reason to believe that it won't. Notice that in both of these examples, we had to think about the robustness tests in context. Many of the things that exist under the banner of "robustness test" are specialized hypothesis tests that only exist to be robustness tests, like White, Hausman, Breusch-Pagan, overidentification, etc. Why not? The final result will not do, it is very interesting to see whether initial results comply with the later ones as robustness testing intensifies through the paper/study. On the other hand, a test with fewer assumptions is more robust. If the coefficients are plausible and robust, this is commonly interpreted as evidence of structural validity. The White test is one way (of many) of testing for the presence of heteroskedasticity in your regression. We are grateful to the participants at the International Symposium on Econometrics of Specification Tests in 30 Years at Xiamen University and the seminars at many universities where this paper was presented. robustness test econometrics 10 November, 2020 Leave a Comment Written by . Here, we study when and how one can infer structural validity from coefficient robustness and plausibility. Narrow robustness reports just a handful of alternative specifications, while wide robustness concedes uncertainty among many details of the model. Robustness of the regression coecient is taken as evidence of structural validity. So is it? Robustness is a different concept. Robustness testing has also been used to describe the process of verifying the robustness (i.e. ‘Introduction to Econometrics with R’ is an interactive companion to the well-received textbook ‘Introduction to Econometrics’ by James H. Stock and Mark W. Watson (2015). If the coefficients are plausible and robust, this is commonly interpreted as evidence of structural validity. Testing the robustness of the results of a model or system in the presence of uncertainty. Often they assume that two variables are completely unrelated. In both settings, robust decision making requires the economic agent or the econometrician to explicitly allow for the risk of misspecification. Does free trade reduce or increase inequality? Robust M-Tests - Volume 7 Issue 1 - Franco Peracchi. 3 Despite being very common practice in economics this isn't really the best way to pick control variables or test for the stability of a coefficient. Journal of Econometrics 178 (2014): 194-206). A video segment from the Coursera MOOC on introductory computer programming with MATLAB by Vanderbilt. But you should think carefully about the A, B, C in the fill-in list for each assumption. etc.. There's another reason, too - sometimes the test is just weak! "To determine whether one has estimated effects of interest, β; or only predictive coefficients, β ^ one can check or test robustness by dropping or adding covariates." A common exercise in empirical studies is a “robustness check”, where the researcher examines how certain “core” regression coefficient estimates behave when the regression specification is modified by adding or removing regressors. It normally refers to the sensitivity of an estimator with respect to the violation of certain assumptions of the model, especially in finite samples. Copyright © 2020 Elsevier B.V. or its licensors or contributors. And that might leave you in a pickle - do you stick with the original analysis because your failed test was probably just random chance, or do you adjust your analysis because of the failed test, possibly ending up with the wrong analysis? We previously developed Ballista [26], a well-known robustness Since you have tests at your fingertips you can run for these, seems like you should run them all, right? That's the thing you do when running fixed effects. Increased understanding of the relationships between input and output variables in a system or model. The aim of the conference, “Robustness in Economics and Econometrics,” is to bring together researchers engaged in these two modeling approaches. Second, the list will encourage you to think hard about your actual setting - econometrics is all about picking appropriate assumptions and analyses for the setting and question you're working with. Does a robustness check https://doi.org/10.1016/j.jeconom.2013.08.016. Also, sometimes, there's not a good E to fix the problem if you fail the robustness test. Because your analysis depends on all the assumptions that go into your analysis, not just the ones you have neat and quick tests for. Because the problem is with the hypothesis, the problem is not addressed with robustness checks. For example, one may assume that a linear regression model has normal errors, so the question may be how sensitivity is the Ordinary Least Squares (OLS) estimator to the assumption of normality. Test-retest method: A method of testing robustness in which the similarity of results in assessed after administering a measure to the sample at two different times. Does the minimum wage harm employment? In that case, our analysis would be wrong. Lu gratefully acknowledges partial research support from Hong Kong RGC (Grant No. Regardless, we have to make the list! Heck, sometimes you might even do them before doing your analysis. Every time you do a robustness test, you should be able to fill in the letters in the following list: If you can't fill in that list, don't run the test! This page is pretty heavy on not just doing robustness tests because they're there. You can test for heteroskedasticity, serial correlation, linearity, multicollinearity, any number of additional controls, different specifications for your model, and so on and so on. Inefficient coefficient estimates Biased standard errors Unreliable hypothesis tests: Geary or runs test But what does that mean? The book also discusses Why bother with this list? However, robustness generally comes at the cost of power, because either less information from the input is used, or more parameters need to be estimated. What is the best method to measure robustness? Thinking about robustness tests in this way - as ways of evaluating our assumptions - gives us a clear way of thinking about using them. Weighted least squares (WLS) 2. Kiefer, Nicholas M. & Bunzel, Helle & Vogelsang, Timothy & Vogelsang, Timothy & Bunzel, Helle, 2000. There's not much you can do about that. Checking of robustness is one of a common procedure in econometrics. ‘Introduction to Econometrics with R’ is an interactive companion to the well-received textbook ‘Introduction to Econometrics’ by James H. Stock and Mark W. Watson (2015). Because a robustness test is anything that lets you evaluate the importance of one of your assumptions for your analysis. Type I error, in other words. So if parental income does increase your income, it will also likely increase the variance of your income in ways my control variables won't account for, and so be correlated with the variance of the error term, use heteroskedasticity-robust standard errors, that my variables are unrelated to the error term (no omitted variable bias), the coefficient on regime change might be biased up or down, depending on which variables are omitted, regime change often follows heightened levels of violence, and violence affects economic growth, so violence will be related to GDP growth and will be in the error term if not controlled for, the coefficient on regime change is very different with the new control. But this is not a good way to think about robustness tests! We also thank the editor and two anonymous referees for their helpful comments. You just found a significant coefficient by random chance, even though the true effect is likely zero. 643711). 355 0 obj > endobj Robustness tests were originally introduced to avoid problems in interlaboratory studies and to identify the potentially responsible factors [2]. After all, they're usually idealized assumptions that cleanly describe statistical relationships or distributions, or economic theory. Second is the robustness test: is the estimate different from the results of other plausible models? Filling in the list includes filling in C, even if your answer for C is just "because A is not true in lots of analyses," although you can hopefully do better than that.2 As a bonus, once you've filled in the list you've basically already written a paragraph of your paper. I have a family. If you really want to do an analysis super-correctly, you shouldn't be doing one of those fill-in lists above for every robustness check you run - you should be trying to do a fill-in list for every assumption your analysis makes. Indeed, if not conducted properly, robustness checks can be completely uninformative or entirely misleading. Beginners with little background in statistics and econometrics often have a hard time understanding the benefits of having programming skills for learning and applying Econometrics. If the coe¢ cients are plausible and robust, this is commonly interpreted as evidence of structural validity. These kinds of robustness tests can include lots of things, from simply looking at a graph of your data to see if your functional form assumption looks reasonable, to checking if your treatment and control groups appear to have been changing in similar ways in the "before" period of a difference-in-difference (i.e. Accordingly, we give a straightforward robustness test that turns informal robustness checks into true Hausman (1978)-type structural speci–cation tests. The researcher carefully scrutinized the regression coefficient estimates when the … logic of robustness testing, provides an operational de nition of robustness that can be applied in all quantitative research and introduces readers to diverse types of robustness tests. Suppose we –nd that the critical core coe¢ cients are not robust. "Simple Robust Testing of Regression Hypotheses," Staff General Research Papers Archive 1832, Iowa State University, Department of Economics. "Simple Robust Testing of Regression Hypotheses," Staff General Research Papers Archive 1832, Iowa State University, Department of Economics. These are things like the White test, the Hausman test, the overidentification test, the Breusch-Pagan test, or just running your model again with an additional control variable. The same problem applies in the opposite direction with robustness tests. When considering how robust an estimator is to the presence of outliers, it is useful to test what happens when an extreme outlier is added to the dataset, and to test what happens when an extreme outlier replaces one of the existing datapoints, and then to consider the … In your econometrics class you learn all sorts of analytic tools: ordinary least squares, fixed effects, autoregressive processes, and many more. Let's say that we are interested in the effect of your parents' income on your own income, so we regress your own income on your parents' income when you were 18, and some controls. Robustness Tests: What, Why, and How. We've already gone over the robustness test of adding additional controls to your model to see what changes - that's not a specialized robustness test. Robustness checks involve reporting alternative specifications that test the same hypothesis. H1: The assumption made in the analysis is false. In the post on hypothesis testing the F test is presented as a method to test the joint significance of multiple regressors. This paper investigates the local robustness properties of a general class of multidimensional tests based on M-estimators.These tests are shown to inherit the efficiency and robustness properties of the estimators on which they are based. So we have to make assumptions. The more assumptions a test makes, the less robust it is, because all these assumptions must be met for the test to be valid. Abstract A common exercise in empirical studies is a "robustness check," where the researcher examines how certain "core" regression coe¢ cient estimates behave when the regression speci–cation is modi–ed by adding or removing regressors. Robustness tests are all about assumptions.