Repository navigation
BUG: Fix the degrees of freedom of the RandomEffects variance components - #798
Merged
Merged
Conversation
The Swamy-Arora estimators of the variance of the idiosyncratic error and of the variance of the effects divided the residual sums of squares of the within and the between regressions by n - N - K + 1 and N - K, where K is the number of columns of exog. The regressions do not estimate a coefficient for every column. A regressor that is constant within entities (a constant, years of education) or that is collinear with the others once the means of the entities are removed (experience, when the model has year dummies) is not in the within regression, and the dummy variables of the years are collinear with the constant in the between regression of a balanced panel. The "+ 1" is also only right for a model with a constant, as the documentation of the estimator says. The variances were slightly wrong for these models, and so were theta, the estimates and the standard errors. The degrees of freedom are now the numbers of coefficients that the regressions estimate, the ranks of the within-demeaned regressors and of the means of the regressors. The trace in the estimator of Baltagi and Chang (1994) uses a pseudo inverse, since the means of the regressors can be collinear. A model with a constant and regressors that all vary within entities and are not collinear in the means, such as the models that are compared with Stata in the tests, does not change. For the wage panel and the model of the panel_examples notebook (year dummies; black, hisp and educ do not vary over time and exper is collinear with the year dummies) the variances are now 0.1231940 and 0.1053672 instead of 0.1233236 and 0.1069460, theta is 0.64291 instead of 0.64506, and the coefficient of union is 0.10613 instead of 0.10587. These agree with plm (ercomp and coef) to 7 digits. The Hausman statistic of the fixed effects and random effects models of the wage panel with year dummies is now 75.3108, which is the statistic of plm::phtest, instead of 74.5621. Stata's xtreg, re counts in the same way: the sigma_e of a model with a time-invariant regressor is the sigma_e of xtreg, fe, which omits it. In A. C. Cameron's log for the Cornwell and Rupert wage panel (https://cameron.econ.ucdavis.edu/bgpe2013/ct_panel.txt), xtreg lwage exp exp2 wks ed, re has sigma_u .31951859, sigma_e .15220316 and rho .81505521. RandomEffects had .31951715, .15222450 and .81501158, and now has .31951860, .15220316 and .81505522. The tests compare the idiosyncratic variance with the residual variance of the regression on the dummy variables of the entities, the estimators written out in NumPy, and plm (results/re-variance-components-reference.R).
…estimator With small_sample=True and an unbalanced panel, RandomEffects estimates the variance of the effects with the estimator of Baltagi and Chang (1994), whose sum of squares of the between regression is the sum of squares of the regression of the data replaced by the means of the entities, and weights each mean by the number of observations of the entity. The trace that is subtracted from the number of observations is that of the same regression. The residuals came from the unweighted regression of the means instead, so the sum of squares and the trace did not belong to the same estimator. A balanced panel is not affected, since all the weights are equal, and neither is the default estimator, which uses the harmonic mean of the numbers of observations as Stata does. The residuals are now those of the regression that weights the means by the numbers of observations. The variance of the effects equals that of plm (ercomp, whose default for an unbalanced panel is this estimator) to 8 digits. For the unbalanced wage panels of the tests it is 0.1083583 and 0.1076269 instead of 0.1083683 and 0.1076332. The difference grows with the differences in the numbers of observations: it was 5% for a panel in which entities enter and leave at different dates.
… of freedom The variance of the idiosyncratic error divides the residual sum of squares of the within regression by nobs - N - rank, and the variance of the effects divides that of the between regression by N - rank, where rank is the rank of the regressors once the means of the entities are removed, and of the means of the regressors. A panel with one observation of each entity, or with as many entities as the means have rank, has no degrees of freedom left. The model raised a ZeroDivisionError, SciPy's "array must not contain infs or NaNs", or returned an infinite variance of the effects (and a theta of one, the within estimator). A panel with a single entity returned a variance of the effects of zero, which is not an estimate. Both degrees of freedom are now checked before they are used. A ValueError says which variance cannot be estimated, the numbers of observations or entities and the ranks, and how many observations or entities are needed. The tests compare the errors with the degrees of freedom that NumPy finds from the ranks of the demeaned regressors and of the means for a grid of balanced and unbalanced panels with and without a constant, and the idiosyncratic variance of the models that can be estimated with the residual variance of the regression on the dummy variables of the entities.
…nits of the regressors The degrees of freedom of the idiosyncratic variance use the rank of the regressors once the means of the entities are removed. A regressor that does not vary within entities is not left at exactly zero when the means are removed, because the mean of the same number repeated T times is not always that number in floating point arithmetic. It is left with rounding error, about 1e-16 times its size, and that is not small next to the other regressors if its units are large. The rank counted such a regressor, for example income in dollars next to the other regressors, which made the degrees of freedom one too small for each of them, the same mistake that the degrees of freedom of earlier versions made for every time-invariant regressor, and the variance depended on the units: for the wage panel with the years of education multiplied by 3.14e6 the idiosyncratic variance was 0.133625 instead of 0.133584. The rank is now that of the demeaned regressors divided by the norms of the regressors before the means were removed, so that the rounding error is 1e-16 whatever the units. Regressors that are not time-invariant and the constant do not change. The tests multiply a time-invariant regressor by 3.14e3, 3.14e6 and 3.14e8 for the balanced and unbalanced panels, and compare the idiosyncratic variance with that of the regression on the dummy variables of the entities, which does not depend on the units.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #798 +/- ##
==========================================
+ Coverage 99.60% 99.62% +0.01%
==========================================
Files 131 141 +10
Lines 24664 28289 +3625
Branches 1823 1987 +164
==========================================
+ Hits 24566 28182 +3616
- Misses 39 41 +2
- Partials 59 66 +7
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The Swamy-Arora estimators of the variance of the idiosyncratic error and of the
variance of the effects divided the residual sums of squares of the within and the
between regressions by n - N - K + 1 and N - K, where K is the number of columns of
exog. The regressions do not estimate a coefficient for every column. A regressor that
is constant within entities (a constant, years of education) or that is collinear with
the others once the means of the entities are removed (experience, when the model has
year dummies) is not in the within regression, and the dummy variables of the years are
collinear with the constant in the between regression of a balanced panel. The "+ 1"
is also only right for a model with a constant, as the documentation of the estimator
says. The variances were slightly wrong for these models, and so were theta, the
estimates and the standard errors.
The degrees of freedom are now the numbers of coefficients that the regressions
estimate, the ranks of the within-demeaned regressors and of the means of the
regressors. The trace in the estimator of Baltagi and Chang (1994) uses a pseudo
inverse, since the means of the regressors can be collinear. A model with a constant
and regressors that all vary within entities and are not collinear in the means, such as
the models that are compared with Stata in the tests, does not change.
For the wage panel and the model of the panel_examples notebook (year dummies; black,
hisp and educ do not vary over time and exper is collinear with the year dummies) the
variances are now 0.1231940 and 0.1053672 instead of 0.1233236 and 0.1069460, theta is
0.64291 instead of 0.64506, and the coefficient of union is 0.10613 instead of 0.10587.
These agree with plm (ercomp and coef) to 7 digits. The Hausman statistic of the fixed
effects and random effects models of the wage panel with year dummies is now 75.3108,
which is the statistic of plm::phtest, instead of 74.5621.
Stata's xtreg, re counts in the same way: the sigma_e of a model with a time-invariant
regressor is the sigma_e of xtreg, fe, which omits it. In A. C. Cameron's log for the
Cornwell and Rupert wage panel (https://cameron.econ.ucdavis.edu/bgpe2013/ct_panel.txt),
xtreg lwage exp exp2 wks ed, re has sigma_u .31951859, sigma_e .15220316 and rho
.81505521. RandomEffects had .31951715, .15222450 and .81501158, and now has .31951860,
.15220316 and .81505522.
The tests compare the idiosyncratic variance with the residual variance of the
regression on the dummy variables of the entities, the estimators written out in
NumPy, and plm (results/re-variance-components-reference.R).