Repository navigation
TST: Use eigh for the symmetric matrix in test_against_ols - #752
Merged
Merged
Conversation
test_against_ols builds an orthogonal basis for the absorbed regressors from the eigenvectors of the symmetric matrix absorb.T @ absorb. np.linalg.eig is the solver for general matrices, and when the matrix is singular, which it is in 32 of the 72 configurations, it returns the whole array of eigenvectors as complex. Each of the 32 tests then emitted "ComplexWarning: Casting complex values to real discards the imaginary part" when IVData converted the regressors to float (iv/data.py:82). Use np.linalg.eigh. It is the solver for symmetric matrices and its eigenvectors are real. It orders the eigenvalues from the smallest, which the test already reverses with argsort(a)[::-1]. Only the span of the leading eigenvectors enters the OLS regression, so the test checks the same thing. Comparing the eig and eigh versions of every configuration: - The imaginary part of the eigenvectors that eig returned and the test used is exactly 0 in all 32 configurations. - The projection matrices on the two bases differ by at most 2e-16. - The OLS parameters, standard errors, covariance, residuals, R2 and s2 differ by at most 6e-15. Running test_against_ols with ComplexWarning turned into an error: 32 of 72 fail before this change, and all 72 pass after it. This is also needed before IVData rejects complex input instead of discarding the imaginary part, which would turn these warnings into errors.
Two deprecated features have no removal date. Say that they will be removed in
linearmodels 8.0.
AbsorbingLS.fit(lsmr_options=...) has been deprecated since 4.17 in favor of
absorb_options. The FutureWarning and the ".. deprecated:: 4.17" note in the
docstring now say that lsmr_options will be removed in linearmodels 8.0. The
docstring also had a typo, "Options to ass to".
linearmodels.__future__.ordering has been empty since 5.0, when the variable
ordering that importing it enabled in 4.31 became the default. Importing it did
nothing and said nothing. It now emits
FutureWarning: linearmodels.__future__.ordering has no effect and will be
removed in linearmodels 8.0
FutureWarning is shown by default. stacklevel=2 attributes the warning to the
statement that imports the module, for `import linearmodels.__future__.ordering`
and for `from linearmodels.__future__ import ordering`, because the warnings
module skips the frames of the import machinery. Nothing in linearmodels or the
examples imports the module. linearmodels/__future__/__init__.py is still empty,
so importing the package does not warn.
The new tests in tests/test_deprecations.py check that
- both warnings have the text above and are attributed to the caller: the file
that calls fit, and the file and line of the import statement;
- the lsmr_options docstring says that it is deprecated and will be removed in
8.0;
- lsmr_options is still passed on to the absorption, by comparing a fit with
maxiter=1, which does not converge, to the same fit using absorb_options.
The ordering module runs only the first time that it is imported, which may
have been earlier in the test session. The tests remove it from sys.modules, run
the import in a code object with a file name and a line number to see where the
warning points, and restore the state afterwards. One more test imports it in a
new interpreter, which is what a user sees. It takes a few seconds, because
importing linearmodels does.
Six of the seven new tests fail on main. The seventh checks that lsmr_options
still works, which it also does on main.
test_examples.py switches asyncio to the Windows selector event loop policy so that zmq, which the notebook kernel client uses, does not warn about the default Proactor loop. WindowsSelectorEventLoopPolicy, get_event_loop_policy and set_event_loop_policy are deprecated in Python 3.14 and will be removed in 3.16. Every xdist worker emitted four DeprecationWarnings when it imported the test module, 16 in a run with -n 4. Only select the policy before Python 3.14. The (3, 8) lower bound is below the minimum Python version, so drop it. I checked that the notebooks run without the policy on Windows 11 with Python 3.14.6, pyzmq 27.1.0, jupyter_client 8.9.1, ipykernel 7.3.0, nbclient 0.11.0 and tornado 6.5.7, using python -m pytest linearmodels/tests/test_examples.py -n 4: | Case | Result | |---|---| | Policy selected (before this change) | 12 passed, 16 warnings (DeprecationWarning) | | Policy not selected | 12 passed, 4 warnings (pyzmq RuntimeWarning) | | Policy not selected, pyzmq warning ignored | 12 passed, 0 warnings | Without the policy pyzmq warns once per worker that the Proactor loop has no add_reader, that it registers a selector thread instead, and recommends the deprecated function. It only describes how zmq works on this loop, so ignore it in test_notebook.
test_c_stat_against_r compared p-values with rtol=1e-5, atol=1e-12 and a comment that the p-value is 1 - cdf, so it cannot resolve probabilities below about 1e-16. That stopped being true in #742: p-values are survival functions, which are accurate in the far tail. The absolute tolerance only hides errors in small p-values. Four of the 23 reference p-values are below 1e-12 (the smallest is 2e-29), so any computed value between 0 and 1e-12 passed for them. Use rtol=1e-8 and no atol, and replace the stale comment. The largest relative difference between a p-value and the R reference in the table is 7e-13, so the tolerance leaves a margin of four orders of magnitude.
The HDFE path sets absorb_options["drop_singletons"] = False before it calls
pyhdfe.create. It did this on the dict that the caller passed, so after fit the
caller's dict had a new key. That key is not an option of LSMR, so reusing the
dict for a model that is fitted with method="lsmr" failed:
options = {}
AbsorbingLS(y, x, absorb=cat).fit(absorb_options=options)
# options is now {"drop_singletons": False}
AbsorbingLS(y, x, absorb=cat).fit(absorb_options=options, method="lsmr")
# TypeError: lsmr() got an unexpected keyword argument 'drop_singletons'
Copy the dict in _first_time_fit. The deprecated lsmr_options were already
copied.
The new test checks both symptoms. On main it fails on the first assertion,
because the options that were passed in have gained a key, and the LSMR fit
that follows raises the TypeError above. The LSMR fit does not use the cache,
which could otherwise hold the result of an earlier fit and skip the call that
fails.
This is a different issue from the lsmr_options deprecation and is in its own
commit.
AbsorbingLS.fit absorbs the variables the first time that it is called and
keeps them. Every later call ignored method, absorb_options, lsmr_options and
use_cache, and used the variables that were absorbed by the first call:
mod = AbsorbingLS(y, x, absorb=cat) # two crossed effects
mod.fit().params # HDFE: x0 0.979689, x1 1.015166
mod.fit(method="lsmr", absorb_options={"maxiter": 1}).params
# x0 0.979689, x1 1.015166
AbsorbingLS(y, x, absorb=cat).fit(
method="lsmr", absorb_options={"maxiter": 1}).params
# x0 0.989950, x1 1.015209
A call that could not be used did not raise either, e.g., method="hdfe" for a
weighted model, or an option that LSMR does not have, where a new model raises.
The variables are now absorbed again if method, absorb_options (also when given
as the deprecated lsmr_options) or use_cache differ from the values that were
used to absorb them. Calls with the same values, for example to change the
covariance estimator, use the absorbed variables as before. The show option is
not compared since it only controls the output. The values are compared when
fit is called, so changing a dict that was passed to a previous call is seen.
This is documented in the Notes of fit.
Absorbing again needs two more changes:
- The state that the first call accumulated is reset: the number of parameters
was increased and the retained columns were selected from the columns that
were left by the previous call. This also fixes a model whose first fit
raised, e.g., because of an invalid option: the degrees of freedom of its
next fit included those of the failed call (df_model 140 instead of 71 in
the tests).
- The cache of absorbed variables (_VARIABLE_CACHE), which is shared by all
models, identified residuals by the regressors and the variable but not by
the LSMR options. With the default use_cache=True, a model that used
maxiter=1 got the residuals of an earlier model that used the default
options whenever the cache held them, and absorbing again would have
returned them too. The options are now part of the key. Passing the default
values explicitly does not change the key.
use_cache=False also wrote to the cache. It stored the residuals of the last
column under an empty key that is never read, which keeps one array in memory
for each model. It does not write any more.
Evidence of the cache problem, without any refit:
clear_cache()
fit(method="lsmr").params # x0 0.979689
fit(method="lsmr", absorb_options={"maxiter": 1}) # x0 0.979689, from the cache
clear_cache()
fit(method="lsmr", absorb_options={"maxiter": 1}) # x0 0.989950
test_options in test_absorbing.py passes without a change. Its second fit asks
for LSMR with other tolerances after an HDFE fit; it did nothing before and now
absorbs the variables with LSMR. The new tests check the results.
Tests (linearmodels/tests/iv/test_absorbing_refit.py, 28 tests): a second fit
with other options is the same as the fit of a new model, in the estimates,
standard errors, df_model, residuals and the absorbed effects, for the HDFE, the
weighted and the continuous variable cases; going back to the defaults; no
change in df_model; reuse of the absorbed variables when only the covariance
estimator, or equal options in another dict, are used (counted); a dict that the
caller changes; lsmr_options; the options of pyhdfe, which are a nested dict;
methods and options that are not valid for a model that has been fit; drop_absorbed
when the absorbed column is not the last one; the cache with different options,
with array valued options and with use_cache=False. 25 of the 28 fail on main.
The three that pass (df_model after a refit) guard against absorbing again
changing the degrees of freedom. Undoing any one part of the change (11 variants
were tried, e.g., not resetting the columns, leaving use_cache or show out of
the comparison, or using the key without the options) makes a test fail.
The fix of the next commit changes AbsorbingLS.fit, which is also changed by two branches that are not merged yet: * lm71-fix-absorbing-refit (3016825) absorbs the variables again if fit is called with other options, and puts the options of LSMR in the key of the cache. * lm71-bug6-hygiene (c8a518f) copies absorb_options before fit adds to it, and marks lsmr_options for removal in linearmodels 8.0. This branch starts from the first and merges the second, so that it applies to both once they are merged. The merge has no conflicts: the three hunks of lm71-bug6-hygiene in linearmodels/iv/absorbing.py (the copy of absorb_options, the documentation of lsmr_options and the text of its FutureWarning) do not overlap the hunks of the other branch. Merge order: lm71-bug6-hygiene and lm71-fix-absorbing-refit, in either order, then lm71-fix-lsmr-options.
Stacked on lm71-fix-absorbing-refit (FIX1) and lm71-bug6-hygiene (BUG6), which
change AbsorbingLS.fit too and are merged into this branch by the previous
commit (no conflicts). Merge order: BUG6 and fix-absorbing-refit, in either
order, and then this branch.
AbsorbingLS.fit with the deprecated lsmr_options and the default method failed
for the models that HDFE can absorb, which are the unweighted models whose
absorbed variables are all categorical:
mod = AbsorbingLS(y, x, absorb=cat)
mod.fit(lsmr_options={"atol": 1e-8})
# TypeError: create() got an unexpected keyword argument 'atol'
The default method, "auto", absorbs using HDFE when it can, and the options are
passed to the algorithm that is used. lsmr_options are options of LSMR
(scipy.sparse.linalg.lsmr), and they reached pyhdfe.create. They worked for
weighted models and models with continuous absorbed variables, since these use
LSMR, and with method="lsmr". The options in absorb_options are for the same
algorithm, and so have the same problem: any option of LSMR, e.g., maxiter,
raised the TypeError when "auto" selected HDFE. This is not new in 7.1.
lsmr_options:
* If they are not empty and method is "auto", the variables are absorbed using
LSMR, which is what they have always been for. The FutureWarning is not
changed.
* With method="hdfe" they raise a ValueError that says to use method="lsmr" or
"auto". They were a TypeError, or a RuntimeError for a model that HDFE cannot
absorb.
* Empty lsmr_options have no options to pass, and do not change the algorithm.
Code that passes lsmr_options=opts, where opts can be {}, keeps using HDFE
for these models.
* The method that is used is the one that fit compares with the arguments of the
previous call (lm71-fix-absorbing-refit), so that calling fit again with the
same lsmr_options, or with method="lsmr" and the same absorb_options, uses the
variables that were absorbed and does not absorb them again.
absorb_options, which does not say which algorithm its options are for:
* Options of LSMR when the variables are absorbed using HDFE are a ValueError
that names the options and says that method="lsmr" is needed, for method
"auto" and "hdfe".
* Options of pyhdfe.create (e.g., residualize_method) when the variables are
absorbed using LSMR were a TypeError from lsmr(). They are a ValueError too.
The two sets of names are disjoint (atol, btol, conlim, damp, maxiter, show and
x0 for LSMR; cluster_ids, compute_degrees, degrees_method, drop_singletons,
options and residualize_method for pyhdfe 0.2), and they are taken from the
signatures of the functions. An option that is not an option of either
algorithm is still the TypeError of the algorithm.
I made the options of LSMR in absorb_options an error and did not select LSMR
for them, as lsmr_options do. A name of an option should not select the
algorithm in a method that is "auto", and a call that selects an algorithm in
this way is hard to take back, whereas the ValueError can be relaxed later without
breaking anyone. These calls raised an error before, so no call that worked
changes, and the message has the remedy, method="lsmr". lsmr_options are different
since their name says that they are for LSMR, and callers that used them before
HDFE was added always got LSMR.
Models without absorbed variables have no algorithm and ignore the options, as
before.
The tests (test_absorbing_lsmr_options.py, 54 tests) check that the estimates of
lsmr_options with the default method are those of method="lsmr" with the same
options, for options with and without an effect (maxiter=1 stops LSMR far from
the HDFE solution), that the algorithm that was used is the one that is
expected by recording the calls of lsmr_annihilate and pyhdfe.create, the errors
and their text, that a model can be used after an error, that empty lsmr_options
use HDFE, and the number of times that the variables are absorbed. 29 of the 54
fail on the base of this branch.
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #752 +/- ##
==========================================
- Coverage 99.58% 99.57% -0.02%
==========================================
Files 110 114 +4
Lines 20328 20895 +567
Branches 1600 1628 +28
==========================================
+ Hits 20244 20806 +562
- Misses 31 34 +3
- Partials 53 55 +2
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
shaneraphel
pushed a commit
to shaneraphel/linearmodels
that referenced
this pull request
Oct 13, 2026
The FutureWarning of AbsorbingLS.fit(lsmr_options=...) said "Use absorb_options". For a model that can be absorbed using HDFE, which is a model without weights whose absorbed variables are all categorical, absorb_options are the options of HDFE unless the method is LSMR, so that advice raises a ValueError (bashtage#752 made the options of LSMR with HDFE an error). The warning and the deprecation note in the docstring now say to use absorb_options with method="lsmr", which gives the same results as lsmr_options and does not warn.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
test_against_ols builds an orthogonal basis for the absorbed regressors from
the eigenvectors of the symmetric matrix absorb.T @ absorb. np.linalg.eig is
the solver for general matrices, and when the matrix is singular, which it is
in 32 of the 72 configurations, it returns the whole array of eigenvectors as
complex. Each of the 32 tests then emitted "ComplexWarning: Casting complex
values to real discards the imaginary part" when IVData converted the
regressors to float (iv/data.py:82).
Use np.linalg.eigh. It is the solver for symmetric matrices and its
eigenvectors are real. It orders the eigenvalues from the smallest, which the
test already reverses with argsort(a)[::-1]. Only the span of the leading
eigenvectors enters the OLS regression, so the test checks the same thing.
Comparing the eig and eigh versions of every configuration:
is exactly 0 in all 32 configurations.
by at most 6e-15.
Running test_against_ols with ComplexWarning turned into an error: 32 of 72
fail before this change, and all 72 pass after it. This is also needed before
IVData rejects complex input instead of discarding the imaginary part, which
would turn these warnings into errors.