Repository navigation
Conversation
…in-plate bases The B-, M- and I-spline transformers turned a NaN input into a finite all-zero basis row: np.nan_to_num in each design matrix, there to clear the NaN that BSpline(extrapolate=False) returns outside the knot span, also zeroed the rows of genuinely missing inputs. For the I-spline that row is exactly the encoding of the training minimum, so "missing" and "minimum" became indistinguishable; for the B/M-spline it is impossible data. ThinPlateSplineTransformer fitted on two features had the same defect: its kernel np.where(r > 0, r**2 * log(r), 0.0) sent a NaN distance to the zero branch. This was reachable standalone and from the Preprocessor with missing_policy="propagate" or numerical_imputation=None, while every other spline family and feature map propagates NaN. The shared B/M/I transform now evaluates only the observed rows and leaves the rows of missing values NaN in that feature's basis block (an optional bias column stays 1). The two-feature thin-plate kernel selects its zero branch by r == 0, so a NaN distance stays NaN. Finite rows are unchanged bit for bit under every out_of_range policy, and a missing value does not trigger the "warn" / "error" policies. The Preprocessor numerical_imputation docstring and the missing-values guide no longer claim that every numerical method raises on (or merely tolerates) a missing value. Fixes #87
This was referenced Oct 16, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #87.
BSplineTransformer,MSplineTransformer,ISplineTransformerand a two-featureThinPlateSplineTransformerencoded a missing input as a finite all-zero row instead of NaN. For the I-spline that row is exactly the encoding of the training minimum. This hit standalone use and thePreprocessorwithmissing_policy="propagate"ornumerical_imputation=None, while every other spline family and the feature maps propagate NaN.Root cause
The B/M/I design matrices pass the basis values through
np.nan_to_num(..., nan=0.0). It is there to clear the NaN that SciPy'sBSpline(..., extrapolate=False)returns outside the knot span (under the"extrapolate"and"warn"out-of-range policies), but it cannot tell those apart from the NaN of a missing input, so it zeroed both. In the two-feature thin-plate kernel,np.where(r > 0, r**2 * log(r), 0.0)sent a NaN distance to the0.0branch, becauseNaN > 0is false.Fix
BaseSplineTransformer.transformevaluates_design_matrixonly on the observed rows of each feature (afterresolve_out_of_range) and leaves the rows of missing values NaN in that feature's basis block. The optional bias column is still prepended as1. Thenan_to_numcalls in the B/M/I design matrices stay for the out-of-span values and no longer see missing inputs.ThinPlateSplineTransformer._tps_kernelselects the zero branch of the two-dimensional kernel byr == 0instead ofr > 0, so a NaN distance stays NaN.Behaviour changes
0in the basis columns of a missing input, standalone and through thePreprocessorwithmissing_policy="propagate"ornumerical_imputation=None. Withinclude_bias=Truethe bias column stays1. Pipelines that disable imputation and pass this output to an estimator that rejects NaN now see the NaN.ThinPlateSplineTransformerreturns NaN for a row with a missing coordinate instead of an all-zero kernel row (withinclude_bias=True,[1, NaN, ..., NaN]instead of[1, 0, ..., 0]).Documentation
BaseSplineTransformerclass docstring: missing values are ignored for knot placement and, at transform time, expand to a NaN row in their feature's basis block (the bias column stays1), as in the other spline families.BaseSplineTransformer._design_matrixdocstring:xholds only the observed values, already resolved against the out-of-range policy;transformfills the missing rows with NaN.Preprocessor,numerical_imputation: the old text said that with imputation disabled all numerical methods raise on missing values. It now says the scalers, splines and therbf/relu/sigmoid/tanhfeature maps propagate a missing value as NaN in that feature's output, while"ple","custombin","fourier"and"polynomial"raise.docs/core_concepts/missing_values.md(note on disabled imputation): "tolerate it, so an affected row's output is itself undefined" becomes "propagate it: the affected row's output columns for that feature areNaN(a spline's optional bias column stays1)".Tests
test_mspline_handles_nan(updated; it asserted finite output): the missing row is NaN and every other row is finite. Fails onmain, passes here.test_bmi_spline_missing_value_expands_to_a_nan_row[{clip,extrapolate,warn,error}-{False,True}-{B,M,I}SplineTransformer]: the basis columns of missing rows are NaN, the bias column is1, the other rows equal the transform of the same values without the missing rows, and a missing value triggers neither the"warn"nor the"error"reaction. All 24 fail onmain, pass here.test_bmi_spline_missing_value_only_affects_its_own_feature_block[{B,M,I}SplineTransformer]: with two features, NaN appears only in the block of the feature that is missing. Fails onmain, passes here.test_tprs_expands_a_missing_value_to_a_nan_row[2]: a two-feature thin-plate spline gives a NaN row for a missing coordinate and leaves the other rows unchanged. Fails onmain, passes here.test_tprs_expands_a_missing_value_to_a_nan_row[1, 3]: the same for one and three features, whose kernels already propagated NaN. Guard test: passes on both by design.test_propagate_expands_a_missing_value_to_a_nan_basis_row[bspline, mspline, ispline]:Preprocessor(missing_policy="propagate")gives a NaN row for the missing value and finite rows otherwise. Fails onmain, passes here.test_partial_missing_propagates_only_on_missing_rows[BSpline, MSpline, ISpline]: the existing edge-case contract test now also covers B/M/I (added toNAN_PROPAGATING, dropping the comment that they clip a missing value to the fitted boundary). Fails onmain, passes here.Reproduction
Output of the script from #87 on this branch:
Checks
just test: 1474 passed, 59 skipped, 7 xfailed; coverage 93.65% (gate 90%).scripts/quickstart.pypasses.just docs: strict build succeeds.just check: all pre-commit and pre-push hooks pass (ruff format, ruff lint, prettier, file hygiene, pyright with 0 errors).Merge notes
main.missing_values.mdandpretab/preprocessor.py, in other places.extrapolate=True, so out-of-range rows under"extrapolate"/"warn"are extrapolated instead of zeroed. The two changes are independent: this branch only removes the missing rows before_design_matrix, and the out-of-range rows are left to A partial spline policy mapping drops the "clip" default, and "extrapolate"/"warn" give all-zero basis rows #84. The PRs for ISplineTransformer: 200-point trapezoid integration produces wrong and all-zero basis columns #53 (fix(splines): evaluate the I-spline basis in closed form #98) (closed-form I-spline) and MSplineTransformer: absolute clipping thresholds make the basis depend on feature scale #63 (M-spline clipping) change_design_matrixinternals, which now receive observed values only. With this branch merged together with the branches for A partial spline policy mapping drops the "clip" default, and "extrapolate"/"warn" give all-zero basis rows #84, ISplineTransformer: 200-point trapezoid integration produces wrong and all-zero basis columns #53, MSplineTransformer: absolute clipping thresholds make the basis depend on feature scale #63, A 2-D y (multi-output target) crashes the B/M/I splines even when unsupervised, and target-aware methods with unrelated errors #88 and add_missing_indicator / impute_with_indicator feeds the indicator through the scaler and basis; lineage crashes #62, the full test suite passes (1573 passed, 59 skipped, 7 xfailed), and so does this branch merged with B/M/I spline knot_locations are sized by output_dim and clipped to the data range instead of used as given #81 (1490 passed) or with Preprocessor crashes on an all-missing column at fit (numerical, one-hot) or silently drops it (int) #95 (1545 passed).This change was previously part of #97, which is closed in favour of one pull request per issue.