Skip to content

fix(splines): expand a missing value to a NaN row in the B/M/I and thin-plate bases - #122

Open
ChrisW09 wants to merge 1 commit into
mainfrom
fix/87-spline-missing-rows
Open

ChrisW09 wants to merge 1 commit into
mainfrom
fix/87-spline-missing-rows

Conversation

@ChrisW09

@ChrisW09 ChrisW09 commented Oct 16, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #87.

BSplineTransformer, MSplineTransformer, ISplineTransformer and a two-feature ThinPlateSplineTransformer encoded a missing input as a finite all-zero row instead of NaN. For the I-spline that row is exactly the encoding of the training minimum. This hit standalone use and the Preprocessor with missing_policy="propagate" or numerical_imputation=None, while every other spline family and the feature maps propagate NaN.

Root cause

The B/M/I design matrices pass the basis values through np.nan_to_num(..., nan=0.0). It is there to clear the NaN that SciPy's BSpline(..., extrapolate=False) returns outside the knot span (under the "extrapolate" and "warn" out-of-range policies), but it cannot tell those apart from the NaN of a missing input, so it zeroed both. In the two-feature thin-plate kernel, np.where(r > 0, r**2 * log(r), 0.0) sent a NaN distance to the 0.0 branch, because NaN > 0 is false.

Fix

  • BaseSplineTransformer.transform evaluates _design_matrix only on the observed rows of each feature (after resolve_out_of_range) and leaves the rows of missing values NaN in that feature's basis block. The optional bias column is still prepended as 1. The nan_to_num calls in the B/M/I design matrices stay for the out-of-span values and no longer see missing inputs.
  • ThinPlateSplineTransformer._tps_kernel selects the zero branch of the two-dimensional kernel by r == 0 instead of r > 0, so a NaN distance stays NaN.

Behaviour changes

  • B/M/I splines return NaN instead of 0 in the basis columns of a missing input, standalone and through the Preprocessor with missing_policy="propagate" or numerical_imputation=None. With include_bias=True the bias column stays 1. Pipelines that disable imputation and pass this output to an estimator that rejects NaN now see the NaN.
  • A two-feature ThinPlateSplineTransformer returns NaN for a row with a missing coordinate instead of an all-zero kernel row (with include_bias=True, [1, NaN, ..., NaN] instead of [1, 0, ..., 0]).
  • Rows without a missing value are unchanged under every out-of-range policy, and no golden regression baseline changed.

Documentation

  • BaseSplineTransformer class docstring: missing values are ignored for knot placement and, at transform time, expand to a NaN row in their feature's basis block (the bias column stays 1), as in the other spline families.
  • BaseSplineTransformer._design_matrix docstring: x holds only the observed values, already resolved against the out-of-range policy; transform fills the missing rows with NaN.
  • Preprocessor, numerical_imputation: the old text said that with imputation disabled all numerical methods raise on missing values. It now says the scalers, splines and the rbf / relu / sigmoid / tanh feature maps propagate a missing value as NaN in that feature's output, while "ple", "custombin", "fourier" and "polynomial" raise.
  • docs/core_concepts/missing_values.md (note on disabled imputation): "tolerate it, so an affected row's output is itself undefined" becomes "propagate it: the affected row's output columns for that feature are NaN (a spline's optional bias column stays 1)".

Tests

  • test_mspline_handles_nan (updated; it asserted finite output): the missing row is NaN and every other row is finite. Fails on main, passes here.
  • test_bmi_spline_missing_value_expands_to_a_nan_row[{clip,extrapolate,warn,error}-{False,True}-{B,M,I}SplineTransformer]: the basis columns of missing rows are NaN, the bias column is 1, the other rows equal the transform of the same values without the missing rows, and a missing value triggers neither the "warn" nor the "error" reaction. All 24 fail on main, pass here.
  • test_bmi_spline_missing_value_only_affects_its_own_feature_block[{B,M,I}SplineTransformer]: with two features, NaN appears only in the block of the feature that is missing. Fails on main, passes here.
  • test_tprs_expands_a_missing_value_to_a_nan_row[2]: a two-feature thin-plate spline gives a NaN row for a missing coordinate and leaves the other rows unchanged. Fails on main, passes here.
  • test_tprs_expands_a_missing_value_to_a_nan_row[1, 3]: the same for one and three features, whose kernels already propagated NaN. Guard test: passes on both by design.
  • test_propagate_expands_a_missing_value_to_a_nan_basis_row[bspline, mspline, ispline]: Preprocessor(missing_policy="propagate") gives a NaN row for the missing value and finite rows otherwise. Fails on main, passes here.
  • test_partial_missing_propagates_only_on_missing_rows[BSpline, MSpline, ISpline]: the existing edge-case contract test now also covers B/M/I (added to NAN_PROPAGATING, dropping the comment that they clip a missing value to the fitted boundary). Fails on main, passes here.

Reproduction

Output of the script from #87 on this branch:

BSplineTransformer   NaN -> [nan nan nan nan nan]  x_min -> [1. 0. 0. 0. 0.]
MSplineTransformer   NaN -> [nan nan nan nan nan]  x_min -> [0.8 0.  0.  0.  0. ]
ISplineTransformer   NaN -> [nan nan nan nan nan]  x_min -> [0. 0. 0. 0. 0.]
PSplineTransformer   NaN -> [nan nan nan nan nan]  x_min -> [1. 0. 0. 0. 0.]
ThinPlateSplineTransformer (2 features), [NaN, 5] -> [nan nan nan nan]
Preprocessor(ispline, propagate): NaN row = [nan nan nan nan nan nan nan] | identical to the x_min row: False

Checks

  • just test: 1474 passed, 59 skipped, 7 xfailed; coverage 93.65% (gate 90%).
  • Minimum dependencies (CI job: Python 3.10, numpy 1.24.4, pandas 2.0.3, scipy 1.10.1, scikit-learn 1.6.0): 1461 passed, 72 skipped, 7 xfailed; scripts/quickstart.py passes.
  • just docs: strict build succeeds.
  • just check: all pre-commit and pre-push hooks pass (ruff format, ruff lint, prettier, file hygiene, pyright with 0 errors).
  • Import direction between subpackages is unchanged.

Merge notes

This change was previously part of #97, which is closed in favour of one pull request per issue.

…in-plate bases

The B-, M- and I-spline transformers turned a NaN input into a finite
all-zero basis row: np.nan_to_num in each design matrix, there to clear
the NaN that BSpline(extrapolate=False) returns outside the knot span,
also zeroed the rows of genuinely missing inputs. For the I-spline that
row is exactly the encoding of the training minimum, so "missing" and
"minimum" became indistinguishable; for the B/M-spline it is impossible
data. ThinPlateSplineTransformer fitted on two features had the same
defect: its kernel np.where(r > 0, r**2 * log(r), 0.0) sent a NaN
distance to the zero branch. This was reachable standalone and from the
Preprocessor with missing_policy="propagate" or numerical_imputation=None,
while every other spline family and feature map propagates NaN.

The shared B/M/I transform now evaluates only the observed rows and
leaves the rows of missing values NaN in that feature's basis block (an
optional bias column stays 1). The two-feature thin-plate kernel selects
its zero branch by r == 0, so a NaN distance stays NaN. Finite rows are
unchanged bit for bit under every out_of_range policy, and a missing
value does not trigger the "warn" / "error" policies. The Preprocessor
numerical_imputation docstring and the missing-values guide no longer
claim that every numerical method raises on (or merely tolerates) a
missing value.

Fixes #87

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

B/M/I spline and two-feature thin-plate spline transformers encode a missing value as an all-zero row

1 participant