[ChrisW09]

Fixes #89.

Cross-fitting a Preprocessor with add_missing_indicator=True or missing_policy="impute_with_indicator" raised a width-mismatch error whenever a fold's training rows held no missing value for a column that has missing values, which always happens for a column with a single missing value. On this branch alone, target-aware splines (bspline, ispline, mspline, cubicspline, naturalspline) with a missing indicator still raise, pinned by a strict xfail test, because main still feeds the indicator into the representation; the PR for #62 (#106) removes that, and with #62, #79 and this PR combined every method works with no further code change (see Merge notes).

Root cause

The indicator comes from the block's SimpleImputer(add_indicator=True), which only emits a column for features that had missing values in the rows it was fit on. On main the whole Preprocessor is refit per fold, so every method failed. With the PR for #79 (#117) (the base of this branch), blocks that do not use y are shared with the all-data fit, but a target-aware block is still refit as a whole, imputer included, so PLE, the feature maps, the target-aware splines and registered target-using encoders still lost the indicator column on such a fold.

Fix

• refit_block_representation(transformer, X, y) (new, pretab/compose/inspection.py) returns a copy of a fitted block in which only the representation step and the steps after it are refit. The steps before it (the imputer with its indicator, scalers) keep their all-data fit and transform the fold rows for the refit. For a separate-state union it keeps the missing branch and recurses into representation. A block that is not a pipeline, or whose first step is the representation, is cloned and refit as before. The fitted block itself is not modified.
• _locate_block_representation (new) also returns the representation step's index; _resolve_block_representation now wraps it.
• Preprocessor._cross_fit_fold refits each target-aware block with refit_block_representation.
• The width-mismatch message describes the general cause instead of adaptive sizing.

Behaviour changes

• In a target-aware block of a wrapped Preprocessor, the imputer (its median / most-frequent statistics and its indicator) and any scaler now come from the all-data fit on every fold; only the representation is refit on the fold's training rows. This also applies to separate-state blocks and to blocks without an indicator. These steps never use y, so nothing leaks, and a held-out missing value is now imputed with the all-data statistic, as transform imputes it. On a PLE column without missing values, the out-of-fold values are the same as with the whole-block refit up to float32 rounding.
• The width-mismatch IncompatibleParamsError has a new message (same type): "... The wrapped transformer's width depends on the rows it is fit on (e.g. adaptive sizing, or category / indicator columns learned from the data); configure it to produce the same columns on every fold."
• Target-aware splines (bspline, ispline, mspline, cubicspline, naturalspline with the default target_aware=True) with a missing indicator and a fold without missing values now raise PretabDataError: Feature at index 1 is constant ... instead of the width error. On main's block structure the imputer feeds its indicator to the spline as a second input (add_missing_indicator / impute_with_indicator feeds the indicator through the scaler and basis; lineage crashes #62); the all-data indicator is constant on such a fold, and a spline basis cannot be built on a constant input. This cannot be fixed here without moving the indicator out of the representation, which is the fix for add_missing_indicator / impute_with_indicator feeds the indicator through the scaler and basis; lineage crashes #62.

Documentation

• docs/core_concepts/target_awareness.md: the cross-fitting paragraph added by CrossFittedTransformer(Preprocessor) relearns column types and category codes per fold, unlike transform #79 now lists imputers with their missing-value indicators and scalers among the parts shared with the all-data fit.
• CrossFittedTransformer docstring, transformer parameter: only the target-aware representations of a Preprocessor are refit per fold; imputers with their indicators and scalers come from the all-data fit.
• Docstrings of refit_block_representation, _locate_block_representation and the private Preprocessor._cross_fit_fold.

Tests

Every new and changed test below fails on main and on the base branch (#79's), and passes here, except the strict xfail as noted.

tests/core/test_cross_fitted.py, with an array_with_one_missing_value fixture (one NaN, so one fold trains on no missing value):

• test_wrapped_preprocessor_keeps_its_missing_indicator_columns[ple-missing0|ple-missing1|rbf-missing2|relu-missing3|sigmoid-missing4|tanh-missing5]: PLE with add_missing_indicator=True and with missing_policy="impute_with_indicator", and rbf, relu, sigmoid and tanh with add_missing_indicator=True. The output has transform's shape, the indicator columns are kept, and the value's representation columns equal the out-of-fold output with missing_policy="impute". The base fails with the width error ("expected 12, got 8" for PLE).
• test_wrapped_preprocessor_refits_only_the_representation_after_the_imputer: exact values. The PLE block's 8 columns (value and indicator, 4 PLE columns each) equal the all-data imputer and scaler followed by a PLE refit on each fold. This test asserts main's block structure (imputer, minmax, ple).
• test_wrapped_preprocessor_keeps_the_indicator_next_to_a_target_aware_spline: xfail(strict=True, raises=PretabDataError), with a reason pointing to add_missing_indicator / impute_with_indicator feeds the indicator through the scaler and basis; lineage crashes #62; it cross-fits a target-aware bspline with an indicator and pins the remaining limitation. On the base it fails with the width error instead, so it is reported as a failure there.
• test_wrapped_preprocessor_keeps_the_separate_state_missing_branch: with missing_policy="separate_state", the representation branch equals the all-data imputer and scaler plus a per-fold PLE (exact), and the __missing column equals isnan. On the base the fold's imputer filled the held-out NaN with the fold median.
• test_wrapped_preprocessor_target_aware_blocks_stay_out_of_fold (changed from CrossFittedTransformer(Preprocessor) relearns column types and category codes per fold, unlike transform #79): now expects the all-data imputer and scaler plus a per-fold PLE, with exact equality. On the base it differs by float32 rounding.
• test_width_mismatch_names_the_general_cause (renamed from CrossFittedTransformer(Preprocessor) relearns column types and category codes per fold, unlike transform #79's test_width_mismatch_message_does_not_blame_adaptive_sizing_only): matches "same columns on every fold".

tests/extension/test_registered_cross_fitting.py:

• test_registered_target_encoder_keeps_its_missing_indicator: a registered TargetEncoder with an indicator keeps both output columns and is refit per fold. The base fails with the width error ("expected 2, got 1").

Reproduction

Output of the script from #89 on this branch:

{'add_missing_indicator': True} | Preprocessor alone: (200, 12)
   CrossFittedTransformer: (200, 12)
{'missing_policy': 'impute_with_indicator'} | Preprocessor alone: (200, 12)
   CrossFittedTransformer: (200, 12)
{'missing_policy': 'separate_state'} | Preprocessor alone: (200, 10)
   CrossFittedTransformer: (200, 10)

The same data with add_missing_indicator=True, output_dim=6 and other numerical methods, on this branch. The target-aware splines are the remaining limitation described above; on the base branch every method except minmax fails with the width error:

minmax         fit_transform: (200, 3) | transform: (200, 3)
rbf            fit_transform: (200, 18) | transform: (200, 18)
relu           fit_transform: (200, 18) | transform: (200, 18)
sigmoid        fit_transform: (200, 18) | transform: (200, 18)
tanh           fit_transform: (200, 18) | transform: (200, 18)
bspline        fit_transform -> PretabDataError: Feature at index 1 is constant (all values equal -1.0); a spline basis cannot be constructed on a zero-range feature.
ispline        fit_transform -> PretabDataError: Feature at index 1 is constant (all values equal -1.0); a spline basis cannot be constructed on a zero-range feature.
mspline        fit_transform -> PretabDataError: Feature at index 1 is constant (all values equal -1.0); a spline basis cannot be constructed on a zero-range feature.
cubicspline    fit_transform -> PretabDataError: Feature is constant (all values equal -1.0); a spline basis cannot be constructed on a zero-range feature.
naturalspline  fit_transform -> PretabDataError: Feature is constant (all values equal -1.0); a spline basis cannot be constructed on a zero-range feature.

With the PR for #62 (#106) added, every row of this table, the splines included, gives the same shape for fit_transform and transform ((200, 13) for all but minmax).

Checks

• just test: 1461 passed, 59 skipped, 8 xfailed (the additional xfail is the strict spline case); coverage 93.76% (gate 90%).
• Minimum dependencies (CI job: Python 3.10, numpy 1.24.4, pandas 2.0.3, scipy 1.10.1, scikit-learn 1.6.0): 1448 passed, 72 skipped, 8 xfailed; scripts/quickstart.py passes.
• just docs: strict build succeeds.
• just check: all pre-commit and pre-push hooks pass (ruff format, ruff lint, prettier, file hygiene, pyright with 0 errors).
• Import direction between subpackages is unchanged.

Merge notes

• Stacked on CrossFittedTransformer(Preprocessor) relearns column types and category codes per fold, unlike transform #79's branch fix/79-crossfit-preprocessor-encoding, because it changes the per-block refit (Preprocessor._cross_fit_fold, block_uses_target) that the PR for CrossFittedTransformer(Preprocessor) relearns column types and category codes per fold, unlike transform #79 (fix(cross-fitting): refit only target-aware Preprocessor blocks per fold #117) introduces. GitHub shows only this issue's commit, and the PR must be retargeted to main after the PR for CrossFittedTransformer(Preprocessor) relearns column types and category codes per fold, unlike transform #79 (fix(cross-fitting): refit only target-aware Preprocessor blocks per fold #117) merges.
• Merge order: after the PRs for CrossFittedTransformer(Preprocessor) relearns column types and category codes per fold, unlike transform #79 (fix(cross-fitting): refit only target-aware Preprocessor blocks per fold #117) and add_missing_indicator / impute_with_indicator feeds the indicator through the scaler and basis; lineage crashes #62 (fix(preprocessor): keep the missing indicator out of the representation #106). Once add_missing_indicator / impute_with_indicator feeds the indicator through the scaler and basis; lineage crashes #62 is on main, this branch needs two test changes and no code change:
  • Remove the xfail marker from test_wrapped_preprocessor_keeps_the_indicator_next_to_a_target_aware_spline, which then passes (a strict XPASS fails the suite).
  • Rewrite test_wrapped_preprocessor_refits_only_the_representation_after_the_imputer, which asserts main's pipeline structure. After add_missing_indicator / impute_with_indicator feeds the indicator through the scaler and basis; lineage crashes #62 the block is a FeatureUnion: compute the expectation from dict(block.transformer_list)["representation"] with _refit_representation_per_fold, as test_wrapped_preprocessor_keeps_the_separate_state_missing_branch does, and check that the indicator is one raw column equal to np.isnan(X[:, 0]).
  • With add_missing_indicator / impute_with_indicator feeds the indicator through the scaler and basis; lineage crashes #62, CrossFittedTransformer(Preprocessor) relearns column types and category codes per fold, unlike transform #79 and this branch combined, refit_block_representation recognises add_missing_indicator / impute_with_indicator feeds the indicator through the scaler and basis; lineage crashes #62's union through _separate_state_branches, keeps its fitted missing branch and refits only the representation step inside it. Only the import lists in pretab/compose/inspection.py and pretab/preprocessor.py conflict (keep both names). The full test suite then has exactly these two failures (1481 passed, 59 skipped, 7 xfailed otherwise).
• The PR for CrossFittedTransformer.fit_transform forces a dense float64 array: string and sparse outputs crash #90 (fix(cross-fitting): return transform's output type from fit_transform #125) rewrites the fold loop and keeps main's width-mismatch message; when both land, take this PR's message. With CrossFittedTransformer converts DataFrames to ndarray; a wrapped Preprocessor treats every column as categorical #58, CrossFittedTransformer(Preprocessor) relearns column types and category codes per fold, unlike transform #79, CrossFittedTransformer(Preprocessor(add_missing_indicator=True)) crashes when a fold's training rows contain no missing value #89 and CrossFittedTransformer.fit_transform forces a dense float64 array: string and sparse outputs crash #90 combined, the full test suite passes (1473 passed, 59 skipped, 8 xfailed).
• The PRs for Preprocessor.get_feature_names_out(input_features) fails after fitting on an ndarray, breaking Pipeline.get_feature_names_out() #65 (fix(preprocessor): accept input_features after fitting on a NumPy array #108), Preprocessor.fit crashes when a column name starts with '_' or contains '__' #71 (fix(preprocessor): keep ColumnTransformer step names valid for any label #111) and Standalone transformers drop DataFrame feature names, so reordered columns are silently misaligned #85 (fix(transformers): record and check DataFrame feature names #135) also edit the imports or __all__ at the top of pretab/compose/inspection.py (Preprocessor.get_feature_names_out(input_features) fails after fitting on an ndarray, breaking Pipeline.get_feature_names_out() #65 adds copy, Any and Pipeline as well; Preprocessor.fit crashes when a column name starts with '_' or contains '__' #71 also adds block_name to the import list in pretab/preprocessor.py). Keep the union of the imports and names.

This change was previously part of #97, which is closed in favour of one pull request per issue.