fix(preprocessor): keep the missing indicator out of the representation

[ChrisW09]

Fixes #62.

With imputation enabled, add_missing_indicator=True and missing_policy="impute_with_indicator" fed the missing indicator through the scaler and the numerical or categorical method as a second input feature. Users got a standardized, basis-expanded, polynomial-crossed or 1/2-coded indicator instead of one 0/1 column, get_feature_lineage() raised for every spline and feature-map method, and get_feature_info() reported no dimension.

Root cause

PreprocessorConfig.imputation_plan sets add_indicator whenever the imputer runs with an indicator, and create_transformer passed it on as add_missing_indicator to get_numerical_transformer_steps / get_categorical_transformer_steps. These put SimpleImputer(add_indicator=True) first in the per-column Pipeline, so the imputer returned two columns (imputed value and indicator) and every later step was fitted on both. The inspection code assumes one input column per block: lineage passed a single column name to a leaf fitted on two inputs, and build_feature_info probed it with a (1, 1) array. Only the separate_state path built its indicator outside the representation, in a parallel FeatureUnion branch.

Fix

• create_transformer (pretab/compose/factory.py) no longer passes add_missing_indicator to the step builders, so the imputer runs without its indicator. When plan["add_indicator"] is set, it returns FeatureUnion([("representation", pipeline), ("missing", MissingIndicator(error_on_new=False))], verbose_feature_names_out=False), the same layout as separate_state. MissingIndicator is what SimpleImputer(add_indicator=True) uses internally, so the indicator keeps its semantics (a column only for features with missing values at fit) and its name (num_a__missingindicator_a).
• build_feature_lineage (pretab/compose/inspection.py) reads the width of the missing branch from the fitted indicator's get_feature_names_out, which may be zero columns, and assigns the rest of the block to the representation.
• The follow-up fix for the representation summary of indicator blocks is part of this change. These blocks are now a FeatureUnion, so _representation_summary (to_spec()["representations"], reproducibility_report()["representations"]) and Preprocessor._log_internal_decisions (the verbose=3 log), which read each block's last step, would have skipped them. The new helper representation_leaf() in pretab/compose/inspection.py (added to __all__) returns the last step of a union's "representation" branch, or of a plain pipeline, and both callers use it.

Behaviour changes

• Each feature with missing values at fit gets exactly one raw 0/1 indicator column next to its representation, with the same name as before. The representation columns are now those of plain imputation (missing_policy="impute"), so outputs of basis methods get narrower: in the issue's example, bspline with output_dim=5 gives 6 columns instead of 10, and polynomial has no indicator terms.
• get_feature_lineage() works for these blocks and labels indicator columns with family missing_state. get_feature_info() reports the block width, and its preprocessing string reads representation(imputer -> ...) + missing, as for separate_state.
• to_spec()["representations"] and reproducibility_report()["representations"] now also list separate_state blocks, which they never did. The fingerprint does not hash this summary.
• Newly fitted indicator blocks have a different fitted state, so their fingerprint_ differs from 1.0.0.
• Specs and pickles written by 1.0.0 with add_missing_indicator=True or missing_policy="impute_with_indicator" keep their old block layout and still load. Checked for bspline, ple, standardization with one-hot, and rbf with int, each from a pickle and from a JSON spec: the output is bit-for-bit identical to 1.0.0, and fingerprint_, to_spec()["representations"] and reproducibility_report() are unchanged. get_feature_lineage() on such a loaded object still raises for basis methods (bspline, rbf), as it did on 1.0.0, because the stored block still has the two-input leaf. Refitting fixes it.

Documentation

• Preprocessor docstring: add_missing_indicator now describes a 0/1 column for each imputed feature with missing values at fit, computed on the raw input next to the representation and never passed through the scaler or basis. The "impute_with_indicator" bullet of missing_policy refers to it.
• docs/core_concepts/missing_values.md: new paragraph saying that with imputation enabled the indicator is a raw 0/1 column computed before imputation, placed next to the representation, never scaled or expanded, and added only for features with missing values at fit.
• Docstrings of PreprocessorConfig.imputation_plan and _representation_summary, and of the new representation_leaf.

Tests

• test_imputer_indicator_is_a_raw_binary_column (with add_missing_indicator=True and with missing_policy="impute_with_indicator"): with standardization, the indicator column equals isna() as 0/1. Fails on main, passes here.
• test_imputer_indicator_adds_exactly_one_column_per_feature_with_missing[bspline, rbf, pspline, fourier, cubicspline, polynomial]: exactly one indicator per feature, the other names equal those of plain imputation, and output_dims_ is the plain width plus one. Fails on main, passes here.
• test_imputer_indicator_lineage_and_feature_info[bspline, rbf, pspline, fourier, cubicspline]: lineage covers every output column, the two indicators have family missing_state, and the get_feature_info dimensions equal output_dims_. Fails on main, passes here.
• test_imputer_indicator_on_categorical_integer_codes: int codes keep one 0/1 indicator. Fails on main, passes here.
• test_missing_state_blocks_report_their_representation (with add_missing_indicator=True, missing_policy="impute_with_indicator" and missing_policy="separate_state"): the summary lists bspline for both columns and equals that of plain imputation. Fails on main, passes here.
• test_verbose_3_logs_internal_decisions_of_missing_state_blocks (with add_missing_indicator=True and missing_policy="separate_state"): the PLE thresholds of these blocks are logged at verbose=3. The separate_state case fails on main, passes here. The add_missing_indicator case passes on main, where that block is still a pipeline; it pins that the new union is unwrapped.
• test_imputer_indicator_only_marks_features_with_missing_values_at_fit: no indicator column for a feature without missing values at fit, as with SimpleImputer(add_indicator=True). Guard test: passes on both by design.
• test_block_with_the_imputer_built_in_indicator_still_loads_and_reports: rebuilds the 1.0.0 block layout (imputer with add_indicator=True in front of the basis) and checks that it transforms, round-trips through a JSON spec with the same output and fingerprint, and lists its representations. Guard test: passes on both by design.

Reproduction

Output of the script from #62 on this branch (the get_feature_lineage() call is wrapped and prints the number of records and the last record):

standardization, indicator values: [0. 1.]
bspline width: 6 ['num_a_bs1', 'num_a_bs2', 'num_a_bs3', 'num_a_bs4', 'num_a__missingindicator_a']
get_feature_info dimension: 6
lineage: 6 records; last: ('num_a__missingindicator_a', 'missing_state')
polynomial: ['num_a__1', 'num_a', 'num_a__a^2', 'num_a__a^3', 'num_a__missingindicator_a']

Checks

• just test: 1459 passed, 59 skipped, 7 xfailed; coverage 93.92% (gate 90%).
• Minimum dependencies (CI job: Python 3.10, numpy 1.24.4, pandas 2.0.3, scipy 1.10.1, scikit-learn 1.6.0): 1446 passed, 72 skipped, 7 xfailed; scripts/quickstart.py passes.
• just docs: strict build succeeds.
• just check: all pre-commit and pre-push hooks pass (ruff format, ruff lint, prettier, file hygiene, pyright with 0 errors).
• Import direction between subpackages is unchanged.

Merge notes

• Base: main.
• The PR for Preprocessor.fit crashes when a column name starts with '_' or contains '__' #71 (fix(preprocessor): keep ColumnTransformer step names valid for any label #111) conflicts with this one in Preprocessor._log_internal_decisions and in tests/integration/test_verbosity.py, where both add a test after test_verbose_3_logs_internal_decisions. Keep Preprocessor.fit crashes when a column name starts with '_' or contains '__' #71's loop header and name = block_name(step_name, columns) together with this PR's last_step = representation_leaf(transformer), and keep both tests. With this branch, Preprocessor.fit crashes when a column name starts with '_' or contains '__' #71's and Preprocessor crashes on an all-missing column at fit (numerical, one-hot) or silently drops it (int) #95's combined this way, the full test suite passes (1539 passed, 59 skipped, 7 xfailed).
• The PR for Preprocessor crashes on an all-missing column at fit (numerical, one-hot) or silently drops it (int) #95 (fix(preprocessor): keep a block for columns with no observed value at fit #130) (keep_empty_features=True on the imputers) changes the same imputer construction in factory.py and the feature-info code in inspection.py; git merges the two cleanly. MissingIndicator still emits a column for an all-missing feature, so the two fixes fit together.
• The fix for Standalone transformers drop DataFrame feature names, so reordered columns are silently misaligned #85 rewrites _representation_summary (it passes each block's column names and no longer swallows errors). Whichever of the two PRs lands second gets a conflict in serialize.py and has to keep leaf = representation_leaf(transformer) inside Standalone transformers drop DataFrame feature names, so reordered columns are silently misaligned #85's rewritten loop.
• The fix for CrossFittedTransformer(Preprocessor(add_missing_indicator=True)) crashes when a fold's training rows contain no missing value #89 (cross-fitting a Preprocessor with missing indicators) builds on this change and on CrossFittedTransformer(Preprocessor) relearns column types and category codes per fold, unlike transform #79's per-fold refit: each fold refits only the representation branch of an indicator block and keeps the all-data indicator. Its PR needs this one first.
• None / pd.NA categorical missing values are not treated as missing (nullable dtypes crash, None becomes a category) #64 (None / pd.NA not treated as missing) now also decides which values MissingIndicator marks.

This change was previously part of #75, which is closed in favour of one pull request per issue.