fix(preprocessor): route columns to the ColumnTransformer by label

[ChrisW09]

Fixes #55.

When a DataFrame's integer column labels did not match their positions, Preprocessor silently swapped columns. Data, per-feature methods from feature_preprocessing, names and lineage then referred to different columns. Non-contiguous labels, and swapped columns of different types, crashed. This hits pd.DataFrame(array) and read_csv(header=None) frames after a column is dropped, reordered or selected.

Root cause

build_column_transformer passed each column label to scikit-learn's ColumnTransformer as the column selector ([feature]). PreTab identifies columns by label everywhere else, but ColumnTransformer reads an int selector as a position.

Fix

• compose.feature_detection.with_string_labels converts every column label to its str form. It returns X itself when all labels are already strings, and otherwise a shallow relabelled copy, so the caller's frame is never modified. Labels that share a string form (1 and "1") raise PretabDataError.
• build_column_transformer selects each column by str(feature). Step names are unchanged.
• Preprocessor.fit and Preprocessor.transform pass self._column_transformer_input(X), which applies with_string_labels, to the internal ColumnTransformer.
• Preprocessor._input_labels maps the string labels back to the input labels, so output_dims_ and get_feature_info stay keyed by the original labels.
• Preprocessor._label_array_by_position keeps array input matched by position after a fit on non-string labels: a NumPy array passed to transform takes the fitted labels by position. Without it, the feature_0, feature_1, ... names that to_dataframe gives an array would fail with "columns are missing".

Behaviour changes

• Feature names and lineage output names for integer labels drop the input name scikit-learn generated. For example, pd.DataFrame(array) with labels in position order now gives num_0 instead of num_0__x0. Lineage source_features are unchanged (("0",)).
• column_transformer_ is fitted on the string labels. Its step columns are ["0"] instead of [0], and it now has a feature_names_in_ attribute (["0", "1"]), which it lacked on main.
• After a fit on integer labels, a DataFrame passed to transform must carry the fitted labels, in any order. On main, any frame of the same width was read by position. A frame with other labels now raises scikit-learn's ValueError ("columns are missing"). A NumPy array is still matched by position.
• After a fit on integer labels, get_feature_names_out(input_features) compares each name by its string form, so the labels seen at fit are accepted as given ([0, 1]) or as strings (["0", "1"]), and give the same names as get_feature_names_out(). Other names now raise ValueError: input_features is not equal to feature_names_in_, as scikit-learn does when it knows the input names; on main they renamed the inputs. The PR for Preprocessor.get_feature_names_out(input_features) fails after fitting on an ndarray, breaking Pipeline.get_feature_names_out() #65 (fix(preprocessor): accept input_features after fitting on a NumPy array #108) restores renaming.
• Mixed int / str labels used to fail inside scikit-learn ("Feature names are only supported if all input features have string names"). They now work.
• Labels that collide as strings (1 and "1") raise PretabDataError with a hint to rename the columns, instead of that scikit-learn TypeError.

Documentation

• Preprocessor.fit: new Raises section. PretabDataError is raised for a duplicate label (already the case on main) or for two labels with the same string form.
• Preprocessor.transform: the X description says that DataFrame and dict columns are matched by label, integer labels included. It also says that an array is matched by position after a fit on non-string labels.
• Preprocessor docstring, column_transformer_ attribute: notes that it is fitted on the string form of every label.
• The docstrings of with_string_labels (with a Raises section) and build_column_transformer explain why selectors are strings.
• No docs page changes. The README and the docs do not discuss integer labels or the ColumnTransformer selectors.

Tests

• test_integer_labels_route_each_column_to_its_own_block: on the issue's [1, 0] frame, each block holds its own column, and the caller's frame keeps its labels. Fails on main, passes here.
• test_feature_preprocessing_with_integer_keys_targets_the_labelled_column: {1: "minmax", 0: "none"} scales column 1 and passes column 0 through. Fails on main, passes here.
• test_non_contiguous_integer_labels_fit_and_transform: the issue's .drop(columns=[0]) frame fits, gives names num_1, num_2, and its first output column is the min-max of label 1. Fails on main, passes here.
• test_integer_labels_of_different_column_types_reach_their_own_pipeline: {1: list("abcabc"), 0: [1.0, ..., 6.0]} fits with explicit methods and with the defaults. The test checks exact min-max values and ordinal codes, and that column 0 is detected as numerical and 1 as categorical. Fails on main, passes here.
• test_integer_labels_are_matched_by_label_at_transform: a reordered frame gives the same num_1 block. Fails on main, passes here.
• test_array_input_after_an_integer_labelled_fit_is_matched_by_position: an ndarray passed after a fit on [1, 0] labels is matched by position, and each block holds the right column. Fails on main, passes here.
• test_integer_label_metadata_uses_the_input_labels: checks names num_1, num_0, output_dims_ == {1: 1, 0: 1}, get_feature_info keyed [1, 0], and the lineage pairs. Fails on main, passes here.
• test_labels_colliding_as_strings_are_rejected: 1 and "1" raise PretabDataError. Fails on main, passes here.
• test_with_string_labels_relabels_without_touching_the_input: [1, "b"] becomes ["1", "b"], the input is left untouched, and an all-string frame is returned as is. Fails on main (the function does not exist), passes here.
• test_get_feature_names_out_accepts_the_integer_labels_seen_at_fit: [0, 1] gives the same names as no input_features. Fails on main (where the names differ, num_0__0 vs. num_0__x0), passes here.

Reproduction

Output of the script from #55 on this branch. The failing fit(A) now prints its feature names. The last line is the mixed-type frame from the issue text, fitted with the defaults:

num_1: [10. 20. 30. 40. 50. 60.]
num_0: [1. 2. 3. 4. 5. 6.]
num_1 (minmax requested for label 1): [0.  0.2 0.4 0.6 0.8 1. ]
num_0 (passthrough requested for label 0): [1. 2. 3. 4. 5. 6.]
non-contiguous labels [1, 2]: ['num_1', 'num_2']
mixed types: numerical [0] categorical [1] -> cat_1 [1. 2. 3. 1. 2. 3.]

Checks

• just test: 1448 passed, 59 skipped, 7 xfailed; coverage 93.70% (gate 90%).
• Minimum dependencies (CI job: Python 3.10, numpy 1.24.4, pandas 2.0.3, scipy 1.10.1, scikit-learn 1.6.0): 1435 passed, 72 skipped, 7 xfailed; scripts/quickstart.py passes.
• just docs: strict build succeeds.
• just check: all pre-commit and pre-push hooks pass (ruff format, ruff lint, prettier, file hygiene, pyright with 0 errors).
• Import direction between subpackages is unchanged.

Merge notes

• Base: main.

• The PR for Preprocessor.fit crashes when a column name starts with '_' or contains '__' #71 (fix(preprocessor): keep ColumnTransformer step names valid for any label #111) edits the same lines, and the two conflict in three places:

  • build_column_transformer: keep Preprocessor.fit crashes when a column name starts with '_' or contains '__' #71's loop over _step_names, use this PR's [str(feature)] as the selector, and keep both docstring paragraphs.
  • Preprocessor docstring, column_transformer_ attribute: keep both paragraphs.
  • tests/integration/test_column_labels.py, added by both (add/add): keep the union. That is this PR's fixture, helpers and imports, plus Preprocessor.fit crashes when a column name starts with '_' or contains '__' #71's scikit-learn imports and tests, with the module docstrings merged.

  With the conflicts resolved this way, the full test suite passes (1456 passed, 59 skipped, 7 xfailed).

• The PR for Preprocessor.fit crashes on any bool column under default settings #54 (fix(preprocessor): encode boolean columns as binary categoricals #99) adds the same Preprocessor._column_transformer_input helper in the same place, and both PRs route fit and transform through it. Conflicts:

  • The helper's body and docstring. Make it return with_string_labels(bool_columns_as_object(X)) and merge both docstring sentences.
  • The feature_detection import line in preprocessor.py, and feature_detection.__all__.
  • The two new functions after to_dataframe.
  • The import line and the appended tests in tests/compose/test_feature_detection.py.

  Keep both sides in each. With the conflicts resolved this way, the full test suite passes (1464 passed, 59 skipped, 7 xfailed).

• When the PR for Preprocessor fitted on a DataFrame rejects a same-width NumPy array at transform (and vice versa) #78 (fix(preprocessor): match array input by position after a named fit #116) lands, its _align_input replaces _label_array_by_position. The transform lines and the helper conflict: keep _align_input, _column_transformer_input and _input_labels, drop _label_array_by_position, and take Preprocessor fitted on a DataFrame rejects a same-width NumPy array at transform (and vice versa) #78's description of X in transform. _align_input reads the fitted labels from column_transformer_.feature_names_in_, which are the string labels after this change. With the conflicts resolved this way, the full test suite passes (1450 passed, 59 skipped, 7 xfailed).

• The PR for Preprocessor.get_feature_names_out(input_features) fails after fitting on an ndarray, breaking Pipeline.get_feature_names_out() #65 (fix(preprocessor): accept input_features after fitting on a NumPy array #108) rewrites get_feature_names_out(input_features), so the two conflict there (the input_features docstring and the body of the method). Take Preprocessor.get_feature_names_out(input_features) fails after fitting on an ndarray, breaking Pipeline.get_feature_names_out() #65's side of both hunks: it accepts the fitted labels and new names on its own. With that resolution the full suite passes (1455 passed), and after an integer-label fit [0, 1] gives num_0, num_1 and ["a", "b"] gives num_a, num_b.

• The PRs for output_format="auto" chooses sparse vs dense per transform batch, breaking dense-only estimators at predict #73 (fix(preprocessor): resolve output_format='auto' once per fit #113), Preprocessor with one-hot encoding or preset="expanded" fails under set_config(transform_output="pandas") #76 (fix(preprocessor): keep the internal ColumnTransformer on default output #114) and Preprocessor crashes on an all-missing column at fit (numerical, one-hot) or silently drops it (int) #95 (fix(preprocessor): keep a block for columns with no observed value at fit #130) change the self.column_transformer_.fit(X, y) line in fit. Keep their change and pass self._column_transformer_input(X) to the ColumnTransformer. Preprocessor crashes on an all-missing column at fit (numerical, one-hot) or silently drops it (int) #95 also adds a PretabDataError condition to the fit Raises section: list both conditions under the one entry and keep its Warns section.

• The PR for Preprocessor.transform does not check n_features_in_ for ndarray input; extra columns are silently misaligned #72 (fix(preprocessor): reject NumPy input of the wrong width at transform #112) adds a width check where this PR adds is_array = ... in transform. Keep both, with the check first.

• The PR for Preprocessor rejects polars DataFrames, so pipelines with set_output(transform="polars") crash #94 (fix(preprocessor): accept polars DataFrames as input #129) adds has_column_labels next to with_string_labels in feature_detection (in __all__ and as a new function), and changes the X type line in transform. Keep both, i.e. Preprocessor rejects polars DataFrames, so pipelines with set_output(transform="polars") crash #94's type line with this PR's description.

• Follow-up with the PR for Preprocessor crashes on an all-missing column at fit (numerical, one-hot) or silently drops it (int) #95 (fix(preprocessor): keep a block for columns with no observed value at fit #130): its Preprocessor._raise_for_empty_features compares the ColumnTransformer's selectors with the input labels and indexes X with them. After this change, an integer-labelled frame skips its message. In a trial merge, labels [0, 1] with an all-missing Box-Cox column raise scikit-learn's ValueError: Data must not be constant. instead of Preprocessor crashes on an all-missing column at fit (numerical, one-hot) or silently drops it (int) #95's PretabDataError. Mapping the selector back with self._input_labels() and indexing self._column_transformer_input(X) restores the message. With that change, the full test suite passes on the trial merge (1518 passed, 59 skipped, 7 xfailed).

• Follow-up with the PRs for CrossFittedTransformer converts DataFrames to ndarray; a wrapped Preprocessor treats every column as categorical #58 (fix(cross-fitting): pass DataFrames to the wrapped transformer unchanged #103) and CrossFittedTransformer(Preprocessor) relearns column types and category codes per fold, unlike transform #79 (fix(cross-fitting): refit only target-aware Preprocessor blocks per fold #117): CrossFittedTransformer(Preprocessor) relearns column types and category codes per fold, unlike transform #79 adds Preprocessor._cross_fit_fold, which picks each target-aware block's columns out of the fold by the ColumnTransformer's selectors. After this change those selectors are strings. Once CrossFittedTransformer converts DataFrames to ndarray; a wrapped Preprocessor treats every column as categorical #58 lets CrossFittedTransformer pass a DataFrame through, an integer-labelled frame would reach _cross_fit_fold and raise a KeyError. _cross_fit_fold should take its input through self._column_transformer_input(X), as fit and transform do.

This change was previously part of #75, which is closed in favour of one pull request per issue.