fix(cross-fitting): pass DataFrames to the wrapped transformer unchanged

[ChrisW09]

Fixes #58.

CrossFittedTransformer converted X to a NumPy array before fitting the wrapped estimator, so a mixed-dtype DataFrame reached it as an object array without column names. A wrapped Preprocessor then treated every column as categorical and silently produced meaningless out-of-fold features, and estimators that select columns by name raised.

Root cause

_fit_full and transform called np.asarray(X), and fit_transform sliced the folds from that array (X_arr[train_idx], X_arr[test_idx]). A frame with a float and a string column becomes an object array; to_dataframe wraps it as-is and detect_column_types classifies every object column as categorical.

Fix

• New _as_2d(X) in pretab/core/supervised.py: a 2-D pandas DataFrame is returned unchanged; any other input is still converted with np.asarray, with 1-D input reshaped to one column. _fit_full and transform use it.
• New _take_rows(X, indices): selects rows by position (.iloc for a DataFrame). The fit_transform fold loop takes each fold's training and held-out rows with it.
• y is still converted with np.asarray(y).ravel().

Behaviour changes

• A pandas DataFrame reaches the wrapped estimator in fit, transform and on every fold as a DataFrame that keeps its column names, dtypes and index labels (a fold is a row subset, its index is not reset). Before, it was an ndarray, so a wrapped Preprocessor named its outputs after feature_0, feature_1, ...; it now uses the column names.
• Any other input is converted to a 2-D NumPy array, as before.
• Not changed: to_dataframe still wraps an object ndarray without infer_objects() (the optional second suggestion in the issue), because that would change type detection for every object-array input.

Documentation

• CrossFittedTransformer docstring, transformer parameter: it may also be an estimator such as a Preprocessor or Pipeline built from PreTab transformers, and a pandas DataFrame X is passed to it unchanged, each fold as a row subset.
• Docstrings of the new private helpers _as_2d and _take_rows.
• No docs page needed a change: docs/core_concepts/target_awareness.md and the README show NumPy examples only and do not describe the input conversion.

Tests

In tests/core/test_cross_fitted.py, with a mixed_frame fixture (a float and a string column, shuffled index):

• test_wrapped_preprocessor_keeps_dataframe_column_types: the wrapped Preprocessor detects num as numerical and city as categorical, its output width and feature names equal a direct fit, the out-of-fold PLE column holds more than one distinct value (a supervised encoding, not the "unseen category" code), and transform has the same shape. Fails on main, passes here.
• test_wrapped_column_transformer_selects_by_column_name: a wrapped ColumnTransformer that selects ["num"] by name cross-fits. Fails on main ("Specifying the columns using strings is only supported for dataframes"), passes here.
• test_wrapped_preprocessor_accepts_feature_preprocessing_by_column_name: with feature_preprocessing={"num": "ple"} and numerical_method="minmax", the outputs are num_num_ple0 ... num_num_ple3 and cat_city. Fails on main (InvalidParamError ... Valid values: ['feature_0', 'feature_1']), passes here.
• test_one_dimensional_input_is_still_a_single_column: 1-D input is still reshaped to one column. Guard test: passes on both by design.

Reproduction

Output of the script from #58 on this branch, with the two further effects the issue lists (feature_preprocessing={"num": "ple"} and a ColumnTransformer selecting ["num"]) added as separate calls:

direct : ['num'] ['city'] 7
wrapped: ['num'] ['city'] 7
names  : ['num_num_ple0' 'num_num_ple1' 'num_num_ple2' 'num_num_ple3'
 'num_num_ple4' 'num_num_ple5' 'cat_city']
distinct OOF values in column 0: [0.         0.07564504 0.1530339  0.39430624 0.43366897 0.4538359
 0.469118   0.65073645 0.71018094 0.71739459 0.73780364 0.95519274
 0.99169701 0.9927367  1.        ]
feature_preprocessing={'num': 'ple'}: (300, 7)
ColumnTransformer by name: (300, 6)

Checks

• just test: 1442 passed, 59 skipped, 7 xfailed; coverage 93.73% (gate 90%).
• Minimum dependencies (CI job: Python 3.10, numpy 1.24.4, pandas 2.0.3, scipy 1.10.1, scikit-learn 1.6.0): 1429 passed, 72 skipped, 7 xfailed; scripts/quickstart.py passes.
• just docs: strict build succeeds.
• just check: all pre-commit and pre-push hooks pass (ruff format, ruff lint, prettier, file hygiene, pyright with 0 errors).
• Import direction between subpackages is unchanged.

Merge notes

• Base: main.
• The PRs for CrossFittedTransformer(Preprocessor) relearns column types and category codes per fold, unlike transform #79 (fix(cross-fitting): refit only target-aware Preprocessor blocks per fold #117), CrossFittedTransformer(Preprocessor(add_missing_indicator=True)) crashes when a fold's training rows contain no missing value #89 (fix(cross-fitting): keep missing indicators of a wrapped Preprocessor #124) (stacked on CrossFittedTransformer(Preprocessor) relearns column types and category codes per fold, unlike transform #79) and CrossFittedTransformer.fit_transform forces a dense float64 array: string and sparse outputs crash #90 rewrite the same fit_transform fold loop and append tests at the end of tests/core/test_cross_fitted.py, so expect textual conflicts. Keep every test section, and take the fold rows with _take_rows:
  • with CrossFittedTransformer(Preprocessor) relearns column types and category codes per fold, unlike transform #79: fold = self._fit_fold(_take_rows(X_arr, train_idx), y_arr[train_idx]) and fold.transform(_take_rows(X_arr, test_idx)). Preprocessor._cross_fit_fold then receives DataFrame folds with their original index; it applies to_dataframe and _safe_indexing, so it needs no change.
  • with CrossFittedTransformer.fit_transform forces a dense float64 array: string and sparse outputs crash #90: keep CrossFittedTransformer.fit_transform forces a dense float64 array: string and sparse outputs crash #90's fold collection and _stack_folds(blocks, test_indices, X_arr), which then receives the DataFrame and keeps its index for scikit-learn set_output frames. Whichever of this PR and the PR for CrossFittedTransformer.fit_transform forces a dense float64 array: string and sparse outputs crash #90 (fix(cross-fitting): return transform's output type from fit_transform #125) lands second can add a test for that index (mixed_frame with a name-based ColumnTransformer and set_output(transform="pandas")).
  • With CrossFittedTransformer converts DataFrames to ndarray; a wrapped Preprocessor treats every column as categorical #58, CrossFittedTransformer(Preprocessor) relearns column types and category codes per fold, unlike transform #79, CrossFittedTransformer(Preprocessor(add_missing_indicator=True)) crashes when a fold's training rows contain no missing value #89 and CrossFittedTransformer.fit_transform forces a dense float64 array: string and sparse outputs crash #90 combined this way, the full test suite passes (1473 passed, 59 skipped, 8 xfailed).
• The PR for An invalid task such as "Regression" is silently treated as classification by the splines and selectors #93 (fix(validation): reject an unknown task instead of using classification #128) conflicts in CrossFittedTransformer._fit_full: keep its validate_task(self.task, type(self).__name__) together with X_arr = _as_2d(X).
• Follow-ups once other fixes land:
  • Preprocessor rejects polars DataFrames, so pipelines with set_output(transform="polars") crash #94 (polars input): add if is_polars_frame(X): return polars_to_pandas(X) to _as_2d, so a polars frame is converted once, before the fold rows are taken. That PR lists the cross-fitting tests to add.
  • Preprocessor.get_feature_names_out(input_features) fails after fitting on an ndarray, breaking Pipeline.get_feature_names_out() #65 and Standalone transformers drop DataFrame feature names, so reordered columns are silently misaligned #85: once a Preprocessor fitted on a frame records feature_names_in_ (Preprocessor.get_feature_names_out(input_features) fails after fitting on an ndarray, breaking Pipeline.get_feature_names_out() #65), CrossFittedTransformer.get_representation_spec() without names asks the wrapped Preprocessor for the outputs of x0, x1, ..., which it rejects. Whichever of CrossFittedTransformer converts DataFrames to ndarray; a wrapped Preprocessor treats every column as categorical #58, Preprocessor.get_feature_names_out(input_features) fails after fitting on an ndarray, breaking Pipeline.get_feature_names_out() #65 and Standalone transformers drop DataFrame feature names, so reordered columns are silently misaligned #85 lands last defaults input_features to getattr(self.estimator_, "feature_names_in_", None) there, with tests.

This change was previously part of #75, which is closed in favour of one pull request per issue.