fix(preprocessor): accept polars DataFrames as input

[ChrisW09]

Fixes #94.

Preprocessor.fit and transform failed with an AttributeError on a polars DataFrame. Every scikit-learn Pipeline with set_output(transform="polars") and a step before the Preprocessor therefore crashed, and so did RepresentationSearchCV.fit on a polars frame.

Root cause

to_dataframe treated every input that was not a dict or a NumPy array as a pandas DataFrame. A polars frame's .columns is a list, so the duplicate-label check (.columns.duplicated()) failed at fit, and a polars frame has no .copy(), so transform failed too. RepresentationSearchCV.fit took fold rows with _row_subset, which turns anything without .iloc into a NumPy array. The folds therefore ran on unnamed arrays (an object array for mixed-type data, so every column became categorical), and the final refit on the polars frame hit the to_dataframe failure.

Fix

• pretab/core/validation.py: new is_polars_frame, which detects a polars frame through sys.modules and never imports polars. Also new polars_to_pandas, which converts column by column with Series.to_numpy, so pyarrow is not needed (polars.DataFrame.to_pandas requires it). Names and order are kept. Integer, float and boolean columns keep their NumPy dtype, and an integer column with nulls becomes float, as in pandas. String, categorical and enum columns, and boolean columns with nulls, become object columns, with nulls mapped to NaN.
• to_dataframe (pretab/compose/feature_detection.py) converts a polars frame with polars_to_pandas. Input without column labels (new has_column_labels) is read with np.asarray and must be 2D, so a list of rows works like an array. A pandas Series is rejected with a PretabDataError.
• RepresentationSearchCV.fit converts a polars frame once, before taking fold rows. Every fold then sees the column types of the whole frame, as the refit does. Converting each fold's rows separately could give a different dtype, for example an integer column whose only null lies outside the fold.

Behaviour changes

• A list of rows is accepted like a 2D array (columns feature_0, feature_1, ...). It used to fail with an AttributeError.
• A pandas Series, a 1D array or a flat list raises PretabDataError ("Expected 2D input (a DataFrame or a 2D array), got ...", with a Fix: hint) instead of an AttributeError or IndexError.
• New helpers has_column_labels (in pretab.compose.feature_detection.__all__) and is_polars_frame / polars_to_pandas (in pretab.core.validation.__all__).

Documentation

• docs/core_concepts/configuration.md: new "Polars DataFrame input" section. It covers where polars input is accepted (Preprocessor, RepresentationSearchCV), the dtype mapping and column-type detection, nulls as missing values, that no pyarrow is needed and that temporal columns are not supported. It ends with a Pipeline / set_output(transform="polars") example.
• docs/getting_started/installation.md: the polars extra note says a polars DataFrame is also accepted as input and links that section.
• README.md (two places) and docs/homepage.md: the supported input types include polars.DataFrame.
• Docstrings: the feature_detection module, to_dataframe (including Raises), has_column_labels, is_polars_frame, polars_to_pandas and RepresentationSearchCV.fit. In Preprocessor, the class feature list and the X type of fit, transform, fit_transform, estimate_output_shape and estimate_memory.

Tests

• test_to_dataframe_converts_polars_frame_with_names_order_and_dtypes: names, order, exact dtypes and null-to-NaN counts for Float64, Int64 (with and without null), Boolean (with and without null), String and Categorical columns. Fails on main, passes here.
• test_detect_column_types_on_polars_frame: float and high-cardinality integer columns are numerical. Low-cardinality integer, String, Boolean and Categorical columns are categorical. Fails on main, passes here.
• test_to_dataframe_reads_a_list_of_rows_as_an_array: a list of rows becomes feature_0, feature_1. Fails on main, passes here.
• test_to_dataframe_rejects_input_that_is_not_2d[Series, 1D array, flat list]: PretabDataError with the 2D message. Fails on main, passes here.
• test_polars_input_matches_pandas_input[minmax, bspline, ple + one-hot]: a polars frame gives the same detected column types, output names, and transform / fit_transform values as the pandas frame. A pandas-fitted Preprocessor transforms the polars frame identically, and a reordered polars frame is matched by name. Fails on main, passes here.
• test_polars_input_with_polars_output: polars in, polars out, with the fitted output names and width. Fails on main, passes here.
• test_polars_nulls_are_missing_values: a polars null is imputed and gets a missing indicator, not a "None" category. Fails on main, passes here.
• test_representation_search_on_polars_matches_pandas: cv_results_, best_method_ and predictions equal those for the pandas frame. Fails on main, and also with only the to_dataframe change. Passes here.
• test_search_converts_a_polars_frame_once: with an Int64 column holding a single null, the search scores equal those for the pandas frame. This pins the single conversion before the folds are taken. Fails on main, passes here.
• test_a_list_of_rows_is_read_like_an_array: a Preprocessor fitted on a list of rows names its columns feature_0, feature_1 and transforms the list like the array. Fails on main, passes here.
• test_pipeline_set_output_hands_frames_to_the_preprocessor[polars]: a SimpleImputer → Preprocessor pipeline returns a polars frame with columns num_income, num_visits. Fails on main, passes here. [pandas] is the same check for pandas output. Guard test: passes on both by design.

Reproduction

Output of the script from #94 on this branch. The Preprocessor.fit call prints ok when it succeeds. Two calls are added for the other failures the issue describes: transform of a polars frame after a pandas fit, and RepresentationSearchCV.fit on a polars frame, compared with pandas. On main the last four lines are AttributeErrors. The environment has no pyarrow.

sklearn ColumnTransformer, polars input -> (200, 2)
Pipeline.set_output('pandas') -> pandas.core.frame
Pipeline.set_output('polars') -> polars.dataframe.frame
Preprocessor.fit(polars DataFrame) -> ok
pandas fit, transform(polars DataFrame) -> equal to transform(pandas): True
RepresentationSearchCV.fit(polars DataFrame) -> best_method_: minmax | cv_results_ equal to pandas: True

Checks

• just test: 1454 passed, 59 skipped, 7 xfailed; coverage 93.70% (gate 90%).
• Minimum dependencies (CI job: Python 3.10, numpy 1.24.4, pandas 2.0.3, scipy 1.10.1, scikit-learn 1.6.0): 1431 passed, 76 skipped, 7 xfailed; scripts/quickstart.py passes.
• just docs: strict build succeeds.
• just check: all pre-commit and pre-push hooks pass (ruff format, ruff lint, prettier, file hygiene, pyright with 0 errors).
• Import direction between subpackages is unchanged.

Merge notes

• Base: main.
• Textual conflicts with other per-issue PRs. All are resolved by keeping both sides:
  • tests/integration/test_preprocessor.py: the PRs for Preprocessor.fit crashes on any bool column under default settings #54 (fix(preprocessor): encode boolean columns as binary categoricals #99), Preprocessor(numerical_method="box-cox") raises at transform for values below the training minimum #59 (fix(preprocessor): keep box-cox finite below the training minimum #104), Preprocessor.get_feature_names_out(input_features) fails after fitting on an ndarray, breaking Pipeline.get_feature_names_out() #65 (fix(preprocessor): accept input_features after fitting on a NumPy array #108), Preprocessor.transform does not check n_features_in_ for ndarray input; extra columns are silently misaligned #72 (fix(preprocessor): reject NumPy input of the wrong width at transform #112), Preprocessor fitted on a DataFrame rejects a same-width NumPy array at transform (and vice versa) #78 (fix(preprocessor): match array input by position after a named fit #116) and A 2-D y (multi-output target) crashes the B/M/I splines even when unsupervised, and target-aware methods with unrelated errors #88 (fix(splines): ignore y when unsupervised and reject a multi-output target #123) also append a block at the end of the file. Keep every block.
  • pretab/compose/feature_detection.py, __all__: this PR adds has_column_labels (with the function below it), the PR for Preprocessor.fit crashes on any bool column under default settings #54 (fix(preprocessor): encode boolean columns as binary categoricals #99) adds bool_columns_as_object and the PR for Preprocessor routes integer column labels positionally, silently swapping columns #55 (fix(preprocessor): route columns to the ColumnTransformer by label #100) adds with_string_labels. Keep all the names and functions.
  • pretab/core/validation.py, imports, __all__ and the new functions after it: the PR for Standalone transformers drop DataFrame feature names, so reordered columns are silently misaligned #85 (fix(transformers): record and check DataFrame feature names #135) adds resolve_input_features and extends the sklearn.utils.validation import, and the PR for A 2-D y (multi-output target) crashes the B/M/I splines even when unsupervised, and target-aware methods with unrelated errors #88 (fix(splines): ignore y when unsupervised and reject a multi-output target #123) adds single_target. Keep all of them, plus this PR's import sys / import pandas as pd.
  • Preprocessor.transform docstring: the PRs for Preprocessor routes integer column labels positionally, silently swapping columns #55 (fix(preprocessor): route columns to the ColumnTransformer by label #100) and Preprocessor fitted on a DataFrame rejects a same-width NumPy array at transform (and vice versa) #78 (fix(preprocessor): match array input by position after a named fit #116) each extend the description of X. Keep their text, with this PR's polars.DataFrame in the type line.
  • docs/core_concepts/configuration.md: the PR for Preprocessor fitted on a DataFrame rejects a same-width NumPy array at transform (and vice versa) #78 (fix(preprocessor): match array input by position after a named fit #116) adds a paragraph on positional matching at the place where this PR starts its "Polars DataFrame input" section. Keep Preprocessor fitted on a DataFrame rejects a same-width NumPy array at transform (and vice versa) #78's paragraph first (it continues the array-input text) and the new section after it.
  • RepresentationSearchCV.fit docstring: the PR for RepresentationSearchCV cannot use group-based CV splitters and accepts no groups or fit parameters #91 (fix(search): pass groups to the splitter and fit params to the estimator #126) adds Parameters / Returns sections. Keep them, with X : pandas.DataFrame, polars.DataFrame or array-like of shape (n_samples, n_features), and keep this PR's paragraph on the polars conversion.
• Preprocessor pandas output (set_output) drops the input index, breaking ColumnTransformer/FeatureUnion #60 (PR fix(output): preserve input index in set_output pandas container #68) and Preprocessor with one-hot encoding or preset="expanded" fails under set_config(transform_output="pandas") #76 are other output-container issues. They share no code with this change and merge cleanly. With fix(output): preserve input index in set_output pandas container #68, a converted polars frame gets the default RangeIndex in pandas output, because polars has no index.
• Follow-ups once other fixes land:
  • CrossFittedTransformer converts DataFrames to ndarray; a wrapped Preprocessor treats every column as categorical #58: CrossFittedTransformer turns pandas and polars frames alike into NumPy arrays on main, so it is not changed here. After CrossFittedTransformer converts DataFrames to ndarray; a wrapped Preprocessor treats every column as categorical #58, its _as_2d passes pandas frames through, so add if is_polars_frame(X): return polars_to_pandas(X) there. _as_2d runs once before the fold rows are taken, so the frame is converted once. Then name CrossFittedTransformer in the configuration.md section and add test_cross_fitted_preprocessor_on_polars_matches_pandas and test_cross_fitting_converts_a_polars_frame_once.
  • Preprocessor fitted on a DataFrame rejects a same-width NumPy array at transform (and vice versa) #78: _align_input decides positional matching with isinstance(X, np.ndarray). Use not has_column_labels(X) there, so a list of rows passed to transform after a frame fit is matched by position like an array. Add test_a_list_of_rows_is_matched_by_position_after_a_frame_fit, and add the sentence "A frame or dict is matched by column name, an array (or a list of rows) by position" to the fit docstring.
  • Preprocessor.get_feature_names_out(input_features) fails after fitting on an ndarray, breaking Pipeline.get_feature_names_out() #65: fit records feature_names_in_ unless isinstance(X, np.ndarray), so a list of rows would record the synthetic feature_0, ... names. Use not has_column_labels(X) there as well, and add the assertions list(from_polars.feature_names_in_) == polars_frame.columns (polars test) and not hasattr(pre, "feature_names_in_") (list-of-rows test).
  • Preprocessor.fit crashes on any bool column under default settings #54: on main a boolean column fails in SimpleImputer for pandas and polars input alike, so the polars test fixture uses String and Categorical columns. Once Preprocessor.fit crashes on any bool column under default settings #54 lands, add a polars Boolean column to the fixture.

This change was previously part of #97, which is closed in favour of one pull request per issue.