fix(preprocessor): encode boolean columns as binary categoricals

[ChrisW09]

Fixes #54.

Any DataFrame or NumPy array with a plain bool column made Preprocessor.fit raise ValueError: SimpleImputer does not support data with dtype bool. This happened under the default settings, every categorical_method, every preset, and the imputing missing policies.

Root cause

detect_column_types classifies bool (dtype kind "b") as categorical. The categorical pipeline starts with SimpleImputer(strategy="most_frequent") whenever imputation is enabled, which is the default, and SimpleImputer rejects the bool dtype.

Fix

• compose.feature_detection.bool_columns_as_object casts boolean columns (bool and pandas' nullable boolean) to object with the values True / False, so they are encoded like any other binary categorical column. A missing value of the nullable boolean dtype becomes NaN, the missing marker the imputer and the encoders recognize. The function returns a shallow copy, so the caller's frame is unchanged. A frame without boolean columns is returned as is.
• The new Preprocessor._column_transformer_input applies it. fit and transform pass their frame through it to the internal ColumnTransformer. Column-type detection still sees the original dtypes.

Behaviour changes

• categorical_method="none" now passes a boolean column through as the Python objects True / False. The block holds those values, and the stacked output becomes an object array:
  • For plain bool, "none" only fitted on main with categorical_imputation=None or missing_policy="propagate". The column was then upcast to 1.0 / 0.0 in a float array.
  • For nullable boolean with the default imputer, main returned float 1.0 / 0.0. A pd.NA that is not imputed (categorical_imputation=None or missing_policy="propagate") made main raise. It now passes through as NaN.
  • With a configured dtype (for example dtype=np.float32), the output is still cast to 1.0 / 0.0. A frame with any string categorical column already produced an object array under "none".
• One-hot feature names for nullable boolean columns change from cat_flag_0.0 / cat_flag_1.0 to cat_flag_False / cat_flag_True. On main, the imputer had turned the column into floats. Without imputation, the missing category is still cat_flag_nan. The values are unchanged.
• pd.NA in a nullable boolean column is now treated as missing everywhere. With categorical_method="int" and missing_policy="propagate" or categorical_imputation=None, main raised TypeError: boolean value of NA is ambiguous. The value now gets the missing code 0.
• Configurations that already fitted on main keep identical names and values. That covers int, one-hot and onehot_from_ordinal on plain bool with categorical_imputation=None or missing_policy="propagate" / "error", and int / onehot_from_ordinal on nullable boolean.

Documentation

• README, "Automatic feature-type detection": boolean columns are categorical and are encoded as a binary categorical with the categories False and True.
• Preprocessor docstring, Notes: bool and nullable boolean columns are binary categoricals, and a nullable missing value is treated as missing.
• detect_column_types docstring: non-numeric dtypes, including bool and nullable boolean, are always categorical.
• New bool_columns_as_object docstring.

Tests

• test_bool_column_is_encoded_as_a_binary_categorical: 11 cases, covering every failing configuration from the issue:

  • the defaults;
  • categorical_method "one-hot", "onehot_from_ordinal" and "none";
  • presets "standard", "expanded" and "adaptive";
  • missing_policy "impute", "impute_with_indicator" and "separate_state";
  • treat_all_integers_as_numerical=True.

  For each, the test checks the exact trailing feature names, categorical_features_ == ["flag"] and the transform width. Fails on main, passes here.

• test_bool_column_integer_codes_follow_the_values: with categorical_method="int" the codes are exactly False -> 1 and True -> 2. Fails on main, passes here.

• test_nullable_boolean_column_with_missing_values_is_imputed: one-hot names are cat_flag_False / cat_flag_True, and the pd.NA row is imputed to True. Fails on main (names cat_flag_0.0 / cat_flag_1.0), passes here.

• test_nullable_boolean_missing_value_gets_the_missing_code_without_imputation: under missing_policy="propagate", the codes are [2, 1, 0, 1, 2, 2] with pd.NA mapped to 0. Fails on main (TypeError), passes here.

• test_bool_numpy_array_is_supported: a bool ndarray fits and transforms. Fails on main, passes here.

• test_bool_columns_as_object_casts_only_boolean_columns: checks the dtypes and values after the cast (NaN for pd.NA) and that the input keeps its dtypes. Fails on main (the function does not exist), passes here.

• test_bool_columns_as_object_returns_frame_without_bool_columns_unchanged: a frame without boolean columns is returned as the same object. Fails on main (the function does not exist), passes here.

Reproduction

Output of the script from #54 on this branch. It runs under the default settings, under each configuration the issue lists as also failing, and on a NumPy bool array. Each line shows the categorical feature names (all names for the array):

{}                                            ['cat_flag']
{'categorical_method': 'one-hot'}             ['cat_flag_False', 'cat_flag_True']
{'categorical_method': 'int'}                 ['cat_flag']
{'categorical_method': 'onehot_from_ordinal'} ['cat_flag_bin_0', 'cat_flag_bin_1']
{'categorical_method': 'none'}                ['cat_flag']
{'preset': 'standard'}                        ['cat_flag']
{'preset': 'expanded'}                        ['cat_flag_False', 'cat_flag_True']
{'preset': 'adaptive'}                        ['cat_flag']
{'missing_policy': 'impute'}                  ['cat_flag']
{'missing_policy': 'impute_with_indicator'}   ['cat_flag']
{'missing_policy': 'separate_state'}          ['cat_flag__representation__flag', 'cat_flag__missing__flag__missing']
{'treat_all_integers_as_numerical': True}     ['cat_flag']
bool ndarray                                  ['cat_feature_0']

Checks

• just test: 1455 passed, 59 skipped, 7 xfailed; coverage 93.71% (gate 90%).
• Minimum dependencies (CI job: Python 3.10, numpy 1.24.4, pandas 2.0.3, scipy 1.10.1, scikit-learn 1.6.0): 1442 passed, 72 skipped, 7 xfailed; scripts/quickstart.py passes.
• just docs: strict build succeeds.
• just check: all pre-commit and pre-push hooks pass (ruff format, ruff lint, prettier, file hygiene, pyright with 0 errors).
• Import direction between subpackages is unchanged.

Merge notes

• Base: main.

• The PR for Preprocessor routes integer column labels positionally, silently swapping columns #55 (fix(preprocessor): route columns to the ColumnTransformer by label #100) adds the same Preprocessor._column_transformer_input helper in the same place, and both PRs route fit and transform through it, so those call lines merge cleanly. Conflicts:

  • The helper's body and docstring. Make it return with_string_labels(bool_columns_as_object(X)) and merge both docstring sentences.
  • The feature_detection import line in preprocessor.py, and feature_detection.__all__.
  • The two new functions after to_dataframe.
  • The import line and the appended tests in tests/compose/test_feature_detection.py.

  Keep both sides in each. With the conflicts resolved this way, the full test suite passes (1464 passed, 59 skipped, 7 xfailed).

• The PRs for output_format="auto" chooses sparse vs dense per transform batch, breaking dense-only estimators at predict #73 (fix(preprocessor): resolve output_format='auto' once per fit #113), Preprocessor with one-hot encoding or preset="expanded" fails under set_config(transform_output="pandas") #76 (fix(preprocessor): keep the internal ColumnTransformer on default output #114) and Preprocessor crashes on an all-missing column at fit (numerical, one-hot) or silently drops it (int) #95 (fix(preprocessor): keep a block for columns with no observed value at fit #130) change the self.column_transformer_.fit(...) line in fit. Keep their change, and pass self._column_transformer_input(X) instead of X to the ColumnTransformer. Preprocessor crashes on an all-missing column at fit (numerical, one-hot) or silently drops it (int) #95 also adds a Notes paragraph next to this PR's boolean paragraph: keep both.

• The PR for Preprocessor fitted on a DataFrame rejects a same-width NumPy array at transform (and vice versa) #78 (fix(preprocessor): match array input by position after a named fit #116) adds _align_input at the place where this PR adds _column_transformer_input: keep both methods.

• The PR for Preprocessor rejects polars DataFrames, so pipelines with set_output(transform="polars") crash #94 (fix(preprocessor): accept polars DataFrames as input #129) adds has_column_labels next to bool_columns_as_object in feature_detection (in __all__ and as a new function): keep both.

• This PR and the PRs for Preprocessor(numerical_method="box-cox") raises at transform for values below the training minimum #59 (fix(preprocessor): keep box-cox finite below the training minimum #104), Preprocessor.get_feature_names_out(input_features) fails after fitting on an ndarray, breaking Pipeline.get_feature_names_out() #65 (fix(preprocessor): accept input_features after fitting on a NumPy array #108), Preprocessor.transform does not check n_features_in_ for ndarray input; extra columns are silently misaligned #72 (fix(preprocessor): reject NumPy input of the wrong width at transform #112), Preprocessor fitted on a DataFrame rejects a same-width NumPy array at transform (and vice versa) #78 (fix(preprocessor): match array input by position after a named fit #116), A 2-D y (multi-output target) crashes the B/M/I splines even when unsupervised, and target-aware methods with unrelated errors #88 (fix(splines): ignore y when unsupervised and reject a multi-output target #123) and Preprocessor rejects polars DataFrames, so pipelines with set_output(transform="polars") crash #94 (fix(preprocessor): accept polars DataFrames as input #129) all append tests at the end of tests/integration/test_preprocessor.py, so whichever lands later gets an append/append conflict there: keep both blocks.

• None / pd.NA categorical missing values are not treated as missing (nullable dtypes crash, None becomes a category) #64 (None / pd.NA in categorical columns) is still open. This PR only turns pd.NA in nullable boolean columns into NaN, and the fix for None / pd.NA categorical missing values are not treated as missing (nullable dtypes crash, None becomes a category) #64 should agree with bool_columns_as_object.

This change was previously part of #75, which is closed in favour of one pull request per issue.