Skip to content

fix(preprocessor): keep box-cox finite below the training minimum - #104

Merged
mkumar73 merged 2 commits into
mainfrom
fix/59-boxcox-unseen-low-values
Oct 18, 2026
Merged

mkumar73 merged 2 commits into
mainfrom
fix/59-boxcox-unseen-low-values

Conversation

@ChrisW09

@ChrisW09 ChrisW09 commented Oct 16, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #59.

Preprocessor(numerical_method="box-cox") raised a ValueError at transform for any value more than about 0.1% of the training range below the training minimum. This broke prediction on new data and plain cross-validation, where three of five folds scored nan in the issue's example.

Root cause

Box-Cox needs strictly positive input, so get_numerical_transformer_steps puts a MinMaxScaler(feature_range=(1e-3, 1)), fitted on the training data, in front of PowerTransformer(method="box-cox"). With the default clip=False, the scaler maps a value far enough below the training minimum to <= 0, and PowerTransformer.transform rejects it. Positivity was guaranteed only for in-sample values.

Fix

  • New private _PositiveMinMaxScaler in pretab/compose/factory.py: a MinMaxScaler whose transform floors the output at feature_range[0] (1e-3) with np.maximum. NaN passes through.
  • get_numerical_transformer_steps uses it for the scale_positive step of "box-cox".

A value below the training minimum now transforms like the minimum. Every other output, including values above the training maximum, is unchanged. MinMaxScaler(clip=True) was not used because it would also clip values above the training maximum, which extrapolate fine today.

Behaviour changes

  • Box-Cox output for any value below the training minimum equals the output for the minimum. This includes values within about 0.1% of the training range below the minimum, which stayed positive before and gave a slightly lower finite output.
  • The scale_positive step of a newly fitted box-cox preprocessor is a _PositiveMinMaxScaler, so its to_spec() state and fingerprint_ differ from 1.0.0. Preprocessors with other numerical methods keep their fingerprint. to_spec()/from_spec() and pickle round trips keep the floor and the fingerprint. A box-cox preprocessor saved by 1.0.0 still holds a plain MinMaxScaler and keeps the old behaviour until it is refitted.

Documentation

  • Preprocessor class docstring (Notes): "box-cox" first rescales each column into [1e-3, 1] with a min-max scaler fitted on the training data, floored at 1e-3, so a value below the training minimum transforms like the minimum and values above the maximum still extrapolate.
  • _PositiveMinMaxScaler docstring.
  • No page under docs/ describes the box-cox internals (they only list the methods), so none needed a change.

Tests

  • test_boxcox_transforms_values_below_the_training_minimum: values 0.5 and 5 below the training minimum transform to finite values equal to the transform of the minimum. Fails on main (ValueError), passes here.
  • test_boxcox_survives_cross_validation: the issue's 5-fold cross_val_score returns finite scores for every fold. Fails on main, passes here.
  • test_boxcox_still_extrapolates_above_the_training_maximum: a value above the training maximum still maps above the maximum's output, so the upper end is not clipped. Guard test: passes on both by design.

Reproduction

Output of the script from #59 on this branch:

[[-2.53352473]]
[[-2.53352473]]
[0.99970011 0.99958985 0.99922595 0.99956836 0.99959617]

The value 0.5 below the training minimum now transforms like the minimum, and all five folds score.

Checks

  • just test: 1441 passed, 59 skipped, 7 xfailed; coverage 93.65% (gate 90%).
  • Minimum dependencies (CI job: Python 3.10, numpy 1.24.4, pandas 2.0.3, scipy 1.10.1, scikit-learn 1.6.0): 1428 passed, 72 skipped, 7 xfailed; scripts/quickstart.py passes.
  • just docs: strict build succeeds.
  • just check: all pre-commit and pre-push hooks pass (ruff format, ruff lint, prettier, file hygiene, pyright with 0 errors).
  • Import direction between subpackages is unchanged.

Merge notes

This change was previously part of #75, which is closed in favour of one pull request per issue.

numerical_method='box-cox' rescales each column into (1e-3, 1) with a
MinMaxScaler fitted on the training data, because Box-Cox requires strictly
positive input. The scaler does not clip, so any value below the training
minimum mapped to <= 0 and PowerTransformer raised at transform, breaking
prediction and plain cross-validation.

The scaler's output is now floored at 1e-3: such values transform like the
training minimum, while every other output, including values above the
training maximum, is unchanged. MinMaxScaler(clip=True) was not used because
it would also clip values above the training maximum, which work today. The
Preprocessor docstring notes the floor.

Fixes #59

@mkumar73 mkumar73 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • Updated main,
  • no issues in CI
  • Changes looks appropriate for merging.

@mkumar73
mkumar73 merged commit d3103a7 into main Oct 18, 2026
22 checks passed
@mkumar73
mkumar73 deleted the fix/59-boxcox-unseen-low-values branch October 18, 2026 10:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Preprocessor(numerical_method="box-cox") raises at transform for values below the training minimum

2 participants