Repository navigation
fix(preprocessor): keep box-cox finite below the training minimum - #104
Merged
Merged
Conversation
numerical_method='box-cox' rescales each column into (1e-3, 1) with a MinMaxScaler fitted on the training data, because Box-Cox requires strictly positive input. The scaler does not clip, so any value below the training minimum mapped to <= 0 and PowerTransformer raised at transform, breaking prediction and plain cross-validation. The scaler's output is now floored at 1e-3: such values transform like the training minimum, while every other output, including values above the training maximum, is unchanged. MinMaxScaler(clip=True) was not used because it would also clip values above the training maximum, which work today. The Preprocessor docstring notes the floor. Fixes #59
This was referenced Oct 16, 2026
mkumar73
approved these changes
Oct 18, 2026
mkumar73
left a comment
Collaborator
There was a problem hiding this comment.
- Updated main,
- no issues in CI
- Changes looks appropriate for merging.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #59.
Preprocessor(numerical_method="box-cox")raised aValueErrorattransformfor any value more than about 0.1% of the training range below the training minimum. This broke prediction on new data and plain cross-validation, where three of five folds scorednanin the issue's example.Root cause
Box-Cox needs strictly positive input, so
get_numerical_transformer_stepsputs aMinMaxScaler(feature_range=(1e-3, 1)), fitted on the training data, in front ofPowerTransformer(method="box-cox"). With the defaultclip=False, the scaler maps a value far enough below the training minimum to<= 0, andPowerTransformer.transformrejects it. Positivity was guaranteed only for in-sample values.Fix
_PositiveMinMaxScalerinpretab/compose/factory.py: aMinMaxScalerwhosetransformfloors the output atfeature_range[0](1e-3) withnp.maximum. NaN passes through.get_numerical_transformer_stepsuses it for thescale_positivestep of"box-cox".A value below the training minimum now transforms like the minimum. Every other output, including values above the training maximum, is unchanged.
MinMaxScaler(clip=True)was not used because it would also clip values above the training maximum, which extrapolate fine today.Behaviour changes
scale_positivestep of a newly fitted box-cox preprocessor is a_PositiveMinMaxScaler, so itsto_spec()state andfingerprint_differ from 1.0.0. Preprocessors with other numerical methods keep their fingerprint.to_spec()/from_spec()and pickle round trips keep the floor and the fingerprint. A box-cox preprocessor saved by 1.0.0 still holds a plainMinMaxScalerand keeps the old behaviour until it is refitted.Documentation
Preprocessorclass docstring (Notes):"box-cox"first rescales each column into[1e-3, 1]with a min-max scaler fitted on the training data, floored at1e-3, so a value below the training minimum transforms like the minimum and values above the maximum still extrapolate._PositiveMinMaxScalerdocstring.docs/describes the box-cox internals (they only list the methods), so none needed a change.Tests
test_boxcox_transforms_values_below_the_training_minimum: values 0.5 and 5 below the training minimum transform to finite values equal to the transform of the minimum. Fails onmain(ValueError), passes here.test_boxcox_survives_cross_validation: the issue's 5-foldcross_val_scorereturns finite scores for every fold. Fails onmain, passes here.test_boxcox_still_extrapolates_above_the_training_maximum: a value above the training maximum still maps above the maximum's output, so the upper end is not clipped. Guard test: passes on both by design.Reproduction
Output of the script from #59 on this branch:
The value 0.5 below the training minimum now transforms like the minimum, and all five folds score.
Checks
just test: 1441 passed, 59 skipped, 7 xfailed; coverage 93.65% (gate 90%).scripts/quickstart.pypasses.just docs: strict build succeeds.just check: all pre-commit and pre-push hooks pass (ruff format, ruff lint, prettier, file hygiene, pyright with 0 errors).Merge notes
main.pretab/compose/factory.py. add_missing_indicator / impute_with_indicator feeds the indicator through the scaler and basis; lineage crashes #62 edits thesklearn.imputeimport two lines below the newimport numpy as np. Git merges both cleanly with this branch.tests/integration/test_preprocessor.py, so expect an append/append conflict there: keep both blocks.This change was previously part of #75, which is closed in favour of one pull request per issue.