Skip to content

Fix crashes on degenerate ROC curves and invalid input (2.0.5) - #2

Merged
CBravoR merged 8 commits into
mainfrom
fix/robustness-2.0.5
Oct 30, 2026
Merged

CBravoR merged 8 commits into
mainfrom
fix/robustness-2.0.5

Conversation

@CBravoR

@CBravoR CBravoR commented Oct 30, 2026 •

Copy link
Copy Markdown
Member

Summary

This pull request fixes how EMP-PY 2.0.4 fails on degenerate or invalid input, and it prepares version 2.0.5. Results on valid input do not change. It also adds a workflow that runs the tests on pushes and pull requests, and a workflow that publishes to PyPI with Trusted Publishing.

Changes

EMP/metrics.py:

  • Replace scipy.spatial.ConvexHull with a monotone-chain upper hull from (0,0) to (1,1). Qhull raised QhullError when the ROC curve had fewer than three points or all its points on one line, for example with constant scores or empCreditScoring([0.3, 0.3, 0.7, 0.7], [0, 1, 0, 1]). The hull uses the integer counts of false and true positives, so the test for collinear points is exact. empChurn uses the same helper.
  • Raise ValueError instead of calling sys.exit, which ended the Python process.
  • Raise a clear ValueError when the labels contain only one class. Version 2.0.4 failed with ValueError: Points cannot contain NaN.
  • Drop the check that scores lie in [0,1]. EMP depends only on the order of the scores, so logits and scorecard points work. Scores that are NaN, infinite or not numeric raise ValueError.
  • Raise ValueError for input that is not one-dimensional.
  • Suppress the divide-by-zero RuntimeWarning from a horizontal last hull segment. Its lambda (or gamma) is infinite, and the existing < 1 filter drops it.
  • Replace the Python loops in the input checks with numpy operations.
  • Explain in the comments of empCreditScoring how EMPC_fraction is chosen when two cutoffs tie at lambda = 1.

Other files:

  • README.md: the functions return the output and do not print it by default. The README now states the label and score conventions.
  • tests/: 54 pytest tests. tests/brute_force.py computes EMP from its definition on a grid, without the hull.
  • setup.py, README.md, NEWS: version 2.0.5 and a changelog.
  • setup.py: require Python 3.10 or later (was 3.8). Give the license as the SPDX expression GPL-3.0-or-later instead of the deprecated classifier, list pandas once, and drop setup_requires.
  • .github/workflows/tests.yml: runs the tests on each push to main and on each pull request. The matrix is Python 3.10 to 3.15 on Linux, Python 3.15 on Windows and macOS, and numpy 1.x on Python 3.12.
  • .github/workflows/publish.yml: a published GitHub release runs the tests (through tests.yml), builds the package, checks that the tag matches the version, and uploads to PyPI after a reviewer approves the pypi environment.

Results on valid input

The PyPI 2.0.4 wheel and this branch ran the same 6,030 calls: 1,005 data sets with 6 parameter settings. Both used numpy 2.5.4, scipy 1.18.1, scikit-learn 1.9.1 and pandas 3.1.0 on Python 3.14.6.

Count
Calls that 2.0.4 computes 6,018
Of those, equal bit for bit in this branch 6,018
Calls that fail in 2.0.4 (QhullError) and work in this branch 12
Calls that work in 2.0.4 and fail in this branch 0
RuntimeWarning: divide by zero, 2.0.4 / this branch 2,598 / 0

Against brute force on 4,850 calls, EMPC and EMP agree within 2.3e-7, which is the grid error.

Behavior changes

  • Invalid input raises ValueError instead of SystemExit.
  • Scores outside [0,1] are accepted.
  • Input that is not a vector raises ValueError. Version 2.0.4 flattened nested lists of any shape, so [[0.1, 0.2], [0.3, 0.4]] was read as four scores.
  • A single-column DataFrame works. In 2.0.4 it raised KeyError.

Known limitation, not changed

Sometimes two ROC points tie exactly for the maximum profit at lambda = 1. Then the fraction at the p1 point mass is not unique. Version 2.0.4 and this branch both choose by how lambda rounds near 1. In 15 small-sample cases with ROI = 0.5, the fraction differs from brute force by up to 0.06. EMPC is not affected. This pull request keeps the 2.0.4 behavior and documents it in the comments of empCreditScoring. In all 165 test calls with an exact tie, the choice follows the documented rule.

Tests

pip install pytest
python -m pytest

All 54 tests pass on this branch and on the built wheel in a clean environment. On 2.0.4, 19 fail, which are the edge cases and the new behavior. The Tests workflow runs them on this pull request.

Before the first release with the workflow

  1. On PyPI, open the EMP-PY project, then Manage > Publishing. Add a GitHub publisher with owner Banking-Analytics-Lab, repository EMP-Py, workflow publish.yml and environment pypi.
  2. On GitHub, open Settings > Environments and create the environment pypi with at least one required reviewer.
  3. After the merge, create a GitHub release from main with the tag v2.0.5.

Not in this pull request

  • metrics.py still imports pandas but does not use it, so pandas stays a requirement.
  • DESCRIPTION, man/ and inst/CITATION are files from the R package.

- Replace scipy.spatial.ConvexHull with a monotone-chain upper hull from
  (0,0) to (1,1). Qhull raised QhullError when the ROC curve had fewer than
  three points or all points on one line (constant scores, a diagonal ROC
  with ties). The hull uses the integer counts of false and true positives,
  so the collinearity test is exact and the hull vertices are the same as
  before on valid input.
- Raise ValueError instead of calling sys.exit.
- Raise a clear ValueError when the labels contain only one class.
- Drop the check that scores lie in [0,1]. EMP depends only on the order of
  the scores. Raise ValueError for NaN, infinite or non-numeric scores.
- Raise ValueError for inputs that are not one-dimensional.
- Suppress the divide-by-zero RuntimeWarning from a horizontal last hull
  segment. Its lambda (or gamma) is infinite and the next line drops it.
- Replace the Python loops in the input checks with numpy operations.
…fault

The README said that the functions print by default, but print_output defaults to False. Also state the label and score conventions, and that invalid input raises ValueError.
Cover the README and test.py examples, random and credit-like data against a brute-force computation of EMP from its definition, the edge cases that failed in 2.0.4, input types, and warnings.
…lishing

A published GitHub release runs the tests, builds the sdist and wheel, checks that the tag matches the package version, and uploads to PyPI after a reviewer approves the pypi environment. No API token is stored. The one-time setup on PyPI and GitHub is in the comments at the top of the workflow.
…requests

The workflow runs pytest on Python 3.10 to 3.15 on Linux, on Python 3.15 on
Windows and macOS, and with numpy 1.x on Python 3.12. The publish workflow
now calls it instead of running its own test job.
setup.py gives the license as an SPDX expression instead of the deprecated
classifier, lists pandas once, drops setup_requires and calls setup() outside
the with block.
@CBravoR
CBravoR merged commit 2ff92fb into main Oct 30, 2026
9 checks passed
@CBravoR
CBravoR deleted the fix/robustness-2.0.5 branch October 30, 2026 19:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant