Skip to content

ConfusionMatrixMetric: ignore_index excludes a channel, not the ignored voxels #9206

Open

Description

@JoaoMarcos44

Describe the bug

get_confusion_matrix() (the backbone of ConfusionMatrixMetric) handles a valid ignore_index by zeroing the ignored channel in y_pred and y. Zeroing a channel does not remove those voxels from the other channels' counts, so a voxel of the ignored class that the model assigned to another class still counts as a false positive for that class.

DiceMetric and MeanIoU instead build a spatial mask with create_ignore_mask() (#9134 fixed compute_iou() for exactly this inconsistency), so ConfusionMatrixMetric disagrees with them for the same input and the same setting.

Two related facets of the same root cause:

  • With a single-channel binary target and ignore_index=0, include_background=True additionally zeroes the only channel, hiding a valid false positive at a non-ignored location (specificity 1.0 instead of 0.0).
  • Boolean one-hot targets crash in create_ignore_mask() (1.0 - bool_tensor raises RuntimeError).

To Reproduce

import torch
from monai.metrics import get_confusion_matrix

# 4 voxels, 3 one-hot classes; voxel 1 belongs to the ignored class 1
y      = torch.tensor([[[1., 0, 0, 1], [0, 1, 0, 0], [0, 0, 1, 0]]])
# a perfect prediction except the ignored voxel is called class 0
y_pred = torch.tensor([[[1., 1, 0, 1], [0, 0, 0, 0], [0, 0, 1, 0]]])

counts = get_confusion_matrix(y_pred, y, include_background=True, ignore_index=1)
counts[0, 0].tolist()  # [2.0, 1.0, 1.0, 0.0] — TP, FP, TN, FN for class 0

Single-channel case:

import torch
from monai.metrics import ConfusionMatrixMetric

y = torch.tensor([[[1., 1., 0.]]])  # single-channel binary target
metric = ConfusionMatrixMetric(metric_name="specificity", reduction="none", ignore_index=0)
metric(y_pred=torch.ones_like(y), y=y)
metric.aggregate()  # specificity 1.0 — the false positive at the only valid voxel is hidden

Expected behavior

counts[0, 0].tolist()  # [2.0, 0.0, 1.0, 0.0] — the ignored voxel is excluded from every class's counts

Class-0 precision on the valid voxels is 1.0, matching DiceMetric/MeanIoU for the same input. In the single-channel case, specificity is 0.0: the false positive at the only valid voxel is retained regardless of include_background.

Environment

Validated on dev @ bbebca6f17cabf98ea5fb8ec5776eeacd10aa60d, Windows, Python 3.14.1, PyTorch 2.12.1+cpu, NumPy 2.3.5.

Additional context

PR #9195 fixes this by excluding the ignored voxels spatially (reusing create_ignore_mask() semantics), while preserving the existing channel exclusion for multi-channel targets, retaining valid single-channel errors regardless of include_background, and supporting boolean targets. Related to #9134, which fixed the same inconsistency for MeanIoU.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions