Skip to content

Ingester: do not count ruler staleness markers as discarded tenant samples - #7887

Merged
danielblando merged 1 commit into
cortexproject:masterfrom
danielblando:ruler-stale-marker-discarded-samples
Oct 21, 2026
Merged

danielblando merged 1 commit into
cortexproject:masterfrom
danielblando:ruler-stale-marker-discarded-samples

Conversation

@danielblando

Copy link
Copy Markdown
Contributor

What this PR does:

Stops counting staleness markers written by the Ruler as discarded samples for the tenant.

The failure case

When a recording rule is moved from one rule group to another within the same namespace, the two groups race over the rule's output series:

  1. Group A reloads and no longer contains rule R. Every series R produced is queued into group A's staleSeries list.
  2. Group B reloads with R and immediately starts writing those same series.
  3. On every subsequent evaluation, group A's cleanupStaleSeries appends a staleness NaN at the current timestamp for series that group B keeps advancing. The append is rejected — out-of-order, too-old or duplicate-timestamp depending on timing.
  4. cleanupStaleSeries clears staleSeries only when Commit() succeeds, so the whole set is re-attempted on every evaluation until the ruler restarts.

While that lasts, every rejected marker increments cortex_discarded_samples_total{reason="sample-out-of-order"} (or sample-too-old / new-value-for-timestamp) for the tenant that owns the rule group. A staleness marker is generated by the Ruler, not sent by the tenant, so the metric reports data loss that did not happen: nothing the tenant sent was dropped.

Prometheus already treats exactly these three errors as expected on this path, and deliberately does not account for them:

case errors.Is(unwrappedErr, storage.ErrOutOfOrderSample),
	errors.Is(unwrappedErr, storage.ErrTooOldSample),
	errors.Is(unwrappedErr, storage.ErrDuplicateSampleForTimestamp):
	// Do not count these in logging, as this is expected if series
	// is exposed from a different rule.

Cortex was still counting them against the tenant. This PR makes the accounting agree with that classification.

No new metric is added. The rejected append is still counted per tenant by cortex_ingester_tsdb_out_of_order_samples_total{type,user}, a pass-through of the TSDB's own counter that is unaffected by this PR, so the failure case remains trackable.

Which issue(s) this PR fixes:

No existing issue — the behaviour is described above.

Checklist

  • Tests updated
  • Documentation added — n/a. No user-facing config, API or metric change; the PR only stops one internal write from being counted in an existing metric.
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]
  • docs/configuration/v1-guarantees.md updated if this PR introduces experimental flags — n/a, no new flags.

@danielblando
danielblando requested a review from a team as a code owner October 20, 2026 18:32
@danielblando
danielblando requested a review from yeya24 October 20, 2026 18:32
…mples

Signed-off-by: Daniel Blando <ddeluigg@amazon.com>
@danielblando
danielblando force-pushed the ruler-stale-marker-discarded-samples branch from 1ebb721 to 48ab570 Compare October 20, 2026 18:33
@danielblando
danielblando merged commit dd856d6 into cortexproject:master Oct 21, 2026
74 of 75 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants