Repository navigation
Ingester: do not count ruler staleness markers as discarded tenant samples - #7887
Merged
danielblando merged 1 commit intoOct 21, 2026
Conversation
…mples Signed-off-by: Daniel Blando <ddeluigg@amazon.com>
danielblando
force-pushed
the
ruler-stale-marker-discarded-samples
branch
from
October 20, 2026 18:33
1ebb721 to
48ab570
Compare
yeya24
approved these changes
Oct 20, 2026
SungJin1212
approved these changes
Oct 20, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does:
Stops counting staleness markers written by the Ruler as discarded samples for the tenant.
The failure case
When a recording rule is moved from one rule group to another within the same namespace, the two groups race over the rule's output series:
R. Every seriesRproduced is queued into group A'sstaleSerieslist.Rand immediately starts writing those same series.cleanupStaleSeriesappends a staleness NaN at the current timestamp for series that group B keeps advancing. The append is rejected —out-of-order,too-oldorduplicate-timestampdepending on timing.cleanupStaleSeriesclearsstaleSeriesonly whenCommit()succeeds, so the whole set is re-attempted on every evaluation until the ruler restarts.While that lasts, every rejected marker increments
cortex_discarded_samples_total{reason="sample-out-of-order"}(orsample-too-old/new-value-for-timestamp) for the tenant that owns the rule group. A staleness marker is generated by the Ruler, not sent by the tenant, so the metric reports data loss that did not happen: nothing the tenant sent was dropped.Prometheus already treats exactly these three errors as expected on this path, and deliberately does not account for them:
Cortex was still counting them against the tenant. This PR makes the accounting agree with that classification.
No new metric is added. The rejected append is still counted per tenant by
cortex_ingester_tsdb_out_of_order_samples_total{type,user}, a pass-through of the TSDB's own counter that is unaffected by this PR, so the failure case remains trackable.Which issue(s) this PR fixes:
No existing issue — the behaviour is described above.
Checklist
CHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX]docs/configuration/v1-guarantees.mdupdated if this PR introduces experimental flags — n/a, no new flags.