[P2] ListObjects can omit readable keys during rolling restarts with concurrent overwrites

[Vonng]

Summary

A successful ListObjects (HTTP 200) can omit keys that exist and are readable while nodes are restarted one at a time under concurrent overwrites. The resolver repair on main (8d06424b1, after the published Server 20260903) removes one confirmed order-sensitive cause; the rolling-restart omission reproduced on a build containing that repair and its internal cause is not yet bound. This is inherited behavior; the 2026-09-16 release decision does not depend on it.

What is fixed on main

mergeXLV2Versions inherited from upstream PR #14125 (2022-01-19) a healing-tolerant selection that is sensitive to input order: a newer minority at the end of the drive order resets the count and an older null version that had quorum is dropped. 8d06424b1 recounts per header only when the original selection lacks quorum, every non-empty input stream has exactly one ordinary null version, none is a free version and all share the same erasure parameters. Only a group reaching the original quorum is returned; mixed histories keep their previous behavior. Nine old-order and seven new-order counterexamples now pass, and 5,620 differential inputs keep the contracted result.

What remains open

On a 4-node x 4-drive cluster with 16 fixed keys under concurrent overwrites:

Load	HTTP 200 LIST	Omitting keys
Steady state, 84 s	20,000	0
Rolling restart, 121 s	27,966	2,624 (9.4%)

Each omission dropped 1 to 4 keys; 8,186 of 8,192 same-endpoint GET/HEAD checks of the missing keys returned 200. The strongest sample started 17.6 s after the last node's health check returned and omitted a key whose last successful PUT was confirmed 11.5 s earlier, with no later write to that key; the same endpoint then served that exact generation. Raw XML, HTTP correlation and body hashes were independently re-verified, so pagination or parsing errors do not explain it.

Code paths that still turn "cannot decide" into "absent" without a request-level error: mergeXLV2Versions when no generation reaches quorum, metaCacheEntries.resolve (fewer valid entries than quorum, no cached metadata, empty merge), the partial callback in listPathRaw/listPath which drops an unresolved entry and only fails the request when more than len(disks)-minDisks readers have failed, and the walker, which skips an entry whose xl.meta read fails with an unexpected error. The listing quorum is computed from the number of drives asked and is not adjusted for readers that fail mid-listing. None of these can be bound to the sample without the per-drive streams, effective quorum, recount eligibility and cache state of the failing request; the post-run xl.meta files are recovered state, not the failing input.

Next steps

1. Instrument one failing LIST in an isolated diagnostic build: drive selection and reader order, the complete per-drive stream and errors, effective quorum/strict, recount eligibility and group counts, merge output, partial and cache publication. Bounded at one to two days.
2. Classify which exit is hit (quorum reached but not emitted, genuinely undecidable treated as absence, or cache reuse of an incomplete result) and build a deterministic regression from the captured input.
3. Only then decide whether the contract changes (unresolved entries fail the request, or a three-state result); that is a design decision, not part of the current repair.

Until then, operators should not run sync tools that delete destination objects missing from a source listing during rolling restarts, and should re-list once the cluster is stable.

Evidence

The harness, raw request logs and the extracted counterexamples are kept outside the repository by the maintainer (2026-09-16 rolling-restart run against a build containing 8d06424b1).

[Vonng]

The 20260916 release includes the narrow null-version quorum repair (8d06424b1), but the rolling-restart omission remains a known limitation. The issue's reproduced LIST responses can still omit readable keys even though same-endpoint GET/HEAD succeeds.

Keeping this open. Until the failing request is instrumented per drive (reader order/streams, effective quorum, cache state and partial handling) and reduced to a deterministic regression, do not run synchronization tools that delete destination objects from a LIST result during rolling restarts; re-list after the cluster is stable.