[filimonov]

Renaming a committed part on a content-addressed disk is one ref-log owner Move (plus SetPublishedAt in the same append) instead of a manifest republish; MergeTree writers return to the upstream text (storage transaction committed right after finalize, renameParts only renames), and the fork's five writer guards go. Logs carrying a Move are stamped format generation 2; the pool records min_reader_generation, which a 26.8 writable mount raises once (26.6.x then refuses to mount). The branch starts with reverts of fork-only changes to upstream files that 26.8 made redundant, then allows EXPORT PART/EXPORT PARTITION on CAS disks (partition_exports is a reserved table-level directory, so the descriptor is not parsed as a part). Every CAS_OWNERSHIP_CONFLICT (1039) is counted in CASOwnershipConflict; the operator section is in docs/en/antalya/cas/operations/debugging.md.

The branch is built on the head of #2579 (packed parts), so this PR carries those commits until 2579 merges; the series itself is the 13 commits starting at "Replace unwrapTableProxy with castStorage in SYSTEM commands".

Measured on local object storage: INSERT/merge/mutation/LWU/ATTACH FROM/fetch manifest_put 2→1, ref_drop 1→0, ref_move 1; INSERT PartsLockHoldMicroseconds 2557→317. A replicated merge renames inside the open storage transaction and commits once under the final name (manifest_put 1, ref_move 0, oracle in 05387).

Related: #2579 (base: packed parts)
Related: #2159 (the CAS port the reverts refer to)
Related: #2540

Follow-up: gate fetch-by-relink on a capability cookie instead of replication protocol version 11.

Changelog category (leave one):

• New Feature

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Content-addressed disks rename a committed part with a single ref-log record instead of republishing its manifest. A pool mounted writable by 26.8 can no longer be mounted by 26.6.x: upgrade every server before the first writable mount. EXPORT PART and EXPORT PARTITION work on CAS disks.

Documentation entry for user-facing changes

• Documentation is written: docs/en/antalya/cas/configuration.md (upgrading a pool from 26.6.x), docs/en/antalya/metrics_and_events.md (CASRefMove, CASOwnershipConflict), docs/en/antalya/cas/operations/debugging.md (CAS_OWNERSHIP_CONFLICT).

CI/CD Options

Exclude tests:

• Fast test
• Integration Tests
• Stateless tests
• Stateful tests
• Unit tests
• Performance tests
• Aarch64 tests
• All with ASAN
• All with TSAN
• All with MSAN
• All with UBSAN
• All with Coverage
• All Regression
• Disable CI Cache

Regression jobs to run:

• Fast suites (mostly <1h)
• Aggregate Functions (2h)
• Alter (1.5h)
• Benchmark (30m)
• CAS (content-addressed storage; Antalya only)
• ClickHouse Keeper (1h)
• Iceberg (2h)
• LDAP (1h)
• OAuth (5m)
• Parquet (1.5h)
• RBAC (1.5h)
• SSL Server (1h)
• S3 (2h)
• S3 Export (2h)
• Swarms (30m)
• Tiered Storage (2h)

🤖 Generated with Claude Code

[github-actions]

Workflow [PR], commit [75a618c]

[alsugiliazova]

PR #2647 Distributed-Systems Audit

Verdict

No High defect. Three findings: two Medium in version skew, one Low in timing.

The central mechanism is sound. A committed rename inside one namespace becomes a single OwnerTransition record whose old and new bindings are both Committed, same manifest, different name. The classification is written once (classifyOwnerTransitionShape, now in Formats/CasRefLogFormat.h) and consumed by both the replay state machine and GC's edge extractor, so the two readers of the format cannot drift. Every rename is conditional on the manifest the caller resolved, and the conditional check runs inside the appendRefOps builder on the flush leader against batch-validated state — not against a snapshot read earlier.

What is not fully closed is the mixed-version story. The pool's min_reader_generation is checked only when a server mounts, so it keeps a 26.6.x server out of a pool it restarts into, but it does nothing about a 26.6.x server that is already mounted. That server's GC then meets a generation-2 log, fails closed, and holds the namespace forever — reclaiming nothing, with a log line as the only signal. This is not theoretical: it happened in this PR's own CI run and is visible in the artifacts (finding 1).

Separately, reserving partition_exports/ as a table-level directory changes where an already-written EXPORT PARTITION descriptor lives, with no migration and outside the generation gate (finding 2).

What the change does

Before	After
Renaming a committed part republished its manifest under the new name and dropped the old ref: 2 manifest_put, 1 ref_drop	One ref-log Move (plus SetPublishedAt in the same append): 1 manifest_put, 0 ref_drop, 1 ref_move
Four legal OwnerTransition shapes	Five — Move added, stamped format generation 2
Every object stamped G_BUILD as its compatibility floor	Objects stamped G_WRITE_FLOOR (1); only a Move-bearing ref log is stamped G_REF_MOVE (2)
A CAS part published once, under its final name, inside Transaction::commit	Published right after finalize under its temporary name; the rename that follows is a Move. The fork's five writer guards go, and MergeTree returns to the upstream text
checkAlterPartitionIsPossible kept a CAS whitelist of partition commands	No whitelist; EXPORT PART and EXPORT PARTITION work, and partition_exports/ is reserved next to deduplication_logs/
transactionIsStagingOverlay	Folded into appliesOperationsEagerly
A rename conflict threw ABORTED	CAS_OWNERSHIP_CONFLICT (1039), counted in CASOwnershipConflict

Model

Actors	The writing server (mounted writable under its own server_root_id), peer servers mounting the same pool — possibly on 26.6.x — the pool-wide GC leader, Keeper on the replicated path, and the object store
Channels	Conditional PUT of ref-log objects on write-once keys; readModifyWrite of _pool_meta and each namespace's _ckpt; the replication protocol's fetch-by-relink offer and its confirm round trip; the Keeper multi
Authoritative state	The ref log per (namespace, writer_epoch); _pool_meta.min_reader_generation for who may mount; Keeper for part and block znodes. RefTableState is a cache rebuilt by replay
Failure model assumed by the code	Crash-recover, not crash-stop. The store may acknowledge a put and lose the response. Conditional create is linearizable. A writable mount is leased per server root, so several servers write one pool concurrently in different namespaces. Every peer is upgraded before the first 26.8 writable mount — assumed, not enforced
Safe failure criterion	No part lost or published twice; no committed ref whose manifest is unreachable; no live blob reclaimed; a rename either completes or is cleanly refused with the source intact

The central argument, and why it holds

The Move is conditional on what the caller read, at the moment it is written. renameCommittedRef resolves the source with Freshness::ForceFresh and hands that manifest to moveRefExact as expected. moveRefExact does not trust it: its appendRefOps builder re-reads state.getCommitted() on the flush leader and refuses unless the source still names expected, the destination is absent, and no live precommit holds the destination name. A writer that lost the race gets CAS_OWNERSHIP_CONFLICT and must re-resolve. The one shape that is not a conflict — source absent, destination already naming expected — returns an empty op list, so a redrive of the same rename is a no-op rather than a second record.

The Move is net-zero for GC. manifestEdgesOfTxn emits no edge for it, which is correct because the manifest keeps exactly one committed owner across the rename: the arm in applyOwnerTransition leaves owned_manifests untouched. The byte accounting is handled carefully — row is copied out of the map before emplace inserts the destination, the two size counters are adjusted for the old name and then for the new one, and the source is erased by key rather than by a possibly-invalidated iterator.

Durability still precedes the Keeper decision. The deleted comment block in renameParts protected an ordering invariant: a part must be durable before its block_id or part znode is registered, otherwise a fault between the two leaves a phantom part whose surviving dedup token silently swallows a byte-identical client retry. Every writer now commits earlier than it used to — right after finalize, before renameTempPartAndReplace — so the invariant is strengthened, not weakened. MergeFromLogEntryTask is the one that commits after the rename, and it still commits before checkPartChecksumsAndCommit.

The two-name scope was widened everywhere it is read. MutationScope gains second_ref_name, and all four consumers moved to names()/forEachName(): confirmExactRef's covers, refNamedOutsideScope, flushRefBatch's seen_refs dedup, and isRefDone. The whole-shard branch is taken before seen_refs is touched, so dropping the old insert("") costs nothing. ref_name_slots is keyed by namespace string, not by ref name, so a two-name scope needs no change there.

A Move is safer for fetch-by-relink than the republish it replaces. A relink receiver adopts the sender's blobs by evidence of the sender's live edge. A republishRef creates a new manifest and retires the old one, moving that edge; a Move keeps the manifest and its in-degree, so a concurrent sender-side rename cannot invalidate the receiver's evidence.

Findings

Configuration and version skew

Medium: the reader-generation floor gates new mounts, not peers that are already mounted — and a 26.6.x peer's GC then stalls permanently with only a log line.

• Impact. On a pool shared by a 26.6.x and a 26.8 server, the 26.8 server's first writable mount raises min_reader_generation to 2 and begins emitting Move-bearing logs. The floor is read only in decodePoolMeta, which runs at mount. A 26.6.x server already serving keeps serving, and when it is the GC leader it cannot fold the 26.8 namespace: decodeRefLogTxn throws UNKNOWN_FORMAT_VERSION, the intake's catch calls hold(log_id, HoldReason::BodyUndecodable) and breaks. Nothing is ever reclaimed from that point. Blobs accumulate on shared object storage indefinitely, and the only signal is one LOG_WARNING per round.
• Anchor. The raise in Pool::open (CasPool.cpp, the !read_only && meta.min_reader_generation < G_REF_MOVE block). The gate in decodePoolMeta (CasPoolMetaFormat.cpp 155). The fail-closed intake in Gc::fold (CasGc.cpp 2945–2951).
• Trigger. Start a 26.6.x and a 26.8 server against one pool, then INSERT on the 26.8 server. No restart, no partition, no timing window needed.
• Why defect. It fails closed, so nothing is deleted or corrupted — but configuration.md#upgrading-from-26-6 presents the floor as the mechanism that keeps 26.6.x out of a generation-2 pool, and for a live peer it is not a mechanism at all. The doc admits this in one clause ("its GC round holds on the undecodable log and performs no deletions") while the product offers no check, no metric, and nothing an operator would notice before storage grows.
• It is not theoretical. The PR's own CI run reproduces it. In test_cas_ref_move_compat, the old node's server log repeats, once per GC round: CAS GC ref intake: log shared_pool/cas/ns/stream/b4f84942.../_log/0000000000000002-0000000000000003.zst invalid: CAS cas_ref_log: object requires reader generation 2 but this build supports at most 1.
• Fix direction. Record each mount's reader ceiling in its mount object, and have the raise refuse (or at minimum warn with a metric) while another server root holds a live lease below G_REF_MOVE. Independently, surface a held namespace in system.cas_gc_log or a CurrentMetrics gauge so a stalled GC is visible without grepping logs.
• Regression test direction. Extend test_cas_ref_move_compat to assert the old node's GC holds, and that the hold is visible in an operator-facing table rather than only in the log.

Medium: partition_exports/ becomes a reserved table-level directory, with no migration for a pool that already ran EXPORT PARTITION.

• Impact. EXPORT_PARTITION was already in the CAS whitelist this PR deletes, so an older build accepted the command and stored <table>/partition_exports/<key>.json through the part path — findPartDirComponent returned a part dir literally named partition_exports. After the change, isReservedTableSubdir routes the same path to parseTableFilePath, which makes it a table-level verbatim file in a different place. Descriptors written before the upgrade become invisible, and the export scheduler starts from no state.
• Anchor. findPartDirComponent (PartPathParser.cpp 190) and parseTableFilePath (PartPathParser.cpp 354); the deleted case MetadataStorageType::CAS whitelist in MergeTreeData::checkAlterPartitionIsPossible, which listed PartitionCommand::EXPORT_PARTITION.
• Trigger. Run ALTER TABLE … EXPORT PARTITION on a CAS disk with 26.6.x, upgrade, read the descriptor.
• Why defect. This is an on-disk layout change, but min_reader_generation covers only the ref log, so the generation gate does not catch it. There is no migration and no detection — the old descriptor is not reported missing, it is simply not found.
• Fix direction. At mount, detect a committed ref whose name is exactly partition_exports and either migrate its entries into namespace files or fail closed with a message naming the command that produced it.
• Regression test direction. Write a descriptor in the old path shape, restart on the new build, and assert it is still listed.
• Note on severity. Medium rather than High only because CAS is pre-release and the upgrade already requires a full stop of every 26.6.x server for an unrelated reason.

Timing and ordering

Low: the rename restamps published_at_ms from the wall clock.

• Impact. published_at_ms is a part's modification time on a CAS disk, and it drives detached/moving aging and clearOldTemporaryDirectories. A backwards NTP step larger than temporary_directories_lifetime makes a just-renamed temporary ref immediately eligible for cleanup; a forwards step delays reclamation by the same amount.
• Anchor. moveRefExact emits makeSetPublishedAtOp(dst.ref, expected, Cas::nowMs()) (PartFolderAccess.cpp 639); Cas::nowMs reads std::chrono::system_clock (Primitives/CasClock.h).
• Trigger. A clock step on the writing server between the rename and the next cleanup round.
• Why defect. Everything else in the ref protocol orders by writer_epoch and ref_sequence; this one field is wall-clock. The exposure equals a local disk's mtime, which is deliberate and stated in the comment, so this is a documented parity choice rather than a new hazard — hence Low.
• Fix direction. None needed if parity with local disk is the intent; say so where published_at_ms is defined, so a future reader does not take it for a logical timestamp.

What was checked and found clean

Worth recording, because several of these are the places a change of this shape usually breaks.

• Idempotency. moveRefExact's already-moved shape returns an empty op list rather than a second record. republishRef's NETWORK_ERROR arm re-reads the destination and completes only on entry equality, rethrowing when the destination is absent and raising a conflict when its content differs. dropRefExact reports Removed/Gone/Mismatch instead of folding every outcome into a bool, which is what lets the rename tell a completed re-drive from a concurrent repoint.
• Precommit probe. hasLivePrecommitFor uses precommits.lower_bound({ref_name, ManifestRef{}}). ManifestRef{} is {0,0,0} and writer_epoch is never 0 for a real manifest, so the probe reliably lands on the first entry for that name.
• Wire format. writeHeaderLine writes exactly one version field, the compatibility version. currentCompatibilityVersion() changed from G_BUILD to G_WRITE_FLOOR but both evaluate to 1, so every non-Move object a 26.8 server writes is byte-compatible with a 26.6 reader. That is what makes the documented "keeps serving namespaces that have no ref_move record" claim true.
• Eager-transaction flag. Replacing transactionIsStagingOverlay with appliesOperationsEagerly is a pure rename: addOperation and the two commit guards are the only consumers, and CAS already answered true to the old flag.
• Thread group. Dropping ThreadGroup::parent_thread_group is safe — ThreadGroup::parent is itself a ThreadGroupPtr, declared ahead of the trackers, so the parent was always kept alive by it and the removed member was a duplicate.
• Ordinary S3 paths. The diffs in IO/S3/Client.cpp, IO/ReadBufferFromS3.cpp, IO/S3/copyS3File.cpp and S3ObjectStorage.cpp are comment and whitespace only. No request behaviour changes.
• New LOGICAL_ERROR in Transaction::commit. Every producer that reaches Transaction::commit with precommitted parts was traced — the sinks, both merge paths, both mutation paths, createEmptyParts, dropPartsImpl, cloneAndLoadDataPart, the fetch path in DataPartsExchange, and the patch-part sinks — and each commits the storage transaction beforehand. The throw converts a future miss from a late commit under DataPartsLock into a query failure, which is the intended trade.
• Format history. REF_LOG_HISTORY and currentWriterVersion() have no runtime consumer; the history table is enforced only by static_assert. Harmless today, but the change-point array is documentation rather than a check.

Coverage

Class	Result
Network / transport	Reviewed. The Move rides the existing append path; attempt bounds, backoff and the Refused/ambiguous split are unchanged. The one new round trip is the floor raise at mount, on the GC request plane.
Partition / availability	Reviewed. Mount leases and fencing are untouched. A conflict is a refusal, never a second owner: applyOwnerTransition's Move arm rejects a destination that is already committed or precommitted.
Timing / ordering	Reviewed. One finding (the wall-clock published_at_ms). Log ordering stays on writer_epoch/ref_sequence; the Move and its SetPublishedAt are in one record and replay in order.
Partial / gray failure	Reviewed. republishRef's ambiguous-publish arm re-reads the destination before deciding; moveRefExact deliberately does not, because the ledger's own resolve-before-reissue read already settles a landed write and an unreadable one wedges the lane.
Idempotency and duplicates	Reviewed, clean. See above.
Backpressure and resource limits	N/A. No queue, buffer or deadline changed. The two-name scope can cut a batch one item earlier, which bounds work rather than growing it.
Recovery and rollback	Reviewed. The early commit opens a window where a crash leaves a committed tmp_ ref; test_cas_early_commit_restart shows clearOldTemporaryDirectories reclaims it and FSCK is clean. Default temporary_directories_lifetime is 86400 s, so on shared storage those blobs persist for a day — worth knowing, not a defect.
Configuration and version skew	Reviewed. Both Medium findings are here.
Security / trust across nodes	N/A. No authentication or authorization change.
Observability and diagnosability	Reviewed. CASRefMove and CASOwnershipConflict are added and documented; ref_move carries the destination in detail['dst_ref'], and debugging.md tells the operator to match on it. The gap is the GC hold in finding 1, which has no counter.

Assumptions and limits. Static reading of 09d0772a, series 2856f19cf43..7d992d82434 only. The packed-parts content inherited from #2579 was not audited. Nothing was built or run; the two CI corroborations come from artifacts of run 39902779170.

Operational readiness

Scope. Upgrading and running a CAS pool at head 09d0772a. Persona: operator / DBA.

1. A 26.6.x server left running during the upgrade silently stops reclaiming storage

• Symptom. Pool size grows without bound. SYSTEM CAS GC RUN reports nothing deleted. The only evidence is CAS GC ref intake: … requires reader generation 2 but this build supports at most 1, once per round, in the old server's log.
• Trigger. Any 26.8 writable mount of a pool a 26.6.x server is still mounted on. See finding 1.
• Signal quality. Poor. No metric, no system-table row, no query error.
• Severity. Medium.
• Recovery. Stop the 26.6.x server and upgrade it. GC resumes from the held cursor on the next round; nothing needs repair.

2. The point of no return is a side effect of starting a server

• Symptom. The first writable 26.8 mount raises the floor one way. From then on no 26.6.x server can mount the pool, and there is no downgrade.
• Trigger. Starting a 26.8 server with the CAS disk configured. There is no per-server switch and no dry-run.
• Signal quality. Adequate: one WARNING, min_reader_generation raised 1 -> 2. But it is emitted after the raise has already committed.
• Severity. Medium — by design, but irreversible and easy to trigger by accident.
• Recovery. None. Restore the pool from a backup taken before the raise, or finish the upgrade.

3. CAS_OWNERSHIP_CONFLICT is new vocabulary on a path that used to say ABORTED

• Symptom. Error 1039 on ATTACH PART, MOVE PARTITION TO TABLE, ALTER … MATERIALIZE, or recorded in system.replication_queue.last_exception.
• Trigger. Two writers racing on one part name.
• Signal quality. Good. debugging.md#ownership-conflict gives the system.cas_log query, and ProfileEvents.CASOwnershipConflict counts refusals. Note the deliberate change: ABORTED was logged at INFO as a cancelled task and not recorded on the queue entry, so these conflicts were previously invisible. Expect them to start appearing.
• Severity. Low.
• Recovery. Re-run the command. A conflict that repeats every retry means two parts genuinely collide under one name; the documented query shows both sides.

Readiness verdict

Usable, with one operator rule that the product does not enforce: stop every 26.6.x server that mounts the pool before starting any 26.8 server against it. Leaving one up does not corrupt anything, but it stops garbage collection for the whole pool and says so only in a log line. If the pool ever served EXPORT PARTITION under 26.6.x, check those descriptors after the upgrade (finding 2).

Recommended fixes

1. Make the floor raise aware of live mounts below G_REF_MOVE, and expose a held GC namespace as a metric or system-table row rather than a log line (finding 1).
2. Detect or migrate pre-existing partition_exports part refs at mount (finding 2).
3. State in the format docs that published_at_ms is wall-clock by design, matching local-disk mtime (finding 3).

[alsugiliazova]

PR #2647 CI Verification

Verdict

Does not pass CI verification. Two failures must be fixed before approval, and two more are unresolved.

Nothing here points at a defect in the server. The two blocking failures are both self-inflicted by this PR's own test material, and both have a known fix:

1. 05014_insert_dedup_disk_commit_failpoint — a new test added by this PR — fails deterministically whenever the harness randomizes min_bytes_for_full_part_storage high enough to select packed part storage, because the failpoint it relies on exists only in DataPartStorageOnDiskFull. The commit that adds it to packed storage is the current tip of base PR #2579, which this branch does not yet carry.
2. test_cas_ref_move_compat::test_old_pool_floor_and_old_reader — also new in this PR — asserts on a log line that the server did emit, but that had already been rotated out of clickhouse-server.log by the time the assertion ran.

Category	Count	Jobs / tests
regression	2	05014_insert_dedup_disk_commit_failpoint, test_cas_ref_move_compat::test_old_pool_floor_and_old_reader
pre-existing-flaky	0	—
infrastructure	2	test_storage_delta (whole module), test_storage_s3 (whole module)
unknown	3	04812_clickhouse_disks_move_directory, test_cas_gcs::test_ordinary_goog4_traffic_keeps_upstream_semantics, test_storage_gcp_auth::test_gcp_auth_ordinary_contract
Policy	1	DCO

Two jobs were still running at the time of this review: Unit tests (asan_ubsan) and RegressionTestsRelease / CASS3CacheLightweightDelete (3). The CAS regression lane matters most for this PR and is not yet reportable.

How the list was reduced

Six checks are red. No cascade failures: no job shows a server death or a killed process group, and each red job names its own failing tests.

1. Two whole integration modules error out in cluster setup. Both report Command [docker compose … pull] return non-zero code 18. That is image-pull failure, not test failure — the ~100 ERROR rows under test_storage_delta and the ~20 under test_storage_s3 are one setup failure each, counted per test.
2. Two failures are in tests this PR adds. Both were traced to root cause in the job logs and artifacts; neither needs a baseline.
3. Three remaining failures touch code the 13-commit series does not modify. They are left unknown rather than rounded up to pre-existing-flaky, because no branch baseline was gathered.

Findings

05014_insert_dedup_disk_commit_failpoint — regression

Job: Stateless tests (amd_asan_ubsan, db disk, distributed plan, sequential, selected tests), 120268327932.

The test is new in this PR, added by d233c70cb5d ("CAS: rename a committed part with one ref-log Move"). It is the regression test for the acked-then-lost data loss the commit-ordering change fixes: enable part_storage_fail_commit_transaction, insert, and require FAULT_INJECTED.

The query succeeded but the server error '710' was expected
INSERT INTO t_dedup_disk_commit SETTINGS insert_deduplicate = 1, ... -- { serverError FAULT_INJECTED }

The in-job minimizer is unambiguous:

Step	Result
Re-run with the same randomized settings	25 runs, 25 failed
Re-run without randomized settings	39 runs, 0 failed
Minimized culprit	--min_bytes_for_full_part_storage 291743465

That setting pushes the part onto packed storage. part_storage_fail_commit_transaction is injected in DataPartStorageOnDiskFull::commitTransaction only; DataPartStorageOnDiskPacked::commitTransaction has no fiu_do_on, so the failpoint never fires, the insert succeeds, and the expectation fails.

The fix already exists on the base PR. Commit 267672c528e, "Apply part commit failpoint to packed storage", is the current head of #2579 and adds exactly that fiu_do_on to DataPartStorageOnDiskPacked.cpp. This branch is built on an older #2579 head and does not contain it.

Action: rebase onto the current #2579 head, or tag the test with a fixed min_bytes_for_full_part_storage so the randomizer cannot select packed storage. The server behaviour under test is correct either way — nothing here suggests the commit-ordering change is wrong.

test_cas_ref_move_compat::test_old_pool_floor_and_old_reader — regression

Job: Integration tests (amd_asan_ubsan, db disk, old analyzer, 1/8), 120268328007.

Also new in this PR. It fails at line 55:

assert new.contains_in_log("min_reader_generation raised 1 -> 2")
E   AssertionError: assert False

The server did the right thing; the assertion looks in the wrong file. The preceding assertion, _count(new, "event_type = 'ref_move'") == 1, passed — which already proves the floor was at 2, because moveRefExact throws LOGICAL_ERROR when refMoveAllowed() is false. The artifacts confirm it directly. In new/logs/clickhouse-server.log.0.gz:

2026.10.23 07:37:41.096944 [ 202 ] <Warning> CasPool: Content-addressed pool '4030f6cad8fc23f14a1eaa0bd307d353':
min_reader_generation raised 1 -> 2; generation-1 readers can no longer mount this pool

Two things compound. The new instance already has storage_conf.xml at cluster start — contrary to the test's own comment, "No CAS disk at start: the pool must be created by the old build first" — so it mounts the pool and raises the floor at 07:37:41, during its first boot. Then restart_clickhouse() rotates the log, the raise line moves to clickhouse-server.log.0.gz, and contains_in_log reads only clickhouse-server.log.

Action: either keep the CAS disk genuinely off the new instance until after the old build has created the pool, or make the assertion read rotated logs. No product change needed.

This test is worth getting green rather than deleting: its old node's log also contains the permanently-held GC rounds described in finding 1 of the audit, which is the real version-skew gap.

test_storage_delta and test_storage_s3 — infrastructure

Jobs: Integration tests (amd_asan_ubsan, db disk, old analyzer, 6/8) 120268328035 and 1/8 120268328007.

Every row is an ERROR in module setup, not a test assertion:

Exception: Command [docker compose … pull] return non-zero code 18:
  zoo2 Skipped - Image is already being pulled by node_with_environment_credentials

Exit 18 from docker compose pull, with the concurrent-pull "Skipped" messages, is registry contention during image fetch. Neither module touches CAS, and the diff changes no Dockerfile, image or package list. Rerun.

04812_clickhouse_disks_move_directory — unknown

Job: Stateless tests (amd_debug, parallel), 120266603873. The only named failure in a job that otherwise reports 13,514 passed.

Output mismatch (return code 83) on the missing-parent case: clickhouse-disks move src aa/bb/cc leaves src in place on test_disk_04812 but creates aa/cc on test_disk_plain_rewritable_04812.

The test exists unchanged on antalya-26.8 and is not touched by the series, which modifies neither clickhouse-disks, the local disk, nor plain-rewritable metadata. That points at pre-existing, but "points at" is not evidence: this needs the branch fail rate for this test on this job before it can be called pre-existing-flaky. Marked unknown deliberately.

test_cas_gcs::test_ordinary_goog4_traffic_keeps_upstream_semantics and test_storage_gcp_auth::test_gcp_auth_ordinary_contract — unknown

Job: Integration tests (amd_asan_ubsan, db disk, old analyzer, 3/8), 120268327952. Two failures in 978 tests.

test_cas_gcs/test.py:866: assert "if-match" not in headers, record

The assertion is that ordinary, non-CAS GCS traffic carries no conditional-write header. On this head it does.

The 13-commit series cannot be the cause: its only changes under src/IO/S3/ are comment and whitespace edits. Client.cpp loses a four-line comment above an unchanged isPreconditionFailedError early return; ReadBufferFromS3.cpp loses a comment; copyS3File.cpp loses two blank lines; S3ObjectStorage.cpp drops two unused declarations and folds a local into a return. No request is constructed differently.

The likely source is the CAS port this branch inherits — test_cas_gcs/test.py does not exist on antalya-26.8 at all, so it arrived with that lineage. Confirming that needs the fail rate for both tests on the base branch and on sibling CAS PRs. Marked unknown.

Worth saying plainly: if this is real, a conditional header on ordinary S3/GCS traffic is a correctness problem for non-CAS users, so it deserves a definite answer rather than a shrug.

DCO — policy, not a test

Every commit in the series carries Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>, and every one also carries Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> with no matching sign-off. DCO requires one sign-off per author, co-authors included. Either add Signed-off-by: Claude Fable 5.1 <noreply@anthropic.com> to the four affected commits or drop the co-author trailer.

Recommendation

Do not approve on CI grounds yet. Three things, in order:

1. Fix 05014 — rebase onto the current #2579 head (which carries the packed-storage failpoint) or pin min_bytes_for_full_part_storage in the test.
2. Fix test_cas_ref_move_compat so the assertion sees the log line the server already writes.
3. Resolve the three unknown rows against the CI database, starting with the two GCS tests.

Then rerun the two integration shards whose image pulls failed, and wait for Unit tests (asan_ubsan) and the CAS regression lane, which had not finished.