Repository navigation
CAS: packed parts - #2579
CAS: packed parts#2579k-morozov wants to merge 73 commits into
Conversation
…next commit) --- Original cherry-pick message follows: Merge pull request #2159 from Altinity/feature/antalya-26.6/CAS [experimental] CAS (Content-addressed storage) over shared object storage for antalya-26.6 # Conflicts: # .github/workflows/fast_builds.yml # .github/workflows/master.yml # .github/workflows/pull_request.yml # .github/workflows/pull_request_community.yml # ci/jobs/functional_tests.py # ci/jobs/scripts/clickhouse_proc.py # ci/jobs/scripts/clickhouse_service.py # ci/workflows/pull_request.py # ci/workflows/pull_request_community.py # ci/workflows/release_branches.py # docs/concepts/features/configuration/server-config/storing-data.mdx # programs/disks/DisksApp.cpp # programs/disks/ICommand.h # programs/server/Server.cpp # src/Common/FailPoint.cpp # src/Common/ThreadStatus.h # src/Core/ServerSettings.cpp # src/Disks/DiskObjectStorage/DiskObjectStorage.cpp # src/Disks/DiskObjectStorage/DiskObjectStorage.h # src/Disks/DiskObjectStorage/DiskObjectStorageTransaction.cpp # src/Disks/DiskObjectStorage/MetadataStorages/IMetadataStorage.h # src/Disks/DiskObjectStorage/MetadataStorages/MetadataStorageFactory.cpp # src/Disks/DiskObjectStorage/ObjectStorages/IObjectStorage.h # src/Disks/DiskObjectStorage/ObjectStorages/Local/LocalObjectStorage.cpp # src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.cpp # src/Disks/DiskObjectStorage/RegisterDiskObjectStorage.cpp # src/Disks/IDiskTransaction.h # src/IO/ReadPipeline.cpp # src/IO/ReadPipeline.h # src/IO/S3/copyS3File.cpp # src/IO/S3/copyS3File.h # src/IO/S3Common.cpp # src/IO/WriteBufferFromS3.cpp # src/IO/tests/gtest_writebuffer_s3.cpp # src/Interpreters/InterpreterSystemQuery.cpp # src/Interpreters/ServerAsynchronousMetrics.cpp # src/Parsers/ASTSystemQuery.h # src/Parsers/tests/gtest_Parser.cpp # src/Storages/MergeTree/DataPartStorageOnDiskBase.cpp # src/Storages/MergeTree/DataPartStorageOnDiskFull.cpp # src/Storages/MergeTree/DataPartsExchange.cpp # src/Storages/MergeTree/IMergeTreeDataPart.cpp # src/Storages/MergeTree/MergeTask.cpp # src/Storages/MergeTree/MergeTreeData.cpp # tests/integration/test_replicated_database/test.py # tests/queries/0_stateless/02253_empty_part_checksums.sh # tests/queries/0_stateless/02254_projection_broken_part.sh # tests/queries/0_stateless/02255_broken_parts_chain_on_start.sh # tests/queries/0_stateless/02369_lost_part_intersecting_merges.sh # tests/queries/0_stateless/02370_lost_part_intersecting_merges.sh # tests/queries/0_stateless/02444_async_broken_outdated_part_loading.sh # tests/queries/0_stateless/04215_replicated_missing_covered_part_on_start.sh # tests/queries/0_stateless/04327_reader_executor_metrics.sql # tests/queries/0_stateless/04328_reader_executor_kpi_async_metric.sql
Source PR: #2159 - `.github/workflows/*.yml` are generated: kept at the base version; the "Check Workflows" job regenerates them from `ci/workflows/*.py` (`python3 -m praktika yaml`), which now include the CAS lanes. - Stateless tests 02253/02254/02255/02369/02370/02444/04215 stay deleted: `antalya-26.8` converted them to integration tests (7f7a30e). - `ci/defs/altinity_jobs.py`: dropped the duplicated `cas_functional_tests_jobs` (already on base from 5aaf212). Adapted: DiskObjectStorageTransaction - PR's `dispatch` mapped onto base `addOperation` (made eager also for `transactionIsStagingOverlay`), keeps `appliesOperationsEagerly` for in-memory metadata Adapted: RegisterDiskObjectStorage - `use_fake_transaction` selection dropped, fake transactions were removed on base (c6254c3) Adapted: copyS3File - `copy_mode` threaded through base `copyS3FileImpl`; ranged copies use `ObjectStorageCopyMode::Default` Adapted: DataPartStorageOnDiskFull - in-flight reads moved into the base `*Impl` overrides (packed skip-index overlay lives in the base wrapper) Adapted: MergeTask/IMergeTreeDataPart - `use_parent_transaction` passed to base `getProjectionNoInitialize` Adapted: DataPartsExchange - kept base `REPLICATION_PROTOCOL_VERSION_WITH_INVALIDATED_SYSTEM_COLUMNS = 10`, negotiation tops out at `WITH_CA_CONFIRM = 11` Adapted: MergeTreeData - empty covering part already commits its disk transaction on base, PR's CAS-only commit dropped as redundant Adapted: functional_tests.py - RustFS start folded into base SeaweedFS-based `start` Source-PR: #2159 (#2159)
…next commit) --- Original cherry-pick message follows: Merge pull request #2300 from Altinity/feature/antalya-26.6/CAS-improvements CAS improvements # Conflicts: # src/Common/ErrorCodes.cpp # src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.cpp # src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.h # src/IO/ReadBufferFromS3.cpp # src/IO/ReadSettings.h # src/IO/S3/tests/TestPocoHTTPServer.h # src/IO/WriteSettings.h
- `ErrorCodes.cpp`: kept antalya-26.8 numbering (1009-1018) and the `CLICKHOUSE_PARSER_MINIMAL_BUILD` guard; added the PR's `CAS_WRITE_UNATTRIBUTED` (1037) and `CAS_DELETE_MARKER` (1038), END = 1038. Dropped the PR's comments describing the antalya-26.6 1009-1011 layout, which does not exist on antalya-26.8. - `S3ObjectStorage`: combined the PR's `client_slot`/`client` reference with antalya-26.8's `client_restricts_server_credentials`, the iterator's `description` with the PR's `attempt_seed`, and antalya-26.8's `UnknownSize`-aware size and `object.etag` args with the PR's `refresh_callback`. - `TestPocoHTTPServer.h`: kept `makeMockServerParams` and added the PR's dedicated thread pool and destructor. - Adapted the PR's `S3::setClickhouseAttemptNumber` calls to `S3::setClickHouseAttemptNumber`, renamed on antalya-26.8 by e8e1e12. Adapted: setClickhouseAttemptNumber -> setClickHouseAttemptNumber (renamed by e8e1e12 on antalya-26.8) Source-PR: #2300 (#2300)
…next commit) --- Original cherry-pick message follows: Merge PR #2307 (cas/configurable-mount-lease-timing) into feature/antalya-26.6/CAS-improvements-cicd-fixes Brings k-morozov's configurable CAS mount lease TTL / renewal period (#2307) so the heartbeat-model work can land on top of it while keeping the author's commits intact. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GhVd7eMAWdFubNk4g1B2Tx # Conflicts: # src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.cpp Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com> # Conflicts: # docs/en/antalya/cas/configuration.md # src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.cpp # src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.h # src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedSettings.cpp
All changes from #2307 (`mount_lease_ttl_ms` / `mount_renew_period_ms` settings, members, wiring into `PoolConfig`, validation, docs) are already present on the base. The conflicting hunks keep the base side, which has later additions on top of them (`unsafe_remount_no_delay`, the `attempt_timeout_ms` check, `connect_timeout_cap_ms`, and the refined `cas_mount_renew_period_ms` doc text). Source-PR: #2307 (#2307)
…next commit) --- Original cherry-pick message follows: Merge pull request #2349 from Altinity/cas/memworker-split-allocated-source MemoryWorker: use sanitizer info for allocated counter # Conflicts: # src/Common/MemoryWorker.cpp
…next commit) --- Original cherry-pick message follows: Merge pull request #2385 from Altinity/fix/antalya-26.6/lldb-budget-and-sigcont CI: lldb stack budget by build flavor, and keep lldb's JIT breakpoint out of the server # Conflicts: # ci/tests/test_print_stacktraces.py # tests/clickhouse-test
tests/clickhouse-test: applied the PR's lldb budget and JIT-loader flag on top of the antalya-26.8 shape of get_stacktraces_from_lldb (sudo/shutil.which lldb path and keep_output_on_error kept; no _ensure_lldb_installed, which antalya-26.8 does not have). Docstring keeps the 'liveness check' call site that exists on antalya-26.8. ci/tests/test_print_stacktraces.py: kept deleted, as on antalya-26.8, where 3a69545 removed all of ci/tests/. The PR's test changes to that file are therefore not carried over. Source-PR: #2385 (#2385)
…next commit) --- Original cherry-pick message follows: Merge pull request #2292 from Altinity/enable-cas-s3-stress Enable cas s3 stress # Conflicts: # .github/workflows/master.yml # .github/workflows/pull_request.yml # ci/tests/test_stress_job.py
Kept the antalya-26.8 `needs` lists of `finish_workflow` in `master.yml` and `pull_request.yml` and added only the four `stress_test_amd_*_cas_s3_storage` jobs introduced by the source PR. Kept both `ensure_ripgrep_installed` and `get_additional_envs` imports in `ci/tests/test_stress_job.py`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Source-PR: #2292 (#2292)
…next commit) --- Original cherry-pick message follows: Merge pull request #2444 from Altinity/docs/antalya-26.6/metrics-and-events Document Antalya-only profile events and metrics. # Conflicts: # src/Common/ProfileEvents.cpp
…next commit) --- Original cherry-pick message follows: Merge pull request #2511 from Altinity/fix/antalya-26.6/cas-renewal-memory-guard Keep the CAS mount lease renewal alive over the server memory limit # Conflicts: # src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasMountRuntime.cpp
The port references removed DEB_AMD_* artifacts and generated workflow jobs with stale dependencies on antalya-26.8. Source-PR: #2292 (#2292) Review: https://gist.github.com/k-morozov/0e7a4ee2ce4675fbf22c01e79adfb350
Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com> # Conflicts: # programs/local/LocalServer.cpp # programs/server/Server.cpp # src/Common/ErrorCodes.cpp # src/Common/FailPoint.cpp # src/Core/ServerSettings.cpp # src/Interpreters/InterpreterSystemQuery.cpp # src/Storages/ObjectStorage/DataLakes/Iceberg/AlterDropPartitionExecutor.cpp # src/Storages/ObjectStorage/DataLakes/Iceberg/AlterDropPartitionExecutor.h # src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergMetadata.cpp
…acked-parts Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com> # Conflicts: # .github/workflows/master.yml # .github/workflows/pull_request.yml
Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>
|
@k-morozov does it supersede #2540 ? or this is a standalone work? maybe I exclude that PR from the scope? |
Yes, this is standalone work and will be merged after #2540. |
PR #2579 CI VerificationVerdictNo failure is attributable to this PR. Every one of the PR's own tests passes. CI does not block approval of #2579, but it does not clear the stack either: six of The bigger problem is not a red check. The ten
How the list was reducedEight jobs are red. 227 individual rows. 1. Setup errors, 225 rows, one cause. Shards 1/8 (91 errors) and 6/8 (134 errors) are That image is the old-version node used by backward-compatibility fixtures, and it is no 2. Three real integration failures, two distinct assertions. After the setup errors, 3. Four regression-suite jobs. Findings
|
| Feature | On 26.8 |
Everything else |
|---|---|---|
/oauth/access control |
64 fails / 64 runs (100 %) | 0 / 473 |
/oauth/groups |
64 fails / 64 runs (100 %) | 0 / 473 |
It fails on the branch itself (REFs/antalya-26.8/ab9c4003d72ad373f88f9695b) and on
unrelated PRs #2545 and
#2588. antalya-26.8 broke it; #2579
inherits it. Someone needs to own it, but not in this PR.
/swarms/…/wrong cluster name and /swarms/…/cpu overload all swarm nodes — regression, owner antalya-26.8
Same shape. Both are 109 fails / 109 runs (100 %) on 26.8, and 0 / 584 everywhere
else, across 10 distinct commits. The error is
Code: 279. DB::Exception: Cannot connect to any replica for query execution.
(ALL_CONNECTION_TRIES_FAILED)
(query: SELECT hostName(), count() FROM s3Cluster('swarm', 'http://minio:9000/warehouse/…'))
Nothing in this PR touches cluster discovery, s3Cluster, or swarm node startup.
test_storage_gcp_auth::test_gcp_auth_ordinary_contract and test_cas_gcs::test_ordinary_goog4_traffic_keeps_upstream_semantics — regression, owner the antalya-26.8 CAS port
These two are the ones worth reading carefully, because the naive rate comparison points
straight at this PR and is wrong.
| Test | Branch runs (pr = 0) |
Branch fails | #2579 runs | #2579 fails |
|---|---|---|---|---|
test_gcp_auth_ordinary_contract |
86 | 4 | 7 | 7 |
test_ordinary_goog4_traffic_keeps_upstream_semantics |
85 | 0 | 7 | 7 |
Never-fails-on-branch against 7-of-7-here is the textbook PR-caused pattern. It does not
hold here, for two reasons.
The baseline is the wrong branch. Every pull_request_number = 0 run of these two tests
is on antalya-26.6 or a v26.6.* tag. There is no antalya-26.8 branch run for either
test, because the CAS code that ships them has not landed on antalya-26.8 — it lives on
the stacked branch this PR sits on. The baseline measures a branch that cannot run the code
under test.
A sibling PR fails identically.
#2647 (CAS: ref-log owner Move for part renames, feature/antalya-26.8/cas-disk-transactions → antalya-26.8) fails both tests on
2026-10-23, and fails the same 26 test_database_backup rows. #2647 shares nothing with
#2579 except the base.
The mechanism has no path from this diff. Both tests fail on the same assertion — an
If-Match header present on an ordinary, non-CAS object-store request:
assert 'if-match' not in headers
E AssertionError: {'headers': {..., 'authorization': 'GOOG4-HMAC-SHA256 …'}, 'key': 'ordinary/ordinary.txt'}
Whatever adds that header lives in S3ObjectStorage or under src/IO/S3/. This PR's own
commits touch neither; its only I/O changes are ReadBufferFromFileView and
PackedFilesReader, and neither sets HTTP headers.
/lightweight delete/immediate removal/…/delete most of the table — pre-existing-flaky
One scenario in each CAS lane: VersionedCollapsingMergeTree in cas_lightweight_delete_3,
MergeTree in cas_s3_cache_lightweight_delete_3. The failing line is a wall-clock budget,
not a correctness check — the data assertion one line above it passed:
r = node.query(f"SELECT count(*) FROM {table_name} FORMAT TabSeparated")
assert r.output == output, error()
assert execution_time < 20, error() # todo rewrite valueThe scenario ran for about 40 s against 100 partitions × 100 000 rows.
| Scenario | Lane | Runs | Fails | Rate |
|---|---|---|---|---|
MergeTree/delete most of the table |
cas_s3_cache_lightweight_delete_3 |
179 | 6 | 3.4 % |
VersionedCollapsingMergeTree/delete most of the table |
cas_s3_cache_lightweight_delete_3 |
179 | 5 | 2.8 % |
VersionedCollapsingMergeTree/delete most of the table |
cas_lightweight_delete_3 |
181 | 3 | 1.7 % |
MergeTree/delete most of the table |
cas_lightweight_delete_3 |
181 | 2 | 1.1 % |
| every engine | suite (non-CAS) |
275 | 1–5 each | 0.4–1.8 % |
It fails on the plain non-CAS lane too. Two lanes each tripping one of seven engines is what
a ~2 % per-scenario rate predicts. A deterministic code defect would hit the same engine in
both lanes; these hit different ones.
/swarms/…/object storage cluster profile events with overloaded node — pre-existing-flaky
6.6 % on 26.8 (7 of 106) against 5.4 % elsewhere (31 of 575), over 13 distinct commits. The
rates are the same on both sides.
What passed, and what was never run
The feature's own tests all pass, on both the debug and the ASan/UBSan stateless lanes:
| Test | Result |
|---|---|
05217_cas_packed_parts |
OK |
05218_cas_packed_fetch_partition |
OK |
05219_cas_packed_backup_restore |
OK |
05220_cas_unique_key_mutation_keeps_full |
OK |
05301_cas_mutation_preserves_source_storage |
OK |
05385_cas_packed_merge_disabled_destination |
OK |
test_cas_s3/test.py::test_cas_s3_packed_parts |
OK |
Unit tests (asan_ubsan) (23 631 rows, incl. the new CAS and file-view gtests) |
success |
The full CAS regression suite ran and was green apart from the two flaky lightweight-delete
scenarios above: cas, cas_s3_cache, cas_aggregate_functions_*, cas_alter_*,
cas_atomic_insert, cas_engines, and the rest.
Two gaps, and they are the reason not to read the green as full coverage.
The CAS stateless lanes were skipped. All ten are SKIPPED on this PR — amd_binary, cas storage, parallel; amd_binary, cas s3 storage, parallel; arm_binary, cas s3 storage, parallel; and the amd_asan_ubsan, amd_tsan, amd_msan batches. The database has zero
rows for any of them on any commit of #2579, while other antalya-26.8 CAS PRs
(#2644,
#2623,
#2614) run them with tens of thousands of
rows each. This PR sets cas_allow_packed_parts = 1 in
tests/config/config.d/cas_storage_policy_for_merge_tree_by_default.xml and
cas_s3_storage_policy_for_merge_tree_by_default.xml — making the whole stateless suite use
packed MergeTree parts on CAS — and that configuration was never executed.
The PR removes the CAS stress lanes. Commit 762c37ae ("Revert port of #2292") deletes
Stress test (amd_debug | amd_asan_ubsan | amd_tsan | amd_msan, cas s3 storage) from
ci/defs/job_configs.py, along with the USE_CAS_S3_STORAGE_FOR_MERGE_TREE plumbing in
ci/jobs/stress_job.py and its unit tests. Those four lanes exist on the base branch and run
on #2644, #2623 and #2614. The PR description does not mention the removal.
The CAS regression lanes do not cover packed parts. cas_allow_packed_parts appears
nowhere in the clickhouse-regression repository, so every green Regression release cas_*
job ran with packed parts off. Those jobs are still meaningful — they exercise the refactored
*InTransactionOrDisk read helpers on Full parts over CAS, which is a live behaviour change
in this PR — but they say nothing about the packed path.
Recommendations
- Un-skip the CAS stateless lanes and re-run, or state explicitly why packed-on-CAS
across the whole stateless suite does not need to run before merge. This is the one item
that would change the verification result. - Explain or revert the deletion of the four CAS stress lanes, which is unrelated to
packed parts and silently reduces coverage on the base branch. - Resolve the merge conflict with
antalya-26.8and add the DCO sign-off. - Rerun integration shards 1/8 and 6/8 once the
clickhouse-server:23.3.19.33.altinitystable
image is back, or pin the compatibility fixtures to an image that still exists. This is a
branch-wide problem, not a CAS: packed parts #2579 problem. - The
/oauth,/swarmsand GCSIf-Matchbreaks need owners onantalya-26.8and its CAS
port. They are 100 % reproducible there and will block every PR on that branch until fixed.
PR #2579 Audit ReviewVerdictNo High or Medium distributed-systems defect in reviewed scope. Four Low findings, all in the resource and configuration/observability classes. The change that One coverage gap worth closing before merge: nothing exercises two servers with two separate CAS What the change doesA CAS disk previously forced every new MergeTree part to 1. Opt-in capability. New setting 2. The gate moved from the policy to the destination disk. 3. Transaction-aware reads, shared between the two part storages. The read-your-writes 4. Reading the archive through a buffer instead of a disk path. 5. Borrowed transactions. A packed projection sub-part that rides the parent's transaction 6. 7. Model
Why replicas cannot diverge, despite choosing the format independently
A part's On the fetch path the two formats are handled separately and deliberately The remaining question is ordering inside a single build: a borrowed packed projection must get FindingsBackpressure & resource limitsLow: a packed part on CAS is built entirely in memory and published by one commit, and mutating
Configuration & version skewLow: the capability gates creation only, so a move imports packed parts onto a disk configured
Observability & diagnosabilityLow: an in-flight file with no blob yet is reported as a file with no remote paths.
Low:
Coverage summary
Deferred: none. Assumptions and limits: static review of the PR's own commits at head, with no build and no Coverage gap to close before mergeNo test exercises two servers with two separate CAS pools exchanging a packed part. A two-node integration test — insert on node 1, let node 2 fetch, |
Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
Added basic support for Packed parts on CAS disks.
Documentation entry for user-facing changes
...
CI/CD Options
Exclude tests:
Regression jobs to run: