Read Parquet bloom filters in one request when a file has many row groups- #2526 by ianton-ru · Pull Request #2588 · Altinity/ClickHouse

ianton-ru · 2026-10-16T14:05:04Z

Separate GETs for each filter dominate cold reads from object storage. input_format_parquet_bloom_filter_coalesce_max_bytes merges contiguous filters, and reads a small file once when that is cheaper.

Changelog category (leave one):

Performance Improvement

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Read Parquet bloom filters in one request when a file has many row groups

Documentation entry for user-facing changes

ClickHouse can build Bloom-filter for Iceberg tables for queries with 'WHERE' condition not on key columns. For this ClickHouse reads metainfo for each row group from source files. In initial implementation it use single read operation for each row group, which is converted to separate GET request when data files are on object storage like S3. For large table it can produce thousands of requests. In customer test first request takes near 500 seconds, while second (when data are in local cache) less than 4 seconds.
This PR introduced reading metainfo for multiple row groups in single request, when possible. It can increase total traffic, but decrease number of requests, which can reduce time for query execution.

CI/CD Options

Exclude tests:

Regression jobs to run:

…oups. Separate GETs for each filter dominate cold reads from object storage. `input_format_parquet_bloom_filter_coalesce_max_bytes` merges contiguous filters, and reads a small file once when that is cheaper. Co-authored-by: Cursor <cursoragent@cursor.com>

… buffer. The bloom-filter header that starts the read is reset as soon as it is parsed, so charging that handle let the scheduler treat a live buffer as free. Related: #2526 Co-authored-by: Cursor <cursoragent@cursor.com>

That field is optional. Filters that only store an offset were left out of coalescing, even though their length is already derived later. Related: #2526 Co-authored-by: Cursor <cursoragent@cursor.com>

…or a whole-file read. Co-authored-by: Cursor <cursoragent@cursor.com>

…compatibility` values restore separate reads. Co-authored-by: Cursor <cursoragent@cursor.com>

…is below `bytes_per_read_task`. Co-authored-by: Cursor <cursoragent@cursor.com>

…and the filters after it stay inside `input_format_parquet_bloom_filter_coalesce_max_bytes`. Co-authored-by: Cursor <cursoragent@cursor.com>

….6 so older `compatibility` values restore separate reads, and count only the bloom ranges `Reader` registers. Co-authored-by: Cursor <cursoragent@cursor.com>

ianton-ru · 2026-10-16T14:06:04Z

@codex review

chatgpt-codex-connector · 2026-10-16T14:06:25Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-22T08:13:54.052151Z	`ac5724f`	Manual request

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

github-actions · 2026-10-16T14:06:45Z

Workflow [PR], commit [ac5724f]

chatgpt-codex-connector · 2026-10-16T14:26:36Z

Codex Review: Something went wrong. Try again later by commenting “@codex review”.

Failed to set up container

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

ianton-ru · 2026-10-16T15:42:33Z

@codex review

chatgpt-codex-connector · 2026-10-16T16:03:00Z

Codex Review: Something went wrong. Try again later by commenting “@codex review”.

Failed to set up container

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

…ilter-coalesce

ianton-ru · 2026-10-19T08:26:42Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 356269bf74

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-10-19T08:40:19Z

 | `input_format_parquet_filter_push_down`                                        | When reading Parquet files, skip whole row groups based on the WHERE/PREWHERE expressions and min/max statistics in the Parquet metadata.                                                                                          | `1`         |
 | `input_format_parquet_bloom_filter_push_down`                                  | When reading Parquet files, skip whole row groups based on the WHERE expressions and bloom filter in the Parquet metadata.                                                                                                          | `0`         |
 | `input_format_parquet_dictionary_filter_push_down`                             | When reading Parquet files (with reader v3), skip whole row groups based on the WHERE/PREWHERE expressions and the dictionary page contents, for equality and `IN` conditions, when all data pages of a column chunk are dictionary-encoded. The value is the maximum dictionary page size (in bytes) for which this optimization is applied; set to `0` to disable. Takes precedence over the bloom filter when both are available. | `1048576`   |
+| `input_format_parquet_bloom_filter_coalesce_max_bytes`                         | Max bytes of one read when loading Parquet bloom filters. Contiguous filters are read together. A file not larger than this, with many filters, is read once. 0 keeps one read per bloom filter.                                      | `67108864`  |


Remove the direct edit to generated Parquet documentation

This line is inside the page's AUTOGENERATED_START/AUTOGENERATED_END region, so editing it in this feature commit will fail the documentation check unless the PR is the dedicated autogeneration change. Keep the corresponding source-registration update in ParquetV3BlockInputFormat.cpp and let the documentation workflow regenerate this file.

AGENTS.md reference: docs/reference/AGENTS.md:L1-L3

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-10-19T08:40:19Z

+        const size_t prospective_start = std::min(start_offset, r.start);
+        const size_t prospective_end = std::max(end_offset, r.end);
+        const bool end_past_cap = same_group && prospective_end - prospective_start > span_limit;
+        if ((initial_req->coalesce_group != 0 && !same_group) ||


Isolate coalescing groups from ungrouped starter requests

When an ungrouped request starts first, this condition does not stop it from absorbing an adjacent grouped bloom-filter header because it checks only whether initial_req belongs to a group. This is reachable when a row group without an applicable bloom filter advances to index or data prefetching while another row group's bloom-header work remains queued; the bloom read then follows ordinary bytes_per_read_task rules and can exceed input_format_parquet_bloom_filter_coalesce_max_bytes (especially when configured below 4 MiB). Treat a group mismatch as a boundary whenever either request has a nonzero group, symmetrically in both scan directions.

Useful? React with 👍 / 👎.

…_bloom_filter_coalesce_max_bytes` row stays in the `Documentation` source. Co-authored-by: Cursor <cursoragent@cursor.com>

…ichever range starts the coalescing scan. Co-authored-by: Cursor <cursoragent@cursor.com>

ianton-ru · 2026-10-21T10:52:08Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 24138136ea

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-10-21T11:15:46Z

+        const bool group_boundary = initial_req->coalesce_group != r.request->coalesce_group
+            && (initial_req->coalesce_group != 0 || r.request->coalesce_group != 0);
+        if (group_boundary ||
+            r.end + gap_limit <= start_offset || // short gap


Preserve coalescing when the seek threshold is zero

When remote_read_min_bytes_for_seek=0 (or the corresponding local-file setting is zero), gap_limit is zero and this <= check stops at two exactly adjacent ranges because r.end == start_offset; the symmetric right-hand scan has the same behavior. ClickHouse writes bloom filters back-to-back in flushBloomFilters, so this valid configuration still issues one request per filter despite the new coalescing setting. Treat zero-byte gaps as contiguous independently of the ordinary seek threshold.

Useful? React with 👍 / 👎.

…or_seek` is 0. Co-authored-by: Cursor <cursoragent@cursor.com>

…ip a filter for a row group that produces no rows. Co-authored-by: Cursor <cursoragent@cursor.com>

…ilter-coalesce

ianton-ru · 2026-10-21T12:02:41Z

@codex review

chatgpt-codex-connector · 2026-10-21T12:23:15Z

Codex Review: Something went wrong. Try again later by commenting “@codex review”.

Failed to set up container

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

ianton-ru · 2026-10-21T13:20:04Z

@codex review

chatgpt-codex-connector · 2026-10-21T13:40:35Z

Codex Review: Something went wrong. Try again later by commenting “@codex review”.

Failed to set up container

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

…er row group. https: //altinity-build-artifacts.s3.amazonaws.com/json.html?PR=2588&sha=6e16e1d87e4b679d422fd7bd4b4f35df4e0bd4be&name_0=PR&name_1=Stateless%20tests%20%28amd_asan_ubsan%2C%20distributed%20plan%2C%20parallel%2C%20selected%20tests%29 https: //github.com//pull/2588 Co-authored-by: Cursor <cursoragent@cursor.com>

ianton-ru · 2026-10-21T15:47:02Z

@codex review

chatgpt-codex-connector · 2026-10-21T16:18:36Z

Codex Review: Something went wrong. Try again later by commenting “@codex review”.

Failed to set up container

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

ianton-ru · 2026-10-21T16:37:15Z

@codex review

chatgpt-codex-connector · 2026-10-21T16:57:40Z

Codex Review: Something went wrong. Try again later by commenting “@codex review”.

Failed to set up container

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

ianton-ru · 2026-10-22T07:56:38Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ac5724ff5d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-10-22T08:13:51Z

+            /// `end_past_cap` is the same-group cap. This addition wraps when the cap is `UInt64` max.
+            (!same_group && r.start + span_limit <= initial_offset) || // task not too big
+            end_past_cap ||
+            (!same_group && !r.request->allow_incidental_read.load(std::memory_order_relaxed))) // range wants to be coalesced


Preserve lazy bloom fallback for dictionary-filtered chunks

When a column chunk is eligible for dictionary filtering and also has a bloom filter, starting its header now absorbs the overlapping full bloom-filter body because same-group ranges bypass allow_incidental_read. This defeats the contract in Reader::initializePrefetches that only the small header is read unless dictionary pruning cannot be used: even when the exact dictionary filter succeeds, the query has already downloaded and pinned potentially every bloom body, causing substantial unnecessary object-storage traffic across many files. Keep fallback bloom bodies out of header-triggered coalescing until the fallback is actually selected.

Useful? React with 👍 / 👎.

ianton-ru and others added 8 commits October 16, 2026 15:36

Count Parquet bloom filters that omit bloom_filter_length.

57636c8

That field is optional. Filters that only store an offset were left out of coalescing, even though their length is already derived later. Related: #2526 Co-authored-by: Cursor <cursoragent@cursor.com>

Skip dictionary-filtered chunks when counting Parquet bloom filters f…

32a3f57

…or a whole-file read. Co-authored-by: Cursor <cursoragent@cursor.com>

Record Parquet bloom-filter coalescing as a 26.6.8 setting so older `…

77c5393

…compatibility` values restore separate reads. Co-authored-by: Cursor <cursoragent@cursor.com>

Honor input_format_parquet_bloom_filter_coalesce_max_bytes when it …

9226b75

…is below `bytes_per_read_task`. Co-authored-by: Cursor <cursoragent@cursor.com>

Cap a coalesced Parquet bloom read by the whole task so a bloom body …

a54aa46

…and the filters after it stay inside `input_format_parquet_bloom_filter_coalesce_max_bytes`. Co-authored-by: Cursor <cursoragent@cursor.com>

Record input_format_parquet_bloom_filter_coalesce_max_bytes in 26.8…

bd0f819

….6 so older `compatibility` values restore separate reads, and count only the bloom ranges `Reader` registers. Co-authored-by: Cursor <cursoragent@cursor.com>

ianton-ru mentioned this pull request Oct 16, 2026

Read Parquet bloom filters in one request when a file has many row groups #2526

Closed

9 of 30 tasks

ianton-ru added antalya antalya-26.8 Session label (releasy session config) labels Oct 16, 2026

Merge branch 'antalya-26.8' into feature/antalya-26.8/parquet-bloom-f…

356269b

…ilter-coalesce

chatgpt-codex-connector Bot reviewed Oct 19, 2026

View reviewed changes

ianton-ru and others added 2 commits October 21, 2026 12:41

Leave the generated Parquet page unchanged. The `input_format_parquet…

8b8061d

…_bloom_filter_coalesce_max_bytes` row stays in the `Documentation` source. Co-authored-by: Cursor <cursoragent@cursor.com>

Keep a Parquet bloom-filter read out of a neighboring column read, wh…

2413813

…ichever range starts the coalescing scan. Co-authored-by: Cursor <cursoragent@cursor.com>

chatgpt-codex-connector Bot reviewed Oct 21, 2026

View reviewed changes

ianton-ru and others added 3 commits October 21, 2026 13:27

Keep back-to-back Parquet bloom filters in one read when `min_bytes_f…

0eb108c

…or_seek` is 0. Co-authored-by: Cursor <cursoragent@cursor.com>

Keep a max Parquet bloom coalesce cap from splitting a filter, and sk…

b5be899

…ip a filter for a row group that produces no rows. Co-authored-by: Cursor <cursoragent@cursor.com>

Merge branch 'antalya-26.8' into feature/antalya-26.8/parquet-bloom-f…

6e16e1d

…ilter-coalesce

chatgpt-codex-connector Bot reviewed Oct 22, 2026

View reviewed changes

Conversation

ianton-ru commented Oct 16, 2026

Changelog category (leave one):

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Documentation entry for user-facing changes

CI/CD Options

Exclude tests:

Regression jobs to run:

Uh oh!

ianton-ru commented Oct 16, 2026

Uh oh!

chatgpt-codex-connector Bot commented Oct 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

github-actions Bot commented Oct 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chatgpt-codex-connector Bot commented Oct 16, 2026

Uh oh!

ianton-ru commented Oct 16, 2026

Uh oh!

chatgpt-codex-connector Bot commented Oct 16, 2026

Uh oh!

ianton-ru commented Oct 19, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Oct 19, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Oct 19, 2026

Choose a reason for hiding this comment

Uh oh!

ianton-ru commented Oct 21, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Oct 21, 2026

Choose a reason for hiding this comment

Uh oh!

ianton-ru commented Oct 21, 2026

Uh oh!

chatgpt-codex-connector Bot commented Oct 21, 2026

Uh oh!

ianton-ru commented Oct 21, 2026

Uh oh!

chatgpt-codex-connector Bot commented Oct 21, 2026

Uh oh!

ianton-ru commented Oct 21, 2026

Uh oh!

chatgpt-codex-connector Bot commented Oct 21, 2026

Uh oh!

ianton-ru commented Oct 21, 2026

Uh oh!

chatgpt-codex-connector Bot commented Oct 21, 2026

Uh oh!

ianton-ru commented Oct 22, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Oct 22, 2026

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

chatgpt-codex-connector Bot commented Oct 16, 2026 •

edited

Loading

github-actions Bot commented Oct 16, 2026 •

edited

Loading