HIVE-30106: Iceberg: Column-stats ANALYZE spends most of its CPU on per-row reader plumbing by deniskuzZ · Pull Request #6834 · apache/hive

deniskuzZ · 2026-09-30T19:56:37Z

What changes were proposed in this pull request?

Skip scan-side basic stats when the storage handler supplies them itself (canProvideBasicStatistics(Table)), so the ANALYZE scan vectorizes
Iceberg row-mode reads set PositionDeleteInfo and RowLineageInfo on the IOContext instead of round-tripping them through the Configuration per row
MapredIcebergRecordReader.getPos() returns -1: a split is a group of file scan tasks, no offset maps to a file position
Vectorized reads of a snapshot with equality deletes fall back to row mode (current snapshot, branch or tag)

Why are the changes needed?

Slow Iceberg: column-stats ANALYZE

Does this PR introduce any user-facing change?

No

How was this patch tested?

…er-row reader plumbing - Skip scan-side basic stats when the storage handler supplies them itself (canProvideBasicStatistics(Table)), so the ANALYZE scan vectorizes - Iceberg row-mode reads set PositionDeleteInfo and RowLineageInfo on the IOContext instead of round-tripping them through the Configuration per row - MapredIcebergRecordReader.getPos() returns -1: a split is a group of file scan tasks, no offset maps to a file position - Vectorized reads of a snapshot with equality deletes fall back to row mode (current snapshot, branch or tag)

…n; one Table per stats method - HiveIcebergStorageHandler: vectorizationUnsupported and hasEqualityDeletes return the decision, disableVectorization acts on it; the read path checks both, the write path only the first - BasicStatsNoJobTask: updatePartitions returns the table it wrote, and aggregateStats hands its parameters to the task's table, as BasicStatsTask does

Copilot

🟡 Changes recommended

Equality-delete detection misses FOR SYSTEM_TIME and FOR SYSTEM_VERSION snapshots, allowing unsupported vectorized reads.

2 open findings

What changed in this PR

Optimizes Iceberg column-statistics analysis by removing per-row configuration plumbing and enabling vectorized scans where safe.

Changes:

Uses storage-handler basic statistics instead of scan-side gathering.
Passes Iceberg virtual-column metadata through IOContext.
Falls back to row mode for snapshots containing equality deletes and expands tests.

File	Description
`ql/.../stats/BasicStatsNoJobTask.java`	Integrates handler-owned statistics with column statistics.
`ql/.../parse/TaskCompiler.java`	Selects no-job statistics for authoritative handlers.
`ql/.../parse/SemanticAnalyzer.java`	Skips redundant scan-side basic statistics.
`ql/.../metadata/HiveStorageHandler.java`	Adds table-aware statistics capability API.
`ql/.../io/RowLineageInfo.java`	Removes configuration serialization.
`ql/.../io/PositionDeleteInfo.java`	Removes configuration serialization.
`ql/.../io/IOContext.java`	Adds direct virtual-column metadata setters.
`ql/.../io/HiveContextAwareRecordReader.java`	Removes per-row configuration parsing.
`itests/.../testconfiguration.properties`	Registers the new LLAP query test.
`.../vectorized_iceberg_read_parquet.q.out`	Verifies vectorized ANALYZE planning.
`.../llap/vectorized_iceberg_read_parquet.q.out`	Verifies LLAP vectorized ANALYZE planning.
`.../llap/iceberg_row_mode_virtual_columns.q.out`	Records row-mode virtual-column results.
`.../iceberg_stats.q.out`	Records expanded statistics scenarios.
`.../vectorized_iceberg_read_parquet.q`	Adds ANALYZE vectorization coverage.
`.../iceberg_stats.q`	Adds basic-statistics and delete coverage.
`.../iceberg_row_mode_virtual_columns.q`	Tests row-mode metadata propagation.
`TestHiveIcebergTimeTravel.java`	Tests branch and tag equality deletes.
`TestHiveIcebergStatistics.java`	Tests ANALYZE with equality deletes.
`MapredIcebergInputFormat.java`	Returns no synthetic split position.
`IcebergAcidUtil.java`	Writes row metadata directly to `IOContext`.
`HiveIcebergStorageHandler.java`	Detects equality deletes and controls vectorization.

🧠 Review effort: Balanced

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

+    Snapshot snapshot = IcebergTableUtil.getTableSnapshot(
+        IcebergTableUtil.getTable(conf, tableProps), tableProps.getProperty(Catalogs.SNAPSHOT_REF));
+    return snapshot != null && PropertyUtil.propertyAsLong(snapshot.summary(), TOTAL_EQ_DELETES_PROP, 0) > 0;


    @Override
    public long getPos() throws IOException {
-      return (long) (splitLength * getProgress());
+      return -1;


Copilot

🔵 Needs a closer look

Equality-delete statistics and as-of time-travel scans can produce incorrect results.

2 open findings

Previously missed (1)

In code that hasn't changed since last review

Mark equality-delete snapshots inaccurate for basic statistics

iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:452

When the Iceberg stats source is selected, this returns true even for a snapshot with equality deletes. The no-job collector then marks computeBasicStatistics() as accurate, but estimateNumRows() deliberately cannot subtract equality deletes and returns the total record count. Since this PR routes column ANALYZE through that no-job path, analyzing three records with one equality-deleted row publishes numRows=3 instead of the scan's exact value 2. Treat such snapshots as unable to provide basic statistics so the already row-mode scan gathers the exact count.

🧠 Review effort: Balanced

…once - canUseBasicStats routes by storageHandlerOwnsBasicStats, TaskCompiler included - metastore stats source: a plain ANALYZE scans instead of marking the commit-time row count accurate - BasicStatsTask applies only an ANALYZE scan's count to handler-provided stats - drop the unreachable partition check in HiveStorageHandlerStatCollector - run the equality-delete CRUD tests vectorized

…er-row reader plumbing - the Iceberg row-mode reader hands each row's position delete and row lineage info to the IOContext as objects, instead of writing them into the Configuration and parsing them back for every row - ANALYZE planning and basic stats of a storage-handler table: SemanticAnalyzer, TaskCompiler, BasicStatsNoJobTask (PR apache#6834) - Impact (cluster, 1TRC 1T-row column ANALYZE): before 110-140K rows/s/task with 10 of 12 threads in reader plumbing (extrapolated 4.5-5 h, 40 x r5d); after 292.8 s (64 x r5d)

sonarqubecloud · 2026-10-12T19:57:18Z

Quality Gate passed

Issues
4 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.8% Duplication on New Code

See analysis details on SonarQube Cloud

…er-row reader plumbing - the Iceberg row-mode reader hands each row's position delete and row lineage info to the IOContext as objects, instead of writing them into the Configuration and parsing them back for every row - ANALYZE planning and basic stats of a storage-handler table: SemanticAnalyzer, TaskCompiler, BasicStatsNoJobTask - every ANALYZE routes by basic-stats ownership (canUseBasicStats via storageHandlerOwnsBasicStats, TaskCompiler included); a plain ANALYZE scans instead of marking the commit-time row count accurate; BasicStatsTask applies only an ANALYZE scan's count to handler-provided stats (iceberg_stats.q counted 5 rows for 3) - vectorized reads of a snapshot with equality deletes fall back to row mode; the equality-delete CRUD tests run vectorized - (PR apache#6834)

asf-ci-hive added the tests pending label Sep 30, 2026

deniskuzZ changed the title ~~HIVE-30106: Iceberg: column-stats ANALYZE spends most of its CPU on per-row reader plumbing~~ HIVE-30106: Iceberg: Column-stats ANALYZE spends most of its CPU on per-row reader plumbing Sep 30, 2026

asf-ci-hive added tests failed tests pending and removed tests pending tests failed labels Sep 30, 2026

asf-ci-hive added tests pending and removed tests failed labels Oct 10, 2026

deniskuzZ requested a balanced review from Copilot October 10, 2026 05:52

Copilot started reviewing on behalf of deniskuzZ October 10, 2026 05:52

View session

Copilot AI reviewed Oct 10, 2026

View reviewed changes

asf-ci-hive added tests failed tests pending and removed tests pending tests failed labels Oct 10, 2026

deniskuzZ requested a balanced review from Copilot October 11, 2026 15:37

Copilot started reviewing on behalf of deniskuzZ October 11, 2026 15:38

View session

Copilot AI reviewed Oct 11, 2026

View reviewed changes

deniskuzZ force-pushed the HIVE-30106 branch from 3601cfb to a4c4d13 Compare October 11, 2026 18:30

asf-ci-hive added tests passed and removed tests pending labels Oct 11, 2026

asf-ci-hive added tests pending tests failed and removed tests passed tests pending tests failed labels Oct 11, 2026

asf-ci-hive added tests failed tests pending and removed tests pending tests failed labels Oct 12, 2026

asf-ci-hive added tests passed and removed tests pending labels Oct 12, 2026

Navigation Menu

Repository navigation

HIVE-30106: Iceberg: Column-stats ANALYZE spends most of its CPU on per-row reader plumbing - #6834

HIVE-30106: Iceberg: Column-stats ANALYZE spends most of its CPU on per-row reader plumbing#6834
deniskuzZ wants to merge 3 commits into
apache:masterfrom
deniskuzZ:HIVE-30106

deniskuzZ commented Sep 30, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

sonarqubecloud Bot commented Oct 12, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

deniskuzZ commented Sep 30, 2026

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

sonarqubecloud Bot commented Oct 12, 2026

Quality Gate passed

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants