Skip to content

[format] Decline ORC pushdown of empty IN/NOT IN instead of crashing the reader - #10015

Merged
JingsongLi merged 1 commit into
apache:masterfrom
zhuxiangyi:orc-empty-in-crash
Sep 21, 2026
Merged

JingsongLi merged 1 commit into
apache:masterfrom
zhuxiangyi:orc-empty-in-crash

Conversation

@zhuxiangyi

Copy link
Copy Markdown
Contributor

Purpose

Bug fix: any ORC read that carries an IN () or NOT IN () predicate fails while the reader is being created.

PredicateBuilder.in(idx, emptyList()) is a legitimate always-false leaf (the builder already special-cases an empty list, exactly as it does for more than 20 literals), and notIn inherits it through negate(). But OrcPredicateFunctionVisitor.visitIn turns that empty list into an empty Object[] and hands it to Hive's SearchArgument.Builder.in(...), which rejects a zero-length call outright:

java.lang.IllegalArgumentException: Can't create in expression with no arguments
	at org.apache.hadoop.hive.ql.io.sarg.SearchArgumentImpl$BuilderImpl.in(SearchArgumentImpl.java:419)
	at org.apache.paimon.format.orc.filter.OrcFilters$In.add(OrcFilters.java:328)
	at org.apache.paimon.format.orc.OrcReaderFactory.createRecordReader(OrcReaderFactory.java:466)

So a predicate that should simply match nothing (or everything, for NOT IN ()) instead takes the whole read down. Nothing nested or Transform-based is needed to hit it — a plain index-based in/notIn on an ORC table is enough.

The fix has visitIn decline the pushdown (Optional.empty()) on an empty literal list, the same way the visitor already declines every other predicate it cannot express in a SARG. visitNotIn delegates to visitIn, so it is covered by the same branch. Correctness is unaffected: engines re-evaluate data filters after the scan (Spark keeps every data filter as a post-scan predicate; Paimon's own row-level filtering is opt-in via TableRead.executeFilter()), so the always-false / always-true semantics still hold — only the stripe-level pruning for that one predicate is given up.

This is the ORC counterpart of the ParquetFilters.visitIn/visitNotIn guard added in #9423 (parquet-mr's SetColumnFilterPredicate refuses an empty set the same way). It is independent of that PR and predates it.

Tests

  • OrcFilterConverterTest.testInPredicateWithEmptyValuesIsNotPushedDownin(idx, []) and notIn(idx, []) both resolve to Optional.empty(); also pins that an empty In leaf inside or(...)/and(...) declines the whole compound rather than crashing (the builder only folds AlwaysFalse.INSTANCE, so the leaf does survive into a compound).
  • OrcFormatReadWriteTest.testEmptyInAndNotInPredicatesDoNotCrashTheReader — writes a real ORC file and opens a reader with each predicate through OrcFileFormat.createReaderFactory; reproduces the IllegalArgumentException before the fix, returns every row unfiltered after it.

API and Format

No changes.

Documentation

No changes.

…the reader

PredicateBuilder.in(idx, emptyList()) is a legitimate always-false leaf, and
notIn inherits it through negate(). OrcPredicateFunctionVisitor.visitIn turned
that empty list into an empty Object[] and handed it to Hive's
SearchArgument.Builder.in(...), which rejects a zero-length call with
IllegalArgumentException("Can't create in expression with no arguments") while
the reader is being created, so any ORC read carrying IN () or NOT IN ()
failed outright.

visitIn now declines the pushdown (Optional.empty()) on an empty literal list,
the same way it already declines every other predicate it cannot express.
Correctness is unaffected: engines re-evaluate data filters after the scan,
so the always-false / always-true semantics still hold; only the stripe-level
pruning for that predicate is given up.

@JingsongLi JingsongLi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requirement fit: SUPPORTED (triage: GO)\nImplementation: CLEAN\n\nThis fixes a concrete ORC reader-construction failure. Declining SARG pushdown for empty IN and NOT IN lists is safe because the engine still evaluates the original data predicate, preserving always-false and always-true semantics respectively. The integration tests exercise the real ORC path; no actionable P0/P1/P2 issue found.

@JingsongLi
JingsongLi merged commit 8336f1a into apache:master Sep 21, 2026
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants