Repository navigation
DRILL-8555: Physical plan cache for parameterized SQL queries - #3086
letian-jiang wants to merge 16 commits into
Conversation
|
@shfshihuafeng Thank you for this PR. Did you take a look at #3023? I realize they are somewhat different, but it seems the overall goal is similar in trying to cache plans. Are there components of that PR that you could incorporate in yours? |
|
@cgivre Thanks for pointing me to #3023. I have reviewed it, and I agree that both PRs share the same underlying goal: reusing planning work to reduce the overhead of repeated queries. Before considering #3023 production-ready, I think we would need to establish the conditions for safe plan reuse, even for identical SQL. For example:
There is also a practical consideration: production workloads often repeat the same query shape with different literals. Restricting reuse to identical SQL text would substantially limit the benefit for those workloads. My implementation focuses on these correctness requirements through eligibility checks, context and table compatibility checks, and reconstruction of a fresh plan and scan state for each execution. It also preserves typed parameter slots so queries with different literals can reuse compatible cached plans. |
cgivre
left a comment
cgivre
left a comment
There was a problem hiding this comment.
A few issues with parameterized planning and the BIGINT literal change; details inline.
| templateTextPlan, cacheReader, cacheContext)); | ||
| } | ||
| return planned; | ||
| } catch (Exception e) { |
There was a problem hiding this comment.
This also swallows genuine planning failures from handler.getPlan(candidate.sql) (validation errors, planner timeouts), so failing queries are planned twice. The retry reuses sqlNode subtrees that the first validation already rewrote in place.
Fix: catch only cache-specific failures here (or rethrow ValidationException/UserException/planner timeouts).
There was a problem hiding this comment.
Agreed. I moved handler.getPlan(...) outside the cache fallback block, so cache lookup or binding failures can fall back before planning, while planning failures propagate without a cache-induced retry.
| } | ||
| if (lExpr.getDynamicParamIndex() < 0) { | ||
| // Small BIGINT values otherwise parse back as INT during a plan round trip. | ||
| sb.append("cast(").append(lExpr.getLong()).append(" as BIGINT)"); |
There was a problem hiding this comment.
This applies to all plans, not just cached ones: after a JSON round trip, BIGINT literals come back as CastExpression. DrillExprToPaimonTranslator has no visitCastExpression, so in multi-fragment plans the Paimon predicate is dropped after Drill has already removed the Filter, and bigint_col = 5 returns unfiltered rows. Only the Iceberg translator was updated.
Fix: either unwrap cast-of-literal in the pushdown translators (Paimon and any others), or have the parser read the round-tripped value back as a BIGINT literal instead of emitting a cast.
There was a problem hiding this comment.
Thanks for identifying the impact beyond cached plans. I addressed this in the expression parser: the BIGINT cast emitted when serializing an ordinary integer literal is now restored as a LongExpression during deserialization. Explicit casts involving parameter slots remain intact.
c74a235 to
f5e64d5
Compare
|
Hi @cgivre, thanks again for your review! I’ve pushed updates addressing your feedback on parameterized planning and BIGINT literal round trips. When you have a chance, could you take another look? Please let me know if there’s anything else you’d like me to address. Thanks! |
|
Suggestion: make the plan cache size and expiration configurable
Could these be boot options in A few related points:
|
|
Question: plans to support the other storage plugins? Right now only HBase and Iceberg opt in ( I understand why they can't simply be switched on. On a hit, the cache rebinds literals in Drill expressions. But most plugins turn those literals into native state during planning and keep only the result:
Reusing those plans without rebuilding that state would silently return wrong results. So each plugin needs to keep the source expression in its scan JSON and redo pushdown/pruning on bind, as A possible path, roughly in order of effort:
It would be worth measuring planning time against bind-time rebuild cost for Given how large this PR already is, I'm fine with plugin expansion coming in follow-ups. But I'd like to understand the roadmap, and to make sure the plugin API in this PR makes opting in as cheap as possible. Could you add the plan (or JIRAs for each plugin) to the design doc? |
|
Review findings Line numbers refer to the PR head. The first three are correctness issues I'd like to see fixed before merge. Correctness
Performance / design
Observability and operations
Docs
|
|
@cgivre Thank you for the detailed and thoughtful review! I’ve learned a lot from your comments. |
I’ve made the cache size and both expiration policies configurable.
|
cgivre
left a comment
cgivre
left a comment
There was a problem hiding this comment.
Line-by-line follow-up to my earlier review comment, with suggested changes where there's a concrete fix.
I applied all the suggestions together on top of 53306c1, and exec/java-exec (with its dependencies, including logical) builds cleanly, checkstyle included. Several depend on each other, so it's easiest to add them to a batch and commit them together:
- the
PlanCacheconstructor, its three imports, andnew PlanCache(config)inDrillbitContext writeAfterSuccess/putinPlanCacheand the serialization change inDrillSqlWorkerkeyFingerprint()inPlanCacheand the key inDrillSqlWorker
The drill-module.conf defaults are in the comment on the PlanCache fields, because the right spot in that file is outside the diff.
Tests: I couldn't find any plan-cache tests in this PR. PLAN_CACHE_PLUGIN_GUIDE.md describes the checks a plugin should pass (results with caching off vs. a hit with different literals, schema changes, plugin config changes, joins with unsupported plugins, etc.), but none are included for HBase, Iceberg or the engine. Could you add them? They should cover at least:
- the fallback path
- HAVING with GROUP BY
- two sessions with different options sharing a template
- a stale table version
- the new boot options
| // Plan exactly once, outside cache fallback. Validation can mutate the SQL tree, | ||
| // so retrying a failed planning attempt could reuse partially rewritten nodes. | ||
| PhysicalPlan planned = handler.getPlan(planningSql); | ||
| if (prepareCacheInsert != null) { | ||
| prepareCacheInsert.accept(planned); | ||
| } | ||
| return planned; |
There was a problem hiding this comment.
[Correctness] Fall back to ordinary planning when the parameterized SQL fails to plan.
On a miss, the query is planned once from candidate.sql, and the try/catch above only covers lookup and preparation. Any query that only plans with real literals fails, but only when the cache is enabled. Examples: JOIN ... ON TRUE (becomes ON ?), PERCENTILE_CONT(0.5) WITHIN GROUP, and functions that need a literal operand but aren't in configurationOperand.
The comment above is right that the mutated tree can't be retried. This suggestion avoids that by re-entering getQueryPlan from the SQL string with planner.enable_plan_cache set to false at query level, the same pattern the Metastore retry in getPlan uses. The fresh parse, converter and handler start clean. A query that fails for a real reason (bad column, etc.) fails again on the second attempt and reports the error from the original SQL.
| // Plan exactly once, outside cache fallback. Validation can mutate the SQL tree, | |
| // so retrying a failed planning attempt could reuse partially rewritten nodes. | |
| PhysicalPlan planned = handler.getPlan(planningSql); | |
| if (prepareCacheInsert != null) { | |
| prepareCacheInsert.accept(planned); | |
| } | |
| return planned; | |
| if (prepareCacheInsert == null) { | |
| return handler.getPlan(planningSql); | |
| } | |
| PhysicalPlan planned; | |
| try { | |
| planned = handler.getPlan(planningSql); | |
| } catch (Exception e) { | |
| // Some queries only plan with real literal values (e.g. ON TRUE, or functions that | |
| // need a literal operand). Validation mutates the SQL tree, so re-parse and plan the | |
| // original SQL with the cache off for this query, like the Metastore retry above. | |
| logger.debug("Parameterized planning failed; replanning without the plan cache", e); | |
| context.getOptions().setLocalOption(PlannerSettings.ENABLE_PLAN_CACHE_OPTION, false); | |
| return getQueryPlan(context, sql, textPlan); | |
| } | |
| prepareCacheInsert.accept(planned); | |
| return planned; |
| select.getGroup(), visitNullable(select.getHaving()), | ||
| select.getWindowList(), visitNullable(select.getQualify()), |
There was a problem hiding this comment.
[Correctness] Keep HAVING and QUALIFY verbatim when GROUP BY / the select list are.
When structuralProjection is true, GROUP BY and the select list keep their literals, but HAVING and QUALIFY still get slots. So they no longer match the grouped or windowed expressions:
SELECT x || 'a', COUNT(*) FROM hbase.t GROUP BY x || 'a' HAVING x || 'a' = 'ba'HAVING becomes x || ? = ?, and Calcite raises "Expression 'x' is not being grouped". Without a fallback, the query fails only when the cache is on. Using the same condition for HAVING and QUALIFY keeps them consistent with what they have to match.
| select.getGroup(), visitNullable(select.getHaving()), | |
| select.getWindowList(), visitNullable(select.getQualify()), | |
| select.getGroup(), | |
| structuralProjection ? select.getHaving() : visitNullable(select.getHaving()), | |
| select.getWindowList(), | |
| structuralProjection ? select.getQualify() : visitNullable(select.getQualify()), |
| if (!lExpr.isDynamicParam()) { | ||
| // Preserve integer width in JSON; the parser restores this as a BIGINT literal. | ||
| sb.append("cast(").append(lExpr.getLong()).append(" as BIGINT)"); | ||
| } else { | ||
| sb.append(lExpr.getLong()); | ||
| } |
There was a problem hiding this comment.
[Correctness / compatibility] Narrow the BIGINT cast to values that need it.
This now writes cast(N as BIGINT) for every non-parameter BIGINT literal in every plan, even with the cache off. That changes EXPLAIN output, scan digests and filter strings (e.g. AbstractGroupScanWithMetadata.getFilterString) for all users.
Only values in int range are ambiguous on read-back, because ValueExpressions.getNumericExpression tries Integer.parseInt first and then Long.parseLong. Values outside int range already round-trip as LongExpression, so they don't need the cast. This suggestion limits the cast to the ambiguous case. The createCast special case in ExprParser.g4 still handles it.
Is the cast needed at all outside the plan cache? Physical plans were already serialized to JSON for remote fragments before this PR. If only cached plans need it, could it be applied only when serializing for the cache?
| if (!lExpr.isDynamicParam()) { | |
| // Preserve integer width in JSON; the parser restores this as a BIGINT literal. | |
| sb.append("cast(").append(lExpr.getLong()).append(" as BIGINT)"); | |
| } else { | |
| sb.append(lExpr.getLong()); | |
| } | |
| long value = lExpr.getLong(); | |
| if (!lExpr.isDynamicParam() && value >= Integer.MIN_VALUE && value <= Integer.MAX_VALUE) { | |
| // Only int-range values are ambiguous: the parser reads them back as INT. | |
| sb.append("cast(").append(value).append(" as BIGINT)"); | |
| } else { | |
| sb.append(value); | |
| } |
| this.optionsFingerprint = Objects.requireNonNull(optionsFingerprint, "optionsFingerprint"); | ||
| this.tableVersions = Collections.unmodifiableMap(tableVersions); | ||
| this.pluginConfigs = Collections.unmodifiableMap(pluginConfigs); | ||
| } |
There was a problem hiding this comment.
[Performance] Put the options and plugin-config fingerprints in the key (1 of 2; the other suggestion is in DrillSqlWorker).
Today, a mismatch in matchesContext invalidates the shared entry. Two sessions running the same template with different options (e.g. ALTER SESSION SET planner.slice_target = ...) keep evicting each other's entry, so the hit rate is 0 and every query also pays to republish. With the fingerprints in the key, each option set gets its own entry. Only a table-version change, which should invalidate, falls through to invalidate.
| } | |
| } | |
| /** | |
| * Session- and plugin-dependent part of the context. It belongs in the cache key, so | |
| * sessions with different options keep separate entries instead of evicting each other. | |
| * Table versions stay in {@link #matches}, where a change should invalidate the entry. | |
| */ | |
| String keyFingerprint() { | |
| return optionsFingerprint + '\n' + pluginConfigs; | |
| } |
| String key = context.getQueryUserName() + '\n' | ||
| + context.getSession().getDefaultSchemaPath() + '\n' | ||
| + candidate.template; |
There was a problem hiding this comment.
[Performance] Include the session/plugin fingerprint in the key (2 of 2; see ContextSnapshot.keyFingerprint() in PlanCache). This keeps sessions with different option values from evicting each other's entries.
| String key = context.getQueryUserName() + '\n' | |
| + context.getSession().getDefaultSchemaPath() + '\n' | |
| + candidate.template; | |
| String key = context.getQueryUserName() + '\n' | |
| + context.getSession().getDefaultSchemaPath() + '\n' | |
| + snapshot.keyFingerprint() + '\n' | |
| + candidate.template; |
| import org.slf4j.Logger; | ||
| import org.slf4j.LoggerFactory; | ||
|
|
||
| import com.fasterxml.jackson.databind.JsonNode; |
There was a problem hiding this comment.
Import for the plan-cache metrics.
| import com.fasterxml.jackson.databind.JsonNode; | |
| import com.codahale.metrics.Gauge; | |
| import com.fasterxml.jackson.databind.JsonNode; |
| ExecConstants.STORAGE_PLUGIN_REGISTRY_IMPL, StoragePluginRegistry.class, this); | ||
|
|
||
| reader = new PhysicalPlanReader(config, classpathScan, lpPersistence, endpoint, storagePlugins); | ||
| planCache = new PlanCache(); |
There was a problem hiding this comment.
Pass the boot config so the cache size and expiry can be configured (see the suggestion on PlanCache's fields).
| planCache = new PlanCache(); | |
| planCache = new PlanCache(config); |
| return null; | ||
| } | ||
| String identifier = ((HBaseScanSpec) selection).getTableName(); | ||
| try (Admin admin = getConnection().getAdmin()) { |
There was a problem hiding this comment.
[Performance] Remote metadata I/O on every lookup, including hits.
ContextSnapshot.resolve calls this before every lookup. Each call opens an Admin and makes two master RPCs (tableExists + getTableDescriptor). Iceberg is similar: IcebergFormatPlugin.planCacheTable runs HadoopTables.load, and IcebergGroupScan loads the table again during bind. A hit can end up costing more than the planning it skips.
Possible fixes:
- Drop
tableExistsand callgetDescriptordirectly (it throwsTableNotFoundException, which can map tonull). That's one RPC instead of two. - Use
Connection.getTable(name).getDescriptor()rather than opening anAdminper query. - Cache the descriptor hash briefly (a few seconds) per Drillbit, accepting a short window where a schema change is caught at bind time instead.
Could you include hit latency against full planning time for an HBase point lookup in the PR benchmarks?
| // Keep this list aligned with the value-dependent rewrites in DrillOptiq | ||
| // and PreProcessLogicalRel; ordinary data operands still get slots. | ||
| switch (call.getOperator().getName().toUpperCase(Locale.ROOT)) { | ||
| case "DATE_PART": |
There was a problem hiding this comment.
[Robustness] Hard-coded list of functions whose operands must stay literal.
Any function not on this list that needs a literal operand during validation, return-type inference or DrillOptiq conversion silently gets a slot and fails planning. That includes format strings, scale arguments, percentile fractions and new UDFs. The fallback suggested in DrillSqlWorker stops these from failing queries. It would still be better not to rely on the list: could literal-only operands come from the operator's operand type checker (e.g. SqlOperandTypeChecker / OperandTypes.LITERAL), or from an annotation on Drill UDFs, rather than a name switch?
| @@ -0,0 +1,103 @@ | |||
| <!-- | |||
There was a problem hiding this comment.
[Docs] Please move PLAN_CACHE_DESIGN.md and PLAN_CACHE_PLUGIN_GUIDE.md from the repo root to docs/dev/, and link them from docs/dev/DevDocs.md. It would also help to document the new boot options and metrics, and the plan for supporting other plugins (see my earlier comment).
|
I've updated the parameterizer to preserve HAVING and QUALIFY expressions whenever the SELECT expressions are preserved for grouping, ordering, or window definitions. This keeps matching expressions consistent across clauses. Eligible WHERE literals can still be rebound. SELECT x + 1 AS k,
ROW_NUMBER() OVER (ORDER BY x + 1) AS rn
FROM (VALUES (1, 1), (1, 2), (2, 3)) AS t(x, v)
WHERE v > 0
QUALIFY x + 1 = 2
AND ROW_NUMBER() OVER (ORDER BY x + 1) = 1
ORDER BY x + 1;With plan caching disabled, this returned one row: (k = 2, rn = 1). With caching enabled, it returned zero rows without any exception, both on initial planning and on a confirmed cache hit. An exception-triggered fallback would not catch this regression. |
This is a difficult problem, and I agree that a hard-coded list carries a maintenance risk. Given Drill’s current architecture, though, I don’t see a clearly better practical approach: DrillOptiq itself uses function-name switches for value-dependent rewrites, so the parameterizer needs explicit knowledge of which operands must remain literals. Moving this information into function metadata could improve organization, but would still require manually declaring and maintaining those rules. For now, I think keeping this list aligned with the existing rewrites and adding targeted regression tests is a reasonable approach, while acknowledging that it does not establish safety for every SQL pattern. |
|
Following up on the remaining points:
|
|
I’ve rerun both benchmarks on the latest commit, [6ee1c592b](letian-jiang@6ee1c59), using the same machine, datasets, queries and sampling method as the original measurements.
|
| SqlSelect select = (SqlSelect) call; | ||
| // Preserve matching expressions across SELECT, GROUP BY, HAVING and | ||
| // QUALIFY. Parameterizing only some occurrences breaks validation or rewrites. | ||
| boolean structuralProjection = preserveProjection |
There was a problem hiding this comment.
[Correctness] QUALIFY with an inline OVER and no ORDER BY is still parameterized.
The SELECT list, HAVING and QUALIFY are preserved only when there is a GROUP BY, ORDER BY or WINDOW clause. An inline OVER(...) is preserved on its own, but the expressions around it are not. Your zero-rows example from the last round was fixed because it ended in ORDER BY. The same query without it still gets x + ? = ? in both the select list and QUALIFY, and the root cause of the zero rows was never identified.
Could structuralProjection also be true when the SELECT contains any OVER call or a QUALIFY clause? Please also add the no-ORDER-BY variant to TestPlanCachePlanning.
| case FunctionNames.GE: { | ||
| LogicalExpression nameRef = call.args().get(0); | ||
| Expression expression = call.args().get(0).accept(this, null); | ||
| Expression expression = call.args().get(1).accept(this, null); |
There was a problem hiding this comment.
[Behaviour change for all queries] This changes Iceberg pushdown with the cache off too.
Several changes in this file and in IcebergPluginImplementor change pushdown for every Iceberg query:
>=previously readargs().get(0)and was never pushed; now it is.- AND/OR are now pushed.
- Successive filters are combined with AND.
- DATE constants become days, and TIME/TIMESTAMP constants become microseconds. Previously they were milliseconds, which could prune the wrong files.
These all look like genuine fixes, but there are no tests for any of them. When a format config sets ignoreResiduals: true, the newly pushed predicates mean rows that should have been filtered can come back.
Could you add translator unit tests and a few Iceberg query tests with the cache off? Cover >=, AND/OR, and date/timestamp filters. Alternatively, move these into their own PR so they can be reviewed separately. Either is fine with me.
| new DrillParseContext(PrelUtil.getPlannerSettings(filter.getCluster().getPlanner())), | ||
| filter.getInput(), | ||
| condition); | ||
| if (groupScan.getCondition() != null) { |
There was a problem hiding this comment.
Related to the translator comment: combining with the existing condition is the right behaviour, but it also affects queries with the cache off. A test with two stacked filters would pin it down.
|
|
||
| // Plan exactly once, outside cache fallback. Validation can mutate the SQL tree, | ||
| // so retrying a failed planning attempt could reuse partially rewritten nodes. | ||
| PhysicalPlan planned = handler.getPlan(planningSql); |
There was a problem hiding this comment.
[Correctness] No fallback when the parameterized SQL fails to plan.
I understand why you don't want to retry on the same SqlNode, since validation mutates it. The fallback doesn't need to reuse it, though. On failure, set planner.enable_plan_cache=false for this query and call getQueryPlan again from the original SQL string. That parses a fresh tree.
Without this, queries that only plan with real literals fail only when the cache is on, with no workaround short of turning the cache off. Examples are ON TRUE, a function that needs a literal operand but isn't on the hard-coded list, and a new UDF.
| return false; | ||
| } | ||
| // Confirm that standard PhysicalPlan JSON can be read back before publishing. | ||
| reader.readPhysicalPlan(json); |
There was a problem hiding this comment.
[Robustness] A bind that always fails cycles forever.
The read-back here proves the JSON parses, but not that Entry.bind succeeds. If a plan's bind always fails, each hit fails, invalidates and replans, and the entry is published again. An example is a non-expression text field that happens to contain bound_dynamic_param(.
A trial bind with the original literals before publishing would catch this. A small per-key failure counter that stops republishing would also work.
| } | ||
| TableMetadata metadata = ((HasTableOperations) table).operations().current(); | ||
| return metadata == null || metadata.uuid() == null ? null | ||
| : new PlanCacheTable(identifier, metadata.uuid() + ":" + metadata.currentSchemaId()); |
There was a problem hiding this comment.
[Correctness] A change to the provided schema doesn't invalidate the entry.
The version is uuid:currentSchemaId. However, the cached IcebergGroupScan also carries Drill's provided schema from .drill.schema when store.table.use_schema_file=true. After CREATE OR REPLACE SCHEMA ... FOR TABLE, hits keep using the old types and casts. The options fingerprint only covers the flag, not the file contents.
Could you fold a hash of the provided schema into the version (or the schema file's modification time and length), or return null when a schema provider is present?
Separately, tables without a UUID silently never cache. That's safe, but worth a line in the plugin guide.
| String identifier = ((HBaseScanSpec) selection).getTableName(); | ||
| try (Table table = getConnection().getTable(TableName.valueOf(identifier))) { | ||
| String version = Hashing.sha256().hashBytes( | ||
| TableDescriptorBuilder.toByteArray(table.getDescriptor())).toString(); |
There was a problem hiding this comment.
[Performance] Thanks for removing tableExists here. The descriptor is still fetched about three times per query:
- once by the existing
AbstractHBaseDrillTablelookup, which also opens anAdminit never closes; - once here;
- once in
HBaseGroupScan.init()on a hit.
Could this hash the descriptor DrillHBaseTable already holds, or memoize it briefly? Iceberg has the same pattern: HadoopTables.load runs here and again in IcebergGroupScan.initTableScan.
| } | ||
| HBaseFilterBuilder builder = new HBaseFilterBuilder(this, pushedFilter); | ||
| HBaseScanSpec rebuilt = builder.parseTree(); | ||
| if (rebuilt == null || (pushedFilterFullyConverted && !builder.isAllExpressionsConverted())) { |
There was a problem hiding this comment.
Minor: this rebuild runs whenever an HBase scan is read from JSON, not only on a cache bind, and a failure is now a hard IllegalArgumentException. Plans submitted as physical-plan JSON, or replayed from a profile, take this path too.
Could non-cache reads fall back to the serialized hbaseScanSpec? Also, TestHBasePlanCache uses a single region and fully converted predicates only. A test for a region split between warm-up and hit would be valuable, as would a partially converted filter and a value that flips full conversion to partial.
| return dynamicParamIndex >= 0; | ||
| } | ||
|
|
||
| public void setDynamicParamIndex(int dynamicParamIndex) { |
There was a problem hiding this comment.
Minor hardening: this public mutator is inherited by shared singletons (BooleanExpression.TRUE/FALSE, QuotedString.EMPTY_STRING). It's unreachable today because DrillOptiq allocates fresh literals. But if anything ever returns a singleton and sets a slot on it, every query in the JVM would serialize that constant as a slot.
A withDynamicParamIndex(i) that returns a copy would make this impossible. Relatedly, the Decimal9–38 classes now extend LiteralExpression, but their serializers would silently drop a slot. Throwing for slotted types without a wrapper would be safer.
| | 'false' | ||
| ; | ||
|
|
||
| BoundDynamicParam : 'bound_dynamic_param'; |
There was a problem hiding this comment.
Nit: this makes bound_dynamic_param a reserved word for all expression parsing, so an unquoted column or function with that name no longer parses. It's unlikely to matter, and date/as already behave this way. If you want to avoid it, match Identifier with a semantic predicate instead of a lexer token.
DRILL-8555: Physical plan cache for parameterized SQL queries
Description
This PR adds a Drillbit-scoped physical plan cache with HBase and Iceberg support, reusing plans across connections to reduce repeated validation and optimization. It is disabled by default; enable it with
ALTER SESSION SET planner.enable_plan_cache = true.Method
flowchart LR SQL[SQL] --> Template[SQL template] Template --> Cache{Plan cache} Cache -->|Hit| Bind[Bind literals] Cache -->|Miss| Planner[Plan query] Bind --> Plan[Physical plan] Planner --> PlanEligible literals become parameter slots in the SQL template. A hit binds current values to a fresh copy of the cached plan; a miss plans the query normally and populates the cache after successful execution.
Safety guarantees
Benchmark
Measured on one local Drillbit with a Ryzen 7 9700X, 30 GiB RAM and OpenJDK 21. HBase used a 1,000-row mini-cluster for point reads, 50-row range scans and column filters. Iceberg ran all 22 TPC-H queries over eight SF0.01 tables (Q15 used a derived table; Q19 exposed the common equijoin).
Cache hits
The benefit is largest when planning dominates latency: it accounts for roughly 73–76% of the reported HBase baseline latency, and hits reduce end-to-end latency by 50–56%. Analytical queries also benefit: Iceberg planning drops 83.6%, reducing aggregate end-to-end latency by 10.2%.
HBase values are medians of three run medians (15 pairs per workload per run). Iceberg values are sums of per-query medians (three pairs per query), not suite wall time. Percentages are latency reductions relative to cache-off execution.
Cache misses
A separate warmed comparison cleared the plan cache before each cache-on query and drained background writes before both modes. Misses added a median 3–5 ms of paired end-to-end latency for HBase (45 pairs per workload). For Iceberg, the sum of query end-to-end medians changed from 20,624 to 21,806 ms (+5.7%). Miss overhead was modest in these local measurements, while hits provided the largest benefit for short queries.
Documentation
PLAN_CACHE_DESIGN.md: basic principles and supported scope.PLAN_CACHE_PLUGIN_GUIDE.md: plugin APIs and scan reconstruction requirements.