[gripleaf]

Purpose

Related: #430 (background).

Manifest file cache hits currently read Arrow IPC batches and inspect partition fields row by row. Snapshot live-entry caches separately serialize merged entries. This change introduces a shared IndexedManifestEntries representation so both paths can skip unmatched partition/bucket groups before constructing ManifestEntry and DataFileMeta objects.

• Cache complete data manifest files by their existing whole-file keys, allowing reuse across snapshots and query filters. Preserve original ADD/DELETE ordering with row ordinals.
• Reuse the indexed representation in SnapshotLiveManifestEntries, retaining its bucket-level keys, snapshot retention, and merged live-entry semantics.
• Keep historical or unknown bucket layouts eligible for inferred-bucket filtering. Rebuild obsolete or malformed cache envelopes, propagate predicate errors without retry, and retain allocator ownership with accurate cache weight accounting.

Cold cache loads still read the complete manifest. Manifest-list and index-manifest caches continue to use Arrow IPC. This draft does not claim a measured performance improvement; benchmarks remain to be run.

Tests

Added coverage for partition/bucket pruning, interleaved ADD/DELETE ordering, inferred bucket layouts, snapshot reuse, malformed envelopes, complete-file reuse across readers and filters, and predicate error propagation. Updated existing cache lifetime, admission, concurrency, and memory accounting tests.

Validation completed locally:

• cmake --build build --target paimon-core-test -j 16 — passed.
• ./build/debug/paimon-core-test — 2,364 tests passed before the final row-range overflow-expression correction.
• ./build/debug/paimon-core-test --gtest_filter='*Manifest*:*FileStoreScan*:*InferredBucket*' — 131 tests passed after that final correction and rebuild.
• pre-commit run --files for all 17 changed files — passed.
• git diff --check — passed.

The standalone integration suite and performance benchmarks have not been run.

API and Format

No changes to public headers under include/paimon, configuration options, persisted manifest formats, or protocols. Existing cache kinds and keys are retained. Private data-manifest and snapshot-live cache encodings change; incompatible cached values are rebuilt from the source. File and snapshot caches remain separate logical cache layers using a shared representation.

Documentation

Updated docs/source/user_guide/manifest_cache.rst and docs/source/user_guide/manifest_entry_cache.rst to describe the shared indexed representation, filtering, cache fallback, and memory accounting.

Generative AI tooling

Generated-by: OpenAI Codex CLI 0.163.0 (GPT-6)