[SteNicholas]

Purpose

Linked issue: close #403

Deletion vectors were applied only when rows were read, so a full-text search on a data-evolution
table with deletion-vectors.enabled still ranked deleted rows. They could take every top-k slot,
and the read then dropped them: a search with limit k returned fewer than k rows, or none, even
when enough live rows matched.

The fix aligns with Java GlobalIndexLiveRowFilter and DataEvolutionFullTextRead#includeRowIds:
the rows live in the snapshot the index is read from are computed once and intersected into the
rows each index shard ranks, while BM25 statistics still come from the whole index.

• GlobalIndexLiveRowFilter::LiveRows: as Java GlobalIndexLiveRowFilter#liveRows does, reads
  the data files of the scanned partitions in the pinned snapshot with ScanMode::ALL, unions
  their row id ranges, then subtracts the rows deleted by their deletion vectors. Every range is
  added before any deletion vector is subtracted, so a deleted row is not added back by another
  file covering the same rows, and only one deletion vector is held at a time. It returns nullopt,
  so every row counts as live, when deletion vectors or row tracking are disabled. A deletion
  vector that cannot be read is reported with its data file, snapshot and table.
• DataFileMeta::NonNullRowIdRange: computes a file's row id range as Java
  DataFileMeta#nonNullRowIdRange does, and rejects a null or negative first row id, a file with
  no rows, or a range that would pass INT64_MAX, instead of overflowing. As in Java,
  DataEvolutionSplitGenerator takes its ranges from it too, which covers the split planning that
  live rows are read through.
• LiveRowGlobalIndexReader: wraps each index shard. A full-text search ranks only the live
  rows of the shard, intersected with the include row ids if given, and is passed on unchanged
  when the whole shard is live. Predicate and vector searches are delegated unchanged.
• LiveRowGlobalIndexReader::CreateUnion: loads live rows on the visiting thread before the
  search fans out to the shards, so a load that reads manifests on the shared executor never waits
  for it from one of its own threads. A search without a positive limit, or whose include row
  ids miss the rows every shard ranks, neither reads live rows nor fails on them.
• GlobalIndexScanImpl: for tables with deletion vectors, wraps the shard readers of every
  index and loads live rows at most once per scan, on its first full-text search that ranks rows,
  pinned to the snapshot the scan reads the index from. A failed load is retried on the next
  search.
• FileStoreScan::WithPartitionFilter: restricts the live-row scan to the partitions the
  global index scan reads.

Vector search still ranks deleted rows, as before; Java filters it too, and the same live-row
helper can be reused for it and for the table-level read in a follow-up.

Tests

UT:

• src/paimon/core/global_index/live_row_global_index_reader_test.cpp:
  TestSearchUnchangedWhenEveryRowIsLive, TestSearchOnlyLiveRowsOfShard (including a shard
  ending at INT64_MAX), TestIncludeRowIdsClippedToShard,
  TestCreateUnionLoadsLiveRowsBeforeSearchingShards (live rows loaded on the visiting thread with
  a one-thread executor), TestCreateUnionRanksLiveRowsOfSearchRanges (overlapping shards),
  TestCacheLiveRows (cached, retried after a failure, and nullopt when every row is live),
  TestLiveRowsErrorOnlyFailsSearchesThatRankRows (a failed load fails a search that ranks rows,
  but not one without a positive limit or whose include row ids miss every searched shard) and
  TestOtherVisitsDoNotReadLiveRows.
• src/paimon/core/io/data_file_meta_test.cpp: TestNonNullRowIdRange (a range ending at
  INT64_MAX, one past it, no rows, a negative and a null first row id).
• src/paimon/core/table/source/split_generator_test.cpp: TestDataEvolutionRowIdRangeAtInt64Max.

IT:

• test/inte/data_evolution_table_test.cpp: DataEvolutionDvTableTest.TestGlobalIndexLiveRows
  (live rows with and without a partition filter, in the snapshot before the deletion, and the
  error when a deletion vector cannot be read).
• test/inte/global_index_test.cpp: TestLuceneSearchSkipsRowsDeletedByDeletionVectors and
  TestFullTextSearchSkipsRowsDeletedByDeletionVectors share CheckSearchSkipsDeletedRows: across
  two shards, live rows keep the scores they had before the deletion, every limit returns the
  top-k live rows, include row ids are intersected with the live rows, a scan created before the
  deletion still ranks every row, and reading the matched rows returns all of them.

API and Format

No change in include/. For tables with deletion vectors and row tracking, full-text searches
through readers created by GlobalIndexScan rank only live rows. Data-evolution scans now reject a
data file whose row id range is invalid instead of overflowing. No storage format change.

Documentation

No new feature. docs/source/user_guide/global_index.rst describes how deleted rows are excluded
from full-text ranking and shows passing one snapshot id to both the index scan and the data scan.

Generative AI tooling

Generated-by: Claude Code 2.1.303 (Claude Opus 5.5)

🤖 Generated with Claude Code

[zjw1111]

+1

[zjw1111]

Thanks for fixing the full-text case. As noted in the description, vector search on data-evolution tables with deletion vectors enabled still has the same top-k issue: deleted rows can occupy result slots and then be dropped during the data read. Do you have plans to address this in a follow-up? If not, we'll schedule support on our side.

[SteNicholas]

Thanks for the review! Yes, I plan to cover vector search in a follow-up and opened #454 to track it. VectorSearch already carries a pre_filter over global row ids, so LiveRowGlobalIndexReader can intersect it with the shard's live rows via ReplacePreFilter, reusing GlobalIndexLiveRowFilter::LiveRows from this PR, in line with Java.

[lxy-9602]

+1