Skip to content

feat(realtime): add hash-based offset indexes with data file rolling - #444

Open
lxy-9602 wants to merge 4 commits into
apache:mainfrom
lxy-9602:rt-hash-key-offset
Open

lxy-9602 wants to merge 4 commits into
apache:mainfrom
lxy-9602:rt-hash-key-offset

Conversation

@lxy-9602

Copy link
Copy Markdown
Member

Purpose

Linked issue: #158

  • Add pool-backed HashMap and DenseHashTable utilities adapted from Havenask.
  • Store key hashes instead of original keys, with lazy disk lookup through a shared CacheManager.
  • Generate one .offset sidecar per rolled data file, including lifecycle and abort cleanup.

Tests

  • UT coverage for hashing, containers, cached reads, rolling and cleanup.
  • IT coverage for reads before/after prepare and refresh, plus restart recovery.

API and Format

No public API changes. Replace the previous .offset layout with version 1 containing key-hash-to-offset entries. Previous files are incompatible; hash collisions are not resolved.

Documentation

Generative AI tooling

Generated-by: OpenAI Codex

Comment thread src/paimon/common/utils/hash_map_test.cpp
Comment thread src/paimon/common/utils/dense_hash_table.h Outdated
Comment thread src/paimon/common/utils/hash_map.h Outdated
Comment thread src/paimon/common/utils/dense_hash_table.h
Comment thread src/paimon/common/utils/dense_hash_table.h Outdated
Comment thread src/paimon/common/utils/dense_hash_table_file_reader.h
Comment thread src/paimon/core/io/data_file_key_offset_index_writer.cpp Outdated
Comment thread src/paimon/core/io/data_file_writer_factory.cpp Outdated
Comment thread src/paimon/core/io/data_file_key_offset_index_writer.cpp Outdated

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants