Skip to content

[kafka] Define DDL table mapping for Kafka compatibility - #4277

Open
gyang94 wants to merge 14 commits into
apache:mainfrom
gyang94:codex/kafka-ddl-table-mapping
Open

[kafka] Define DDL table mapping for Kafka compatibility#4277
gyang94 wants to merge 14 commits into
apache:mainfrom
gyang94:codex/kafka-ddl-table-mapping

Conversation

@gyang94

@gyang94 gyang94 commented Sep 10, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Purpose

Closes #4275. Part of #4185.

Define the shared DDL table mapping contract used by Metadata and Produce. A Kafka topic named database.table resolves directly to that Fluss table, so tables with the same name in different databases remain distinct. The physical schema and kafka.* custom properties define how record fields map to table columns.

This is PR03 in the foundation sequence, based on PR02 (#4260, commit 5742b36627fe156a8cf499f0aa7b4b8c4163e9a7); Metadata follows as PR04 (#4261).

Brief change log

  • Map fully qualified topic names in both directions and derive stable topic IDs from Fluss table IDs. Validate both Fluss component names and Kafka's full topic-name length limit.
  • Validate non-partitioned Arrow log tables without primary keys and resolve raw/string key and value formats.
  • Resolve key/value field projections and optional timestamp/headers columns; reject invalid types, missing fields, overlapping projections, and unsupported mapping options. Build projected field names without repeatedly allocating the full schema's field-name list.
  • Cover naming boundaries, schema validation, and native Fluss create-table → metadata-readback behavior, including same-named tables in different databases. Reuse the existing curator-test version in test scope for the cluster fixture.

Relative to PR02: 10 files, +1374 / -0 lines (791 production/build and 583 test lines). The change keeps the mapping contract and its validation together. Record decoding, Arrow encoding, Metadata handlers, Produce, JSON, and authentication follow in later PRs.

Tests

Validated on Java 11 with the source tree committed as 4361fa11b227f6d5573757b21499f692a289cc12:

  • 67 Kafka unit tests passed using the same-checkout Maven reactor.
  • KafkaRequestITCase and KafkaTableMappingITCase passed: 2 integration tests, including cross-database mapping and native metadata readback.
  • Checkstyle, Spotless, RAT, and git diff --check passed.

The full repository test suite and a Flink SQL runtime were not run. The later delivery stage will cover the full DDL → Metadata → Kafka producer → Fluss readback workflow.

API and Format

Uses the existing table schema and custom-property storage. kafka.value.format is required; key mapping is optional. Each mapped raw or string component requires exactly one BYTES or STRING column respectively. Configured key fields require kafka.value.fields-include=EXCEPT_KEY; timestamp and headers use explicitly named physical columns and are excluded from the value projection. Column names are user-defined.

Topic names must use database.table; bare table names do not fall back to a configured database. Reverse mapping returns the same fully qualified name.

No Kafka API is newly advertised by this PR. Table creation continues through existing Fluss DDL/Admin or the Flink catalog; the resolver validates the stored mapping when called by subsequent API layers. Kafka CreateTopics/DeleteTopics and automatic creation are out of scope.

Documentation

The contract is described in this PR, source Javadoc, and executable tests. User-facing delivery examples follow with the later Produce capability.

Stack and review

  1. [kafka] Add request dispatch and transport framework #4259 — PR01: request and transport framework.
  2. [kafka] Serve ApiVersions from registered capabilities #4260 — PR02: ApiVersions.
  3. This PR — PR03: DDL table mapping ([kafka] Define DDL table mapping for Kafka compatibility #4275).
  4. [kafka] Implement Metadata API over Fluss table metadata #4261 — PR04: Metadata ([kafka] Implement Metadata API for pre-created Fluss tables #4266).

This PR remains a draft while its prerequisites are unmerged. It targets Apache main, so the Files changed view includes the foundation prerequisites. Review DDL mapping alone using the incremental comparison against PR02.

Generative AI disclosure: Codex assisted with implementation, testing, and commit organization. Human review is required before merge.

This was referenced Sep 10, 2026
Open
Open
Open
Open
@gyang94
gyang94 force-pushed the codex/kafka-ddl-table-mapping branch from 6b9626d to 4361fa1 Compare September 17, 2026 03:40
@gyang94
gyang94 force-pushed the codex/kafka-ddl-table-mapping branch from 4361fa1 to c508de6 Compare October 10, 2026 13:19
@gyang94
gyang94 marked this pull request as ready for review October 12, 2026 01:39
@gyang94
gyang94 force-pushed the codex/kafka-ddl-table-mapping branch 2 times, most recently from 3d50c53 to d661491 Compare October 14, 2026 03:07
gyang94 and others added 12 commits October 14, 2026 13:45
@gyang94 @codex
[kafka] Serve ApiVersions from registered capabilities
41ecc50 
Route requests through the dispatcher and advertise only implemented APIs.
Return version-aware errors for unsupported APIs and invalid requests.

Validated with mvn -o -pl fluss-kafka verify (23 unit tests and 1 IT).

Co-Authored-By: Codex <noreply@openai.com>
AI-Model: gpt-6
AI-Contributed/Feature: 313/313
AI-Contributed/UT: 137/137
@gyang94 @codex
[kafka] Cover ApiVersions wire negotiation and capability ranges
4317d87 
Exercise future-version decoding, v0 error serialization, correlation IDs,
unknown tagged fields, buffer release, and renegotiation on one connection.
Verify registry-defined ranges, hidden APIs, ordering, and invalid client
software fields across the supported ApiVersions schemas.

Validated with Java 11 reactor clean verify: 37 unit tests and 2 integration
tests passed, with Checkstyle, Spotless, RAT, and git diff checks.

Co-Authored-By: Codex <noreply@openai.com>
AI-Model: gpt-6
AI-Contributed/Feature: 0/0
AI-Contributed/UT: 238/238
@gyang94 @codex
[kafka] Adapt ApiVersions to immutable API registry
229fc94 
Construct the registry once with all handlers and defer ApiVersions registry
access until request handling. Advertised capabilities and dispatch continue
to share the same immutable registry without a separate capability list.
Update coverage to verify lazy access and handler identity across versions.

Validated with Java 11 reactor clean verify: 44 unit and 2 integration cases
passed in both Maven test executions, including inherited error-mapping and
idle regressions. Kafka main and test sources also compile with --release 8.
Spotless, Checkstyle, RAT, and diff checks passed.

Co-Authored-By: Codex <noreply@openai.com>
AI-Model: gpt-6
AI-Contributed/Feature: 28/28
AI-Contributed/UT: 33/33
@gyang94 @codex
[kafka] Adapt version fallback regression to PR02 handler
10bf516 
Use the dispatcher-backed handler constructor in the fallback test added by
PR01. Preserve its unsupported-version, listener, response, and buffer checks.

Validated with Java 11 reactor clean verify: 39 unit and 2 integration cases
passed in both Maven test executions. Kafka main and test sources compile
with --release 8; Spotless, Checkstyle, RAT, and diff checks passed.

Co-Authored-By: Codex <noreply@openai.com>
AI-Model: gpt-6
AI-Contributed/Feature: 0/0
AI-Contributed/UT: 3/3
@gyang94 @codex
[kafka] Clarify uninitialized API registry failures
062047d 
Check the deferred registry before reading advertised capabilities so an
initialization-order failure reports an explicit diagnostic message.

Validated with Java 11 reactor tests: 16 ApiVersions and request-handler
cases passed, along with Spotless, Checkstyle, RAT, and diff checks.

Co-Authored-By: Codex <noreply@openai.com>
AI-Model: gpt-6
AI-Contributed/Feature: 4/4
AI-Contributed/UT: 0/0
@gyang94 @codex
[kafka] Define DDL table mapping for Kafka compatibility
3496440 
Extract topic identity and the raw/string table mapping contract before Metadata. Validate table kinds, field projections and metadata columns independently of request handling and record decoding.

Validation: Java 11, mvn -o -pl fluss-kafka clean verify (47 unit tests and 2 integration tests); Checkstyle, Spotless and RAT passed.

Co-Authored-By: Codex <noreply@openai.com>
AI-Model: gpt-6
AI-Contributed/Feature: 769/769
AI-Contributed/UT: 480/480
@gyang94 @codex
[kafka] Route qualified topics and streamline field projections
3252d38 
Resolve database.table topic names directly to Fluss table paths and return
qualified names for metadata. Validate name boundaries and cover same-named
tables in different databases through native metadata round trips.

Read projected field names directly from DataField to avoid rebuilding the
complete field-name list for every projected column.

Validated on Java 11: 67 Kafka unit tests and 2 integration tests passed,
along with Checkstyle, Spotless, RAT, and git diff --check.

Co-Authored-By: Codex <noreply@openai.com>
AI-Model: gpt-6
AI-Contributed/Feature: 52/52
AI-Contributed/UT: 167/167
@gyang94 @codex
[kafka] Define Kafka timestamp metadata as TIMESTAMP(3)
1275b33 
Co-Authored-By: Codex <noreply@openai.com>
AI-Model: gpt-6
AI-Contributed/Feature: 16/16
AI-Contributed/UT: 33/33
@gyang94 @codex
[test] Wait for table readiness in snapshot lease validation
dea30c2 
Wait for the newly created log table to become ready before acquiring or
releasing snapshot leases. This prevents the coordinator's asynchronous
table-ID registration from masking the expected NonPrimaryKeyTableException.

Validated on Java 11 with the same-worktree reactor:
FlussAdminITCase#testKvSnapshotLeaseForLogTable passed, along with
Checkstyle, Spotless, RAT, and git diff --check.

Co-Authored-By: Codex <noreply@openai.com>
AI-Model: gpt-6
AI-Contributed/Feature: 0/0
AI-Contributed/UT: 1/1
@gyang94 @codex
[kafka] Align DDL table mapping with the FIP contract
6e9dc55 
Register table.kafka options and validate the declared fixed-column mapping
through a shared schema contract. Support binary, string and flat/nested JSON
values with optional key, headers and rescue columns.

Validate CREATE and ALTER before persistence, and fence schema publication
with the table version and coordinator epoch so concurrent schema and
property changes cannot bypass the contract. Resolve bare topic aliases
through kafka.default.database and stop automatically creating that database.

Keep real offset persistence deferred and reject Kafka enablement for tables
that declare offset metadata until that capability is available.

Validation: 146 targeted tests passed on Java 11; Checkstyle, Spotless, RAT
and git diff checks passed.

Co-Authored-By: Codex <noreply@openai.com>
AI-Model: gpt-6
AI-Contributed/Feature: 1131/1131
AI-Contributed/UT: 1451/1451
@gyang94 @codex
[kafka] Validate recursive JSON schemas and default to fluss
48c84f9 
Default bare topic names to the native fluss database while preserving
explicit database overrides, canonical names and no automatic Kafka database
creation.

Validate JSON business types recursively before CREATE, ALTER and enablement.
Reject non-STRING map keys at every nesting level and schemas deeper than the
supported limit. Expose the validation entry point and depth constant for
subsequent decoders. Count business fields equally in flat and nested layouts,
without charging a level for the physical _kafka_value ROW container.

Keep disabled tables outside Kafka validation and preserve user schemas and
explicit properties. Cover complex scalar/container types, depth boundaries,
failed DDL state preservation and successful enablement with real RPC tests.

Validation: Java 11 clean reactor verify passed 161 targeted cases, including
snapshot lease and concurrent ALTER regressions. Checkstyle, Spotless, RAT and
git diff checks passed. The five prior PR03 commits were rebased onto PR02
97d20b1 and checked with range-diff.

Co-Authored-By: Codex <noreply@openai.com>
AI-Model: gpt-6
AI-Contributed/Feature: 92/92
AI-Contributed/UT: 292/292
@gyang94 @codex
[kafka] Enable metadata mapping and fix table change notifications
e7673be 
Allow optional _kafka_metadata with timestamp, timestamp_type and partition,
without a physical offset field. Cover optional fixed-column combinations,
metadata validation and CREATE/ALTER behavior.

Ignore unchanged table registrations when schema ALTER advances the ZK
version. Preserve atomic concurrency fencing and notifications for real
property and bucket changes, with deterministic regression coverage.

Validation: Java 11 reactor verify passed 111 targeted test cases;
Checkstyle, Spotless and git diff --check passed.

Co-Authored-By: Codex <noreply@openai.com>
AI-Model: gpt-6
AI-Contributed/Feature: 57/57
AI-Contributed/UT: 265/265
@gyang94
gyang94 force-pushed the codex/kafka-ddl-table-mapping branch from d661491 to e7673be Compare October 14, 2026 06:11
gyang94 and others added 2 commits October 14, 2026 17:17
@gyang94 @codex
[kafka] Focus DDL mapping changes and streamline tests
b4e5a02 
Create the configured Kafka default database when missing and preserve existing
metadata. Fence schema and property ALTER with schema-parent version checks
without rewriting table registrations, restoring the original table watcher.

Remove redundant configuration, Flink conversion and mapping test coverage while
retaining DDL validation, optional system columns and concurrent ALTER checks.

Validation: targeted clean verify passed 173 tests with Checkstyle and Spotless.

Co-Authored-By: Codex <noreply@openai.com>
AI-Model: gpt-6
AI-Contributed/Feature: 120/120
AI-Contributed/UT: 683/683
@gyang94 @codex
[kafka] Simplify DDL validation and default database setup
7bcda8b 
Reuse native initialization for the fluss database and create a custom
Kafka default database only when it is missing.

Remove PR03-only schema version and Coordinator epoch checks, preserving
PR02 metadata writes and existing protections. Drop the corresponding
concurrency tests while retaining Kafka DDL mapping validation.

Validation: 121 focused tests passed; Maven verify checks passed.

Co-Authored-By: Codex <noreply@openai.com>
AI-Model: gpt-6
AI-Contributed/Feature: 132/132
AI-Contributed/UT: 224/224
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[kafka] Define DDL table mapping for Kafka compatibility

1 participant

@gyang94