[Java] Add sbe-jackson tree codec and benchmark harness - #1124
Closed
strangelydim wants to merge 9 commits into
Closed
strangelydim wants to merge 9 commits into
strangelydim wants to merge 9 commits into
Conversation
Co-authored-by: omnigent <noreply@omnigent.ai>
Add the sbe-jackson Gradle module (settings include, version catalog entries for Jackson 2.16.1 baseline and 2.21.6 latest, project block with sbe-tool and jackson-databind api dependencies, generated test codecs, jqwik in the test suite, and a testLatestJackson task that forces the latest 2.21.x release). Add the policy enums, ErrorCode, Limits and SbeJsonException described in DESIGN.md section 4. Co-authored-by: omnigent <noreply@omnigent.ai>
… table PlanCompiler walks the IR tokens of a message the way OtfMessageDecoder does (block fields, then groups, then var-data, recursively) and records one FieldPlan per field, composite member, group or var-data section with scope-relative offsets, enum and bit set tables, group dimension and var-data length layouts. MessagePlan carries the flat array, the root child range and the sorted distinct sinceVersion thresholds that form the effective-layout table. PlanCompilerTest compares the plan walk against a spy TokenListener over OtfMessageDecoder for the Car, extension, composite-elements, group-with-data and nested-group schemas. Co-authored-by: omnigent <noreply@omnigent.ai>
…de and error model SbeJson compiles every message of an Ir into a MessagePlan, builds the JacksonCaches (constant nodes, enum name nodes, serialized field names) and hands out thread-confined SbeJsonDecoder and SbeJsonEncoder instances. SbeJsonDecoder performs the frame prologue (explicit offset + length bounds, header via OtfHeaderDecoder, schema id and template routing, NewerVersions.REJECT) and exposes lastHeader(). PlanMessageCodec.decodeCopy walks the plan into stock Jackson nodes: scope-relative offsets, cursor = entryBase + actingBlockLength, present fields validated inside the acting block, every read bounded by the frame end, Limits on group entries, var-data bytes and depth, numInGroup checked against the IR range, version-absent fields omitted. The MessageCodec SPI, WalkContext (path and counters), HeaderLayout, WireTypes and Utf8 helpers support both directions. Tests: DecodeConformanceTest against JsonPrinter output, VersioningTest (older messages with smaller root and group blocks, newer rejected, header block length larger and smaller than the schema), LimitsTest (hostile numInGroup, var-data budgets, depth pre-check, frame overflow inside buffer capacity, null sentinel numInGroup, unknown template and foreign schema id). Co-authored-by: omnigent <noreply@omnigent.ai>
…tion, properties PlanTreeEncoder walks the plan over a JsonNode tree: header from IR, block fields at fixed offsets (zero-filled block so padding is deterministic), groups and var-data in schema order, per-ObjectNode unknown property detection via recognised count versus size(), strict coercions from DESIGN.md section 7 and an optional sizing pass for encodedLength. EncoderOracleTest proves byte equality with the generated encoders for Car, extension Car, composite elements and group with data; RoundTripTest covers bytes -> decodeCopy -> encode -> bytes and the semantic fixed point under every policy combination plus property order independence; EncodeValidationTest reaches every encode-side ErrorCode; RoundTripPropertyTest uses jqwik for random values within schema ranges, nested groups, UTF-8 var-data with surrogate pairs and lone surrogates. Co-authored-by: omnigent <noreply@omnigent.ai>
…n, build wiring broken)
Stringify generator system properties while retaining incremental inputs and outputs. Repair inherited IR and XML fixtures, constant assertions, composite error paths, and UTF-8 replacement expectations. Count base64 payload bytes before allocation without rejecting padding or whitespace at exact limits. Verified ./gradlew :sbe-jackson:check :sbe-jackson:testLatestJackson :sbe-tool:check --rerun-tasks and incremental codec generation. Co-authored-by: omnigent <noreply@omnigent.ai>
… strict text and retained destination Fix loop 2 on the sbe-jackson module after cross-review of 59cc817. Blocking: - Optional scalars no longer accept their numeric null sentinel: the sentinel exception in signedValue / uint64Value / floatingValue now applies only to elements of optional numeric arrays. cupHolderCount: 255 and optU64 at 2^64-1 are OUT_OF_RANGE for both encode and encodedLength; omission or JSON null still selects the sentinel. - Optional float / double arrays with finite sentinels: elements equal to the sentinel at wire precision (float compares as float) are accepted, so omission -> encode -> decode -> re-encode is byte-identical and encodedLength accepts the decoded tree. New Arrays message in edge-cases-schema.xml carries float[3] (nullValue -1.1) and double[2] (nullValue -1) optional arrays plus a UTF-8 char[8]. Non-blocking: - Var-data limit ordering is now proven: a TextNode subclass whose binaryValue() throws shows base64 payloads are rejected on their counted length, and a test-only charset (SpyCharset, registered through META-INF/services) records that boundedBytes hands the CharsetEncoder budget + 1 bytes of scratch, not the text's upper bound. - PlanTreeEncoder.encode clears the destination in a finally so a retained encoder pins no caller buffer after success or failure. - One text policy for fixed char arrays and var-data in every charset: malformed input (unpaired surrogates, unmappable characters) is rejected with TYPE_MISMATCH and never replaced. Utf8.encodedLength reports an unpaired surrogate as -1 and Utf8.encode refuses it. The property test that asserted U+FFFD replacement now asserts rejection. Documented in DESIGN.md sections 4, 7 and 9 and the SbeJsonEncoder javadoc. - ProgrammaticIrs.edgeCases widens only the BigEnum BEGIN_ENUM..END_ENUM tokens (asserting 10 across messages and 5 in the type map) instead of every int64 token, and rewrites the captured type entry too. - paddedBase64FitsExactPayloadLimits gains a limit + 1 negative case that must fail before binaryValue() runs. Co-authored-by: omnigent <noreply@omnigent.ai>
Add shared Car corpus state, stock-tree decode baselines, tree-fed encode benchmarks, and JSON serialization baselines. Document execution and allocation caveats without tuning production code. Co-authored-by: omnigent <noreply@omnigent.ai>
Author
|
Opened by mistake by an automated worker; this work is intentionally local-only for now. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds the sbe-jackson tree codec and a JMH harness for measuring stock-tree decoding, tree-fed encoding, and JSON serialization. The harness reuses the generated Car codecs, provides baseline and bounded heavy corpora, and checks adapter correctness during trial setup. Production tuning and borrowed-path benchmarks are deferred.
This draft includes prerequisite sbe-jackson commits because base commit
536825811is not on a published base branch. The benchmark-only change is commitf7d733f82(compare536825811..f7d733f82); rebase or retarget after the prerequisite is reviewed.Validation:
./gradlew :sbe-benchmarks:build :sbe-jackson:checkpassed, including compilation with warnings as errors and Checkstyle; 177 collected Jackson test cases, zero failures/errors/skips.java -jar sbe-benchmarks/build/libs/sbe-benchmarks.jar -llists all eight new methods.-f 1 -wi 1 -i 1 -w 1s -r 1s -t 1 -prof gc, JDK 17.0.11, a 256 MiB heap, and fail-on-error enabled. These are smoke checks, not authoritative performance results.See
sbe-jackson/BENCHMARKS.mdfor commands, corpus details, allocation interpretation, and semantic differences between the baselines.