Repository navigation
feat(eql, stack-encrypt): chain json_tree tokens by ordinal and probe the walk to the first miss - #1329
Draft
coderdan wants to merge 9 commits into
Draft
feat(eql, stack-encrypt): chain json_tree tokens by ordinal and probe the walk to the first miss#1329coderdan wants to merge 9 commits into
coderdan wants to merge 9 commits into
Conversation
… the walk to the first miss
The stored entry shrinks from nine fields to four, `s c p op`, and the walk
stops depending on the document's width.
Three fields nothing read: once the projection `q` was stored, no SQL function
read an entry's value selector `vs` or its group tags `n` and `g`, and the
engine read them only to rebuild `q` on deserialize. The engine now reads `q`
back as the projection. That changes no trust boundary: only `c` is
authenticated, through its AAD, and `q` was always what containment ran over.
Two fields the walk did not need: a child's token is now
`H("t" ‖ t(parent) ‖ LE32(o))`, which the server derives as it walks, so the
wrapped token `w` is not stored; and the ordinals under a node are dense from
zero, so the walk probes `0, 1, 2, …` until the first miss and neither `o` nor
a fan-out bound is stored. A subtree of N nodes costs about 2N hashes where it
cost the widest node in the document per node visited.
What the client gives up is deriving an object member's token, since an
ordinal is a rank among sibling keys it does not know. `JsonSubtreeKey` now
holds the deepest token the client can derive (a chain head, then array
positions) and one gate per further step, a selector or `#n`, and
`json_tree_subtree_at(doc, token, gates text[], budget)` descends through
them, returning the reached node at `d = 0` and its descendants beneath. The
children functions take the same optional gates.
Verified: stack-encrypt nextest (733), eql-bindings tests, the Postgres oracle
suite against a fresh v4 install built from the tree, clippy and fmt, the EQL
schema gate and documentation coverage. The documented-sql validation step
needs the Docker database and was not run.
Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
Contributor
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: true
Comment |
Contributor
Mutation testing (cargo-mutants,
|
| caught | missed | unviable | timeout |
|---|---|---|---|
| 18 | 0 | 12 | 0 |
Every mutant in the changed lines was caught by a test.
…nts, and drop the old format from the docs The benchmark run against this branch has come in, so the design doc's Phase 3b section now carries the measured outcome in place of "results to follow": the walks are 30 to 55% faster at 10,000 and 100,000 rows, the stored nested document is 25% smaller, containment and field queries are unchanged within the run's noise, and the benches correctness check passed all 462 of its comparisons at both tiers. The earlier sections of the design still describe the wrapped token, the group tags and the stored ordinal; they are kept as the record, with a dated note wherever one would now mislead. Several doc comments still described the old format: the header's SQL reader computing group tags, the projection being built from group tags, the test suite unwrapping tokens, `json_tree_subtree_at` matching a single selector, and the parent map costing a level its fan-out. They now say what the code does. The walk's cost was stated as about 2N hashes; a subtree of N nodes takes about 2N probes, each a hash and a lookup, plus a hash per hit for its token, so the SQL comment, the design doc and the changeset say that. The changeset also says that the gates argument replaces the earlier selector one and that the fallback root still comes back at d = 0. No SQL or Rust behaviour changes. Checked with clippy and fmt for stack-encrypt and eql-bindings, the EQL schema gate, and the EQL documentation coverage and required-tags checks; the documented-sql step needs the Docker database and could not connect. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
🦋 Changeset detectedLatest commit: a1c504c The changes in this PR will be included in the next version bump. This PR includes changesets to release 12 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
…s opening and the projection's wire The in-diff mutants gate on the ordinal-chain commit left four survivors. Two were timeouts: `JsonDocument::children` probed ordinals up to `u32::MAX` until the first miss, so a mutant that never misses never ends. A node has at most as many children as the document has entries, so the loop is now bounded by that count; the behaviour on real documents is unchanged. Two were misses with no test behind them. A gated walk's rows carry the node the gates reached at depth 0, and `DecryptInto<Value>` returns that node's value rather than assembling a container from the rows; no engine test exercised that branch, only the Postgres oracle did. And `JsonProjection` gained its own serde impls for the stored `q` without a round-trip test. Both are pinned now. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
…'s walk The count bound alone ends a walk over tags that do not form a tree: every hit adds an entry, and the walk stops once it has returned as many entries as the document holds. The depth bound added beside it could never fire in a test, so the mutants gate reported its three operators as unkilled. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
This was referenced Oct 21, 2026
Open
coderdan
added this pull request to stack #1333
October 22, 2026 10:15
…e index declares
The leaf ordering term `op` was derived for every number and string leaf
of a JSON document. After the ordinal-chain change it was the largest
field of a stored document (24,832 of 85,157 characters on the nested
benchmark document, about 80 bytes of hex per leaf), and the one field
that says something across rows at rest: the order of a leaf's value
among every row that holds one, on paths nobody asked to order.
A JSON index now names the paths it orders. `JsonOptions` gains
`ordered`, a set of `JsonPath`s beside `sealed_root`; the walk derives a
leaf's term only when some ordered path covers the node's positional
path (`JsonPath::covers`): a wildcard step covers every array position,
a position covers that element alone, so `$.orders[*].total` and
`$.scores[0]` are both allowed. No path is ordered by default. The set is
part of the index's identity and crosses the data-plan wire as
`{"json": {"sealed_root": true, "ordered": ["$.age", ...]}}`; it is not in
the header, since a stored leaf shows whether it carries a term.
An ordering query at a path the index does not order is refused when the
query is awaited (`TermError::JsonPathNotOrdered`), before any keyset
loads. Without the refusal `eql_v4.ord_term` is NULL for every stored
leaf at the path and the comparison silently matches no row.
The typed plan had no place for options: `encrypt_into::<JsonDocument>()`
reads the type's static `EncryptFrom`, so every typed plan sealed under
the defaults and a query under other options was refused as
`PlanError::IndexOptions`. `target::Layout<S>` is a target's layout
chosen by value, with the index it answers queries through;
`Json::new(options)` implements it, and `Plan::value().encrypt_as(layout)`
builds a `ValuePlan<_, Laid<X, T>>` that seals under the options and
resolves every query through the plan's own index, so a query does not
name the options again. `eql_bindings::v4::JsonTree` is the same layout
for the EQL type, under an identifier.
The SQL is unchanged: `ord_term` was already NULL for an entry without
`op`, and the entry validator already admitted its absence. The
PostgreSQL oracle seals its corpus with `$.hello` and `$.number` ordered,
the paths its ordering checks ask at.
Refs #1231
Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
…ntext check cargo-mutants over the opt-in commit left five mutants alive, all on the new `Laid` plan shape: its `specs` (and the `Layout::specs` default it calls) could return nothing, its `Debug` could print nothing, and the `Opens::check` of one document and of a batch could accept a call that names a context the plan already has. `ValuePlan::specs` is added so a test can read the indexes a plan declares, as `FieldPlan::indexes` does, and the laid-out plan test asserts the four. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
…rm opt-in The benches re-sealed both corpora with one ordered path each and measured the stored sizes at 10,000 and 100,000 rows; the Phase 3c note carries the table in place of the estimate it gave, and the two side effects the measurement found (the flat document leaving TOAST, the flat GIN growing with its row spread). Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
… before its keyset loads The two `Opens::check` impls of the `Laid` plan survived mutation: the opening itself resolves the context again, so a call that names a second context failed either way and the test could not tell the early refusal from the late one. The counting cipher can: the refusal in `check` runs before any keyset loads, and the test now asserts that count. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
…-ordinal-chain Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
EQL (Encrypt Query Language) is the SQL bundle that stores and queries encrypted values in PostgreSQL; EQL v4 is the version for values written by Stack Encrypt, the Rust encryption engine in
packages/stack-encrypt. Itsjson_treetype (unreleased) stores an encrypted JSON document as one entry per JSON node: a small ciphertextcholding that node alone, plus keyed tags that let PostgreSQL search and walk the document without the decryption key. Two of those tags matter here. A selectorsis a keyed hash of a node's path. A token is a secret 16-byte value per node that the client hands the server in a query: from it the server can find that node's children, through each child's parent tagp, and so walk the subtree under the node.This PR shrinks what each entry stores from nine fields to four (
s,c,p,op) and changes how the server walks a subtree, so a walk no longer costs the document's widest node for every node it visits. Stacked on #1325. The stored format changes with it, so it has to land before the family ships.qwas stored (perf(eql, stack-encrypt): store the json_tree projection, walk by parent map, budget the subtree #1325).qis the document's selector pairs grouped by array element, so PostgreSQL's ownjsonb @>over it decides containment. Those three fields were the per-entry value selectorvsand the two group tagsnandg. The engine now readsqback as the projection instead of rebuilding it from the entries.wand the ordinalo. A child's token is nowH("t" ‖ t(parent) ‖ LE32(o)), a hash of its parent's token and its position among its siblings, which the server derives as it walks. The positions under a node run 0, 1, 2, … with no gaps, so the walk probes them in turn and stops at the first miss. A subtree of N nodes costs about 2N probes (a SHA-256 and a lookup each) plus N token hashes, where it cost every position up to the document's widest node at every node visited.#nfor an array position. The server descends through the gates and returns the node they reach at depthd = 0, with its descendants beneath it.opis opt-in per path (commitba62e3fc5).opis an order-preserving ciphertext of a number or string leaf, about 80 bytes each, whichORDER BYand range queries at that path use througheql_v4.ord_term. It was derived for every number and string leaf, which made it the largest field of a nested document (24.8 KB of 85 KB) and the one field that reveals something across rows at rest (the order of that leaf's value among every row that holds one), on paths nobody orders. A JSON index now names the paths it orders, with[*]covering every element of an array and[2]one element; nothing is ordered by default; and an ordering query at a path the index does not order is refused before any key loads, instead of silently matching no row.Changes
packages/eql/src/v3/json_tree/, rendered into EQL v4 only):s,c,pandoponly, so a document written by the earlier build of this unreleased type is refused.eql_v4.json_tree_subtree_at(doc, token, gates text[], budget DEFAULT 64)replaces the earlier(doc, token, selector, budget)form. It walks the gates and returns the reached node atd = 0with its descendants.json_tree_subtree(doc, token, budget)is now that function with no gates, and still returns the token's node's descendants fromd = 1.json_tree_childrenandjson_tree_child_counttake the same optionalgates.budgetentries, the walk returns the root entry alone atj = 0,d = 0, and the client opens it and selects the path itself.eql_v4_internal.json_tree_child_tokenis added;json_tree_unwrap,json_tree_xor16andjson_tree_fanoutare removed.src/json/):JsonEntryis{s, c, p, op}, andJsonDocumentcarries its projection, which the wire form requires asq.JsonSubtreeKeyholds a token andgates().JsonDocument::children()probes positions in turn.JsonOptionsgainsordered: BTreeSet<JsonPath>(so it isClone, no longerCopy),JsonOptions::ordering([..])parses paths into it, andJsonPath::coversis the match rule the walk and the query share. The data-plan wire is{"json": {"sealed_root": true, "ordered": ["$.age", "$.orders[*].total"]}}; the header does not record the set, since a stored leaf shows whether it has a term.JsonHeader::options()is removed andjson_subtree_keytakes&JsonOptions.TermError::JsonPathNotOrdered(codestack_encrypt::json_path_not_ordered) refusesLt,Lte,Gt,GteandBetweenat an unordered path when the query is awaited.src/plan/,src/target/): a typed plan had no place for index options, sinceencrypt_into::<JsonDocument>()reads the type's staticEncryptFrom.target::Layout<S>is a target's layout chosen by value, with the index it answers queries through;Json::new(options)implements it, andPlan::value().encrypt_as(layout)builds aValuePlan<_, Laid<X, T>>that seals under the options and resolves every query through the plan's own index, so a query does not name the options again.ErrorgainsFrom<JsonPathError>.src/v4/json_tree.rs):JsonTreeEntrydropsvs,o,w,nandg, and the typed document handsqto the engine.JsonTree::new(options)is thejson_treelayout under options (aLayout<Value>whose target isJsonTreeDocument); the type's ownEncryptFromis that layout under the defaults.docs/plans/2026-10-17-json-index-stack-encrypt-spike.md): the Phase 3b implementation note carries the measured outcome below, and a Phase 3c note records the opt-in and its measured sizes. The earlier sections, which still describe the wrapped token, group tags and stored ordinal, are kept as the record, with dated notes where they would otherwise mislead.@cipherstash/eqlminor.Verification
cargo nextest run -p stack-encrypt --all-features: 733 passed.cargo test -p eql-bindings --all-features json_tree: passed.mise run test:encryption:postgresagainst a fresh database with the v4 bundle built from this tree: all suites passed, including the json_tree oracle, which compares containment over the encrypted column withjsonb @>over the plaintexts (postgres_containment_over_the_encrypted_column_is_jsonb_containment,postgres_extracts_walks_and_orders_by_path).After the docs follow-up commit:
cargo clippy --all-features --all-targets -D warningsandcargo fmt --checkclean for stack-encrypt and eql-bindings,mise run test:schemapassed, andmise run docs:validate's coverage and required-tags checks passed. Itsdocumented-sqlstep needs the Docker database on port 7432 and could not connect locally; CI runs it.For the opt-in commit (
ba62e3fc5):cargo nextest run -p stack-encrypt --all-features737 passed (new:an_ordering_term_sits_only_at_an_ordered_path,a_laid_out_plan_opens_what_it_wrote, the refusal cases inan_accessor_asks_at_a_pathandan_accessor_is_refused_before_its_keyset_loads, theorderedwire round trip); the 46 doctests, including the newencrypt_asexample; rustdoc with-D warnings; clippy and fmt clean; the trybuild snapshotop_matches_on_integer.stderrregenerated, its only change being the newLaidimpl in rustc's help list.mise run test:encryption(eql-bindings and the encryption test crate, no database) passed. The json_tree PostgreSQL oracle (postgres_extracts_walks_and_orders_by_path,postgres_containment_over_the_encrypted_column_is_jsonb_containment) passed against a database carrying this branch's v4 bundle, with the corpus sealed under$.helloand$.numberordered; the SQL did not change in that commit. Docker was not running locally, so the fulltest:encryption:postgrestask was not re-run. cargo-mutants over the commit's diff found five survivors, all on the newLaidplan shape, pinned by2e405c462(ValuePlan::specsadded for the test).Benchmarks, cipherstash/benches@2d9d5f2 (branch
feat/json-tree-benches,report/JSON_TREE_COMPARISON.md, "Third measurement"). They ran at 10,000 and 100,000 rows on one machine (Apple M1 Max, PostgreSQL 17.10), over the same documents before and after, so every scenario returned the same rows. Query-only criterion medians in ms, before → after, on the nested corpus (about 213 entries a document):subtree/orders, default budgetsubtree_at/items($.orders[*].items), default budgetsubtree/orders, walked out (budget 0)subtree_at/items, walked out (budget 0)subtree/userpath_query/wildcardfield_order/nestedchildren/orderschild_count/orderschildrenandchild_countmoved only 6 to 9%. The report's explanation, not yet tested, is that a per-call fixed cost (detoasting the document and building the parent-tag map) dominates them, and this change does not touch it.Containment and field queries are unchanged within noise. The flat corpus read +7% to +13% on byte-identical operands, plans and indexes, and the native
jsonbcontrols over unchanged plaintext drifted +5% to +19% in the same run, so this reads as machine noise rather than the format.Sizes:
sv94,324 → 66,023 (−30%).pg_column_size: 81,264 → 62,143 bytes.pg_column_size2,452 → 2,250 bytes.Sizes with the ordering term opt-in (cipherstash/benches@cd9643d, "Fourth measurement"), each corpus re-sealed with the one path its scenarios order (
$.ageflat,$.company.salarynested), correctness 462 of 462 at both tiers:pg_column_size: 62,143 → 40,578 bytes (−35%); heap 621 → 408 MiB at 10,000 rows, 6,201 → 4,069 MiB at 100,000; GIN unchanged.pg_column_size: 2,250 → 1,596 bytes (−29%); heap 27.1 → 17.0 MiB at 10,000 rows. A flat document now sits under the TOAST threshold, so the flat heap moves into the main relation and the flat GIN grows 12% at 100,000 rows from the wider row spread. Query timings were not re-run.Correctness: a new benches task,
check:v4:json_tree <N>, ran at both tiers. It decrypted every walk form for 25 random rows and compared each with the plaintext, and it compared the encrypted id sets with nativejsonbfor 10 needles per predicate. 462 passed, 0 failed, at both tiers.Related
->(item 1) is 15 to 19% faster only becausesvis shorter.Review notes
functions.sqlandjson_tree_subtree_at, thensubtree_key_forinstack-encrypt/src/json/derive.rs, which decides what the client derives and what it gates.DecryptInto<Value> for JsonSubtree: ad = 0row means "the node the gates reached", so its value is returned, rather than a container assembled from the rows. A sealed-root fallback also comes back atd = 0, atj = 0, and is told apart by that.cis authenticated, through its AAD (the associated data the AEAD binds), andqwas already what containment ran over. Thewmask's binding tocis gone. A swapped ciphertext no longer breaks the walk on the server, but it still fails when the client opens the entry, becausecis bound tosand its positionjin the document.What a query looks like to a user
Users never see selectors, tokens or gates. The SDK derives them from a path and a plaintext needle, and the only encrypted thing in the SQL text is one typed operand.
SQL, as the v4 bundle ships it, for a column
profileof typepublic.eql_v4_json_tree:Rust, as stack-encrypt has it on this branch:
TypeScript through Drizzle, as the EQL v3 surface reads today; a v4 column keeps it and gains the walk, which has no adapter shape yet:
https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu