Skip to content

feat(eql, stack-encrypt): chain json_tree tokens by ordinal and probe the walk to the first miss - #1329

Draft
coderdan wants to merge 9 commits into
perf/json-tree-walk-and-projectionfrom
feat/json-tree-ordinal-chain
Draft

coderdan wants to merge 9 commits into
perf/json-tree-walk-and-projectionfrom
feat/json-tree-ordinal-chain

Conversation

@coderdan

@coderdan coderdan commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Summary

EQL (Encrypt Query Language) is the SQL bundle that stores and queries encrypted values in PostgreSQL; EQL v4 is the version for values written by Stack Encrypt, the Rust encryption engine in packages/stack-encrypt. Its json_tree type (unreleased) stores an encrypted JSON document as one entry per JSON node: a small ciphertext c holding that node alone, plus keyed tags that let PostgreSQL search and walk the document without the decryption key. Two of those tags matter here. A selector s is a keyed hash of a node's path. A token is a secret 16-byte value per node that the client hands the server in a query: from it the server can find that node's children, through each child's parent tag p, and so walk the subtree under the node.

This PR shrinks what each entry stores from nine fields to four (s, c, p, op) and changes how the server walks a subtree, so a walk no longer costs the document's widest node for every node it visits. Stacked on #1325. The stored format changes with it, so it has to land before the family ships.

  • Three fields had no reader once the containment projection q was stored (perf(eql, stack-encrypt): store the json_tree projection, walk by parent map, budget the subtree #1325). q is the document's selector pairs grouped by array element, so PostgreSQL's own jsonb @> over it decides containment. Those three fields were the per-entry value selector vs and the two group tags n and g. The engine now reads q back as the projection instead of rebuilding it from the entries.
  • Two more fields existed only so the server could continue the token chain: the wrapped token w and the ordinal o. A child's token is now H("t" ‖ t(parent) ‖ LE32(o)), a hash of its parent's token and its position among its siblings, which the server derives as it walks. The positions under a node run 0, 1, 2, … with no gaps, so the walk probes them in turn and stops at the first miss. A subtree of N nodes costs about 2N probes (a SHA-256 and a lookup each) plus N token hashes, where it cost every position up to the document's widest node at every node visited.
  • The client can no longer derive an object member's token: its position is a rank among sibling keys the client does not know. So a query sends the deepest token it can derive (a top-level field's, then any array positions directly after it) plus one gate per further path step: that step's selector, or #n for an array position. The server descends through the gates and returns the node they reach at depth d = 0, with its descendants beneath it.
  • The leaf ordering term op is opt-in per path (commit ba62e3fc5). op is an order-preserving ciphertext of a number or string leaf, about 80 bytes each, which ORDER BY and range queries at that path use through eql_v4.ord_term. It was derived for every number and string leaf, which made it the largest field of a nested document (24.8 KB of 85 KB) and the one field that reveals something across rows at rest (the order of that leaf's value among every row that holds one), on paths nobody orders. A JSON index now names the paths it orders, with [*] covering every element of an array and [2] one element; nothing is ordered by default; and an ordering query at a path the index does not order is refused before any key loads, instead of silently matching no row.

Changes

  • SQL (packages/eql/src/v3/json_tree/, rendered into EQL v4 only):
    • The entry validator accepts s, c, p and op only, so a document written by the earlier build of this unreleased type is refused.
    • eql_v4.json_tree_subtree_at(doc, token, gates text[], budget DEFAULT 64) replaces the earlier (doc, token, selector, budget) form. It walks the gates and returns the reached node at d = 0 with its descendants. json_tree_subtree(doc, token, budget) is now that function with no gates, and still returns the token's node's descendants from d = 1.
    • json_tree_children and json_tree_child_count take the same optional gates.
    • The budget is unchanged: when the document's root is sealed (its ciphertext holds the whole document) and a walk would return more than budget entries, the walk returns the root entry alone at j = 0, d = 0, and the client opens it and selects the path itself.
    • eql_v4_internal.json_tree_child_token is added; json_tree_unwrap, json_tree_xor16 and json_tree_fanout are removed.
  • stack-encrypt (src/json/):
    • JsonEntry is {s, c, p, op}, and JsonDocument carries its projection, which the wire form requires as q.
    • JsonSubtreeKey holds a token and gates().
    • JsonDocument::children() probes positions in turn.
    • Rows from a gated walk open to the reached node's value.
    • JsonOptions gains ordered: BTreeSet<JsonPath> (so it is Clone, no longer Copy), JsonOptions::ordering([..]) parses paths into it, and JsonPath::covers is the match rule the walk and the query share. The data-plan wire is {"json": {"sealed_root": true, "ordered": ["$.age", "$.orders[*].total"]}}; the header does not record the set, since a stored leaf shows whether it has a term. JsonHeader::options() is removed and json_subtree_key takes &JsonOptions.
    • TermError::JsonPathNotOrdered (code stack_encrypt::json_path_not_ordered) refuses Lt, Lte, Gt, Gte and Between at an unordered path when the query is awaited.
  • stack-encrypt (src/plan/, src/target/): a typed plan had no place for index options, since encrypt_into::<JsonDocument>() reads the type's static EncryptFrom. target::Layout<S> is a target's layout chosen by value, with the index it answers queries through; Json::new(options) implements it, and Plan::value().encrypt_as(layout) builds a ValuePlan<_, Laid<X, T>> that seals under the options and resolves every query through the plan's own index, so a query does not name the options again. Error gains From<JsonPathError>.
  • eql-bindings (src/v4/json_tree.rs): JsonTreeEntry drops vs, o, w, n and g, and the typed document hands q to the engine. JsonTree::new(options) is the json_tree layout under options (a Layout<Value> whose target is JsonTreeDocument); the type's own EncryptFrom is that layout under the defaults.
  • Tests in all three places are updated for the new fields and the gated walk.
  • Design doc (docs/plans/2026-10-17-json-index-stack-encrypt-spike.md): the Phase 3b implementation note carries the measured outcome below, and a Phase 3c note records the opt-in and its measured sizes. The earlier sections, which still describe the wrapped token, group tags and stored ordinal, are kept as the record, with dated notes where they would otherwise mislead.
  • Doc comments that still described the old format are corrected (the header's SQL reader, the projection's source, the walk's cost).
  • Changeset: @cipherstash/eql minor.

Verification

  • cargo nextest run -p stack-encrypt --all-features: 733 passed.

  • cargo test -p eql-bindings --all-features json_tree: passed.

  • mise run test:encryption:postgres against a fresh database with the v4 bundle built from this tree: all suites passed, including the json_tree oracle, which compares containment over the encrypted column with jsonb @> over the plaintexts (postgres_containment_over_the_encrypted_column_is_jsonb_containment, postgres_extracts_walks_and_orders_by_path).

  • After the docs follow-up commit: cargo clippy --all-features --all-targets -D warnings and cargo fmt --check clean for stack-encrypt and eql-bindings, mise run test:schema passed, and mise run docs:validate's coverage and required-tags checks passed. Its documented-sql step needs the Docker database on port 7432 and could not connect locally; CI runs it.

  • For the opt-in commit (ba62e3fc5): cargo nextest run -p stack-encrypt --all-features 737 passed (new: an_ordering_term_sits_only_at_an_ordered_path, a_laid_out_plan_opens_what_it_wrote, the refusal cases in an_accessor_asks_at_a_path and an_accessor_is_refused_before_its_keyset_loads, the ordered wire round trip); the 46 doctests, including the new encrypt_as example; rustdoc with -D warnings; clippy and fmt clean; the trybuild snapshot op_matches_on_integer.stderr regenerated, its only change being the new Laid impl in rustc's help list. mise run test:encryption (eql-bindings and the encryption test crate, no database) passed. The json_tree PostgreSQL oracle (postgres_extracts_walks_and_orders_by_path, postgres_containment_over_the_encrypted_column_is_jsonb_containment) passed against a database carrying this branch's v4 bundle, with the corpus sealed under $.hello and $.number ordered; the SQL did not change in that commit. Docker was not running locally, so the full test:encryption:postgres task was not re-run. cargo-mutants over the commit's diff found five survivors, all on the new Laid plan shape, pinned by 2e405c462 (ValuePlan::specs added for the test).

  • Benchmarks, cipherstash/benches@2d9d5f2 (branch feat/json-tree-benches, report/JSON_TREE_COMPARISON.md, "Third measurement"). They ran at 10,000 and 100,000 rows on one machine (Apple M1 Max, PostgreSQL 17.10), over the same documents before and after, so every scenario returned the same rows. Query-only criterion medians in ms, before → after, on the nested corpus (about 213 entries a document):

    scenario 10,000 rows 100,000 rows
    subtree/orders, default budget 9.48 → 5.71 8.77 → 5.46
    subtree_at/items ($.orders[*].items), default budget 12.95 → 8.21 10.92 → 7.18
    subtree/orders, walked out (budget 0) 22.97 → 10.92 24.57 → 11.35
    subtree_at/items, walked out (budget 0) 23.12 → 10.93 24.83 → 11.27
    subtree/user 3.90 → 2.38 3.99 → 2.45
    path_query/wildcard 2.24 → 1.64 1.80 → 1.41
    field_order/nested 4.81 → 3.81 5.23 → 4.12
    children/orders 1.63 → 1.54 1.64 → 1.49
    child_count/orders 1.62 → 1.53 1.66 → 1.52

    children and child_count moved only 6 to 9%. The report's explanation, not yet tested, is that a per-call fixed cost (detoasting the document and building the parent-tag map) dominates them, and this change does not touch it.

    Containment and field queries are unchanged within noise. The flat corpus read +7% to +13% on byte-identical operands, plans and indexes, and the native jsonb controls over unchanged plaintext drifted +5% to +19% in the same run, so this reads as machine noise rather than the format.

  • Sizes:

    • Nested document: 113,458 → 85,157 characters (−25%); sv 94,324 → 66,023 (−30%).
    • Nested pg_column_size: 81,264 → 62,143 bytes.
    • Nested heap: 810 MB → 621 MB at 10,000 rows, 8,093 MB → 6,201 MB at 100,000.
    • GIN index unchanged (40 MB / 358 MB). GIN build 5 s at 10,000 rows; 58 → 65 s at 100,000, one sample.
    • Flat document: 2,346 → 2,136 characters; pg_column_size 2,452 → 2,250 bytes.
  • Sizes with the ordering term opt-in (cipherstash/benches@cd9643d, "Fourth measurement"), each corpus re-sealed with the one path its scenarios order ($.age flat, $.company.salary nested), correctness 462 of 462 at both tiers:

    • Nested pg_column_size: 62,143 → 40,578 bytes (−35%); heap 621 → 408 MiB at 10,000 rows, 6,201 → 4,069 MiB at 100,000; GIN unchanged.
    • Flat pg_column_size: 2,250 → 1,596 bytes (−29%); heap 27.1 → 17.0 MiB at 10,000 rows. A flat document now sits under the TOAST threshold, so the flat heap moves into the main relation and the flat GIN grows 12% at 100,000 rows from the wider row spread. Query timings were not re-run.
  • Correctness: a new benches task, check:v4:json_tree <N>, ran at both tiers. It decrypted every walk form for 25 random rows and compared each with the plaintext, and it compared the encrypted id sets with native jsonb for 10 needles per predicate. 462 passed, 0 failed, at both tiers.

Related

Review notes

  • Start with the walk comment block in functions.sql and json_tree_subtree_at, then subtree_key_for in stack-encrypt/src/json/derive.rs, which decides what the client derives and what it gates.
  • The one semantic subtlety is in DecryptInto<Value> for JsonSubtree: a d = 0 row means "the node the gates reached", so its value is returned, rather than a container assembled from the rows. A sealed-root fallback also comes back at d = 0, at j = 0, and is told apart by that.
  • Trust is unchanged: only c is authenticated, through its AAD (the associated data the AEAD binds), and q was already what containment ran over. The w mask's binding to c is gone. A swapped ciphertext no longer breaks the walk on the server, but it still fails when the client opens the entry, because c is bound to s and its position j in the document.
  • At rest the format leaks less: there are no stored positions to count children from.
  • The changesets from feat(stack-encrypt, eql): the JSON index and the json_tree EQL v4 domain family #1288 and perf(eql, stack-encrypt): store the json_tree projection, walk by parent map, budget the subtree #1325 still describe the format this PR replaces. They will sit beside this one in the same release's changelog, and this one says what superseded them.

What a query looks like to a user

Users never see selectors, tokens or gates. The SDK derives them from a path and a plaintext needle, and the only encrypted thing in the SQL text is one typed operand.

SQL, as the v4 bundle ships it, for a column profile of type public.eql_v4_json_tree:

-- containment, served by the GIN index; $1 is the encrypted needle
SELECT id FROM users WHERE profile @> $1::eql_v4.query_json_tree;

-- range at a path, served by a btree on the functional index; $sel is the path selector
SELECT id FROM users WHERE (profile -> $sel) >= $2::eql_v4.query_json_tree_leaf;
SELECT id FROM users ORDER BY eql_v4.ord_term(profile -> $sel);

-- one node, opens on the client
SELECT profile -> $sel FROM users WHERE id = 7;

-- a subtree: the token and the gates this PR introduces; rows open on the client into the value
SELECT e FROM users, eql_v4.json_tree_subtree_at(profile, $tok, $gates::text[]) e WHERE id = 7;

Rust, as stack-encrypt has it on this branch:

// The plan names the paths the index orders; nothing is ordered by default.
let options = JsonOptions::default().ordering(["$.age", "$.orders[*].total"])?;
let plan = Plan::value::<Value>().context("users/profile").encrypt_as(Json::new(options)).build()?;
let q = cipher.query(op::Contains, &json!({"orders": [{"id": 2}]})).on(&plan).await?;
let b = cipher.query(op::AtPath(op::Gte), &("$.age", 18)).on(&plan).await?;   // ordered: derives
// cipher.query(op::AtPath(op::Gte), &("$.name", "m")).on(&plan): refused, `$.name` is not ordered
let k = keyset.json_subtree_key(&"$.orders[*].items".parse()?, &options, column).await?;
// k.token() and k.gates() are the arguments of json_tree_subtree_at
let items: Vec<Value> = cipher.decrypt_as(JsonTreeSubtree::new(rows), column).await?;

TypeScript through Drizzle, as the EQL v3 surface reads today; a v4 column keeps it and gains the walk, which has no adapter shape yet:

db.select().from(users).where(await ops.contains(users.profile, { orders: [{ id: 2 }] }))
db.select().from(users).where(await ops.selector(users.profile, "$.age").gte(18))
db.select().from(users).orderBy(ops.asc(ops.selector(users.profile, "$.age")))

https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu

… the walk to the first miss

The stored entry shrinks from nine fields to four, `s c p op`, and the walk
stops depending on the document's width.

Three fields nothing read: once the projection `q` was stored, no SQL function
read an entry's value selector `vs` or its group tags `n` and `g`, and the
engine read them only to rebuild `q` on deserialize. The engine now reads `q`
back as the projection. That changes no trust boundary: only `c` is
authenticated, through its AAD, and `q` was always what containment ran over.

Two fields the walk did not need: a child's token is now
`H("t" ‖ t(parent) ‖ LE32(o))`, which the server derives as it walks, so the
wrapped token `w` is not stored; and the ordinals under a node are dense from
zero, so the walk probes `0, 1, 2, …` until the first miss and neither `o` nor
a fan-out bound is stored. A subtree of N nodes costs about 2N hashes where it
cost the widest node in the document per node visited.

What the client gives up is deriving an object member's token, since an
ordinal is a rank among sibling keys it does not know. `JsonSubtreeKey` now
holds the deepest token the client can derive (a chain head, then array
positions) and one gate per further step, a selector or `#n`, and
`json_tree_subtree_at(doc, token, gates text[], budget)` descends through
them, returning the reached node at `d = 0` and its descendants beneath. The
children functions take the same optional gates.

Verified: stack-encrypt nextest (733), eql-bindings tests, the Postgres oracle
suite against a fresh v4 install built from the tree, clippy and fmt, the EQL
schema gate and documentation coverage. The documented-sql validation step
needs the Docker database and was not run.

Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
@coderabbitai

coderabbitai Bot commented Oct 21, 2026

Copy link
Copy Markdown
Contributor

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

@github-actions

github-actions Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Mutation testing (cargo-mutants, --in-diff, stack-auth + stack-encrypt)

caught missed unviable timeout
18 0 12 0

Every mutant in the changed lines was caught by a test.

…nts, and drop the old format from the docs

The benchmark run against this branch has come in, so the design doc's
Phase 3b section now carries the measured outcome in place of "results to
follow": the walks are 30 to 55% faster at 10,000 and 100,000 rows, the
stored nested document is 25% smaller, containment and field queries are
unchanged within the run's noise, and the benches correctness check
passed all 462 of its comparisons at both tiers. The earlier sections of
the design still describe the wrapped token, the group tags and the
stored ordinal; they are kept as the record, with a dated note wherever
one would now mislead.

Several doc comments still described the old format: the header's SQL
reader computing group tags, the projection being built from group tags,
the test suite unwrapping tokens, `json_tree_subtree_at` matching a single
selector, and the parent map costing a level its fan-out. They now say
what the code does. The walk's cost was stated as about 2N hashes; a
subtree of N nodes takes about 2N probes, each a hash and a lookup, plus a
hash per hit for its token, so the SQL comment, the design doc and the
changeset say that. The changeset also says that the gates argument
replaces the earlier selector one and that the fallback root still comes
back at d = 0.

No SQL or Rust behaviour changes. Checked with clippy and fmt for
stack-encrypt and eql-bindings, the EQL schema gate, and the EQL
documentation coverage and required-tags checks; the documented-sql step
needs the Docker database and could not connect.

Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
@changeset-bot

changeset-bot Bot commented Oct 21, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: a1c504c

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 12 packages
Name Type
@cipherstash/eql Minor
stash Patch
@cipherstash/stack-prisma Patch
@cipherstash/basic-example Patch
@cipherstash/e2e Patch
@cipherstash/prisma-example Patch
@cipherstash/stack-drizzle Patch
@cipherstash/stack-supabase Patch
@cipherstash/stack Patch
@cipherstash/wizard Patch
@cipherstash/bench Patch
@cipherstash/test-kit Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

…s opening and the projection's wire

The in-diff mutants gate on the ordinal-chain commit left four survivors.
Two were timeouts: `JsonDocument::children` probed ordinals up to `u32::MAX`
until the first miss, so a mutant that never misses never ends. A node has
at most as many children as the document has entries, so the loop is now
bounded by that count; the behaviour on real documents is unchanged.

Two were misses with no test behind them. A gated walk's rows carry the
node the gates reached at depth 0, and `DecryptInto<Value>` returns that
node's value rather than assembling a container from the rows; no engine
test exercised that branch, only the Postgres oracle did. And
`JsonProjection` gained its own serde impls for the stored `q` without a
round-trip test. Both are pinned now.

Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
…'s walk

The count bound alone ends a walk over tags that do not form a tree: every
hit adds an entry, and the walk stops once it has returned as many entries
as the document holds. The depth bound added beside it could never fire in
a test, so the mutants gate reported its three operators as unkilled.

Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
…e index declares

The leaf ordering term `op` was derived for every number and string leaf
of a JSON document. After the ordinal-chain change it was the largest
field of a stored document (24,832 of 85,157 characters on the nested
benchmark document, about 80 bytes of hex per leaf), and the one field
that says something across rows at rest: the order of a leaf's value
among every row that holds one, on paths nobody asked to order.

A JSON index now names the paths it orders. `JsonOptions` gains
`ordered`, a set of `JsonPath`s beside `sealed_root`; the walk derives a
leaf's term only when some ordered path covers the node's positional
path (`JsonPath::covers`): a wildcard step covers every array position,
a position covers that element alone, so `$.orders[*].total` and
`$.scores[0]` are both allowed. No path is ordered by default. The set is
part of the index's identity and crosses the data-plan wire as
`{"json": {"sealed_root": true, "ordered": ["$.age", ...]}}`; it is not in
the header, since a stored leaf shows whether it carries a term.

An ordering query at a path the index does not order is refused when the
query is awaited (`TermError::JsonPathNotOrdered`), before any keyset
loads. Without the refusal `eql_v4.ord_term` is NULL for every stored
leaf at the path and the comparison silently matches no row.

The typed plan had no place for options: `encrypt_into::<JsonDocument>()`
reads the type's static `EncryptFrom`, so every typed plan sealed under
the defaults and a query under other options was refused as
`PlanError::IndexOptions`. `target::Layout<S>` is a target's layout
chosen by value, with the index it answers queries through;
`Json::new(options)` implements it, and `Plan::value().encrypt_as(layout)`
builds a `ValuePlan<_, Laid<X, T>>` that seals under the options and
resolves every query through the plan's own index, so a query does not
name the options again. `eql_bindings::v4::JsonTree` is the same layout
for the EQL type, under an identifier.

The SQL is unchanged: `ord_term` was already NULL for an entry without
`op`, and the entry validator already admitted its absence. The
PostgreSQL oracle seals its corpus with `$.hello` and `$.number` ordered,
the paths its ordering checks ask at.

Refs #1231

Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
…ntext check

cargo-mutants over the opt-in commit left five mutants alive, all on the
new `Laid` plan shape: its `specs` (and the `Layout::specs` default it
calls) could return nothing, its `Debug` could print nothing, and the
`Opens::check` of one document and of a batch could accept a call that
names a context the plan already has. `ValuePlan::specs` is added so a
test can read the indexes a plan declares, as `FieldPlan::indexes` does,
and the laid-out plan test asserts the four.

Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
…rm opt-in

The benches re-sealed both corpora with one ordered path each and
measured the stored sizes at 10,000 and 100,000 rows; the Phase 3c note
carries the table in place of the estimate it gave, and the two side
effects the measurement found (the flat document leaving TOAST, the flat
GIN growing with its row spread).

Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
… before its keyset loads

The two `Opens::check` impls of the `Laid` plan survived mutation: the
opening itself resolves the context again, so a call that names a second
context failed either way and the test could not tell the early refusal
from the late one. The counting cipher can: the refusal in `check` runs
before any keyset loads, and the test now asserts that count.

Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant