feat(eql, stack-encrypt): chain json_tree tokens by ordinal and probe the walk to the first miss

.changeset/eql-json-tree-ordinal-chain.md

@@ -0,0 +1,13 @@
+---
+"@cipherstash/eql": minor
+---
+
+EQL v4 `json_tree` (unreleased): the token chain is derived by ordinal and the walk probes to the first miss.
+
+A stored entry now carries four fields, `s`, `c`, `p` and `op`. The value selector `vs` lives only in the document's projection `q`, which the engine now reads back as the projection instead of rebuilding it from the entries; the group tags `n` and `g` are gone with it, since nothing read them once `q` was stored. The wrapped token `w` and the ordinal `o` are gone too: a child's token is `H("t" ‖ t(parent) ‖ s ‖ LE32(o))`, `s` being the child's own selector, which the server derives as it walks, and the ordinals under a node are dense from zero, so the walk probes them in turn and stops at the first miss. A subtree of `N` nodes costs about `2N` probes (a SHA-256 and a lookup each) and `N` token hashes whatever the document's width, where it cost a probe for every ordinal up to the document's widest node at every node visited. The entry validator accepts the four fields only, so a document written by the earlier build of this unreleased family is refused.
+
+The client can derive a token only for a chain head and the array positions that directly follow it, since an object member's ordinal is a rank among its siblings' keys. `JsonSubtreeKey::gates()` gives the rest of a path as one gate per step, a selector, or `#n:<selector>` for an array position with the array elements' selector, and `eql_v4.json_tree_subtree_at(doc, token, gates text[], budget DEFAULT 64)` walks them, replacing the earlier `json_tree_subtree_at(doc, token, selector, budget)`. `json_tree_children` and `json_tree_child_count` take the same `gates`, optional and empty by default. A gated walk returns the node the gates reach with `d = 0` and its descendants beneath it, so a wildcard reaches several nodes and opens as a list; `json_tree_subtree(doc, token, budget)` is now `json_tree_subtree_at` with no gates and still returns the descendants of the token's node from `d = 1`. Past the budget a walk still returns the sealed root alone, with `j = 0` and `d = 0`. `eql_v4_internal.json_tree_unwrap`, `json_tree_xor16` and `json_tree_fanout` are removed; `eql_v4_internal.json_tree_child_token(token, s, o)` is added. Binding the child's selector into its token, and the element selector into a position gate, is what makes a position name an array element only: an ordinal alone is an element's position or an object member's rank, so in a row whose node at the path is an object, `$.a[0]` would otherwise reach the member that ranks first, where `jsonb` has no node.
+
+**Byte encodings.** Every field the SQL compares, derives or decodes is padded standard base64, as PostgreSQL's `encode(x, 'base64')` writes it: the selectors `s`, the `s` and `vs` pairs of `q`, the parent tag `p`, the ordering term `op`, and the tokens and gates `json_tree_subtree_at`, `json_tree_children` and `->` take, where they were hex; the header `h` was base64 already. The ciphertext `c`, which no SQL function decodes, is RFC 1924 base85. The parent tag `p` is the first eight bytes of its hash, since it is a lookup key within one document and nothing authenticates through it. `eql_v4_internal.json_tree_ope_cllw` is added for `eql_v4.ord_term`, because the `ope_cllw(jsonb)` the scalar types share reads hex. A document written by the earlier build of this unreleased family is refused by the engine. Re-encoding 1,000 stored rows of each benchmark corpus: a nested document measures 22% smaller on disk (`pg_column_size` 40,808 to 31,894 bytes) and a flat one 12% (1,596 to 1,398).
+
+Measured on 10,000 and 100,000 rows of a nested corpus (about 213 entries a document), query-only medians: subtree walks 30 to 55% faster (a 122-entry subtree walked out, 23 ms to 11 ms), `json_tree_children` and `json_tree_child_count` 6 to 9%, a stored nested document 25% smaller (heap 810 MB to 621 MB at 10,000 rows), containment and field queries unchanged within the run's noise.

.changeset/eql-json-tree-walk-and-projection.md

@@ -5,9 +5,11 @@
 **The `json_tree` family's read path no longer pays for the document's size per query.** Three changes to the EQL v4 `json_tree` SQL, each measured (cipherstash/benches, `report/JSON_TREE_COMPARISON.md` at 57abcf9) on a 10,000-row corpus of nested documents (about 220 nodes each) where the shipped surface was one to four orders of magnitude behind native `jsonb`:
 
 - **A stored document carries its own containment projection, `q`.** The client writes the document's `{s, vs}` pairs nested by array element beside `sv`, and `eql_v4.json_tree_query(doc)` now reads that field instead of rebuilding the projection from the group tags on every call. The GIN index recheck, which runs the projection once per candidate row, was the whole cost of `@>` on a nested document: 75 ms for ten rows, 16 s for a two-thousand-row result set, 44 s to build the index on 10,000 rows. With the projection stored the same queries measure in the low milliseconds and the index builds in seconds. `json_tree_query` returns `jsonb` rather than the needle domain, since the domain's CHECK would run per recheck; the document's own CHECK validates `q` when the row is written. It is a plpgsql function costed at 1000 rather than an inlinable SQL one: inlined to `value -> 'q'` the planner costs the filter as one operator, and a column of large documents is TOASTed out of the heap, so a sequential scan over it looked nearly free and was chosen over the GIN index while in fact decompressing every row (1.5 s for a one-row match on 10,000 rows; 0.5 to 0.7 ms through the index). The index expression is `(eql_v4.json_tree_query(col)) jsonb_path_ops`; the `::jsonb` cast earlier guidance wrote is harmless and no longer needed. **Wire change:** `q` is required in `public.eql_v4_json_tree`, and a document written by the earlier build of this unreleased family is refused by the domain CHECK.
-- **The token-chain walk reads the document once.** `json_tree_subtree`, `json_tree_subtree_at`, `json_tree_children` and `json_tree_child_count` computed each parent tag inside a scan of `sv`, once per (node, ordinal, entry); a 244-entry subtree took ten seconds. The walk now builds one map of the document's entries keyed by parent tag and looks each computed tag up in it, so a level costs its fan-out in SHA-256 evaluations and the document is never scanned per node. The same subtree walks out in 23 ms; `children` and `child_count` in 1.6 ms. The walk is also bounded by the document's size: it probes at most as many ordinals as the document has entries and returns at most as many rows, so a stored document whose unauthenticated chain fields claim a huge ordinal, or wrap a child's token back to its parent's, can no longer keep a query running until `statement_timeout`.
-- **A walk has a budget, and past it returns the sealed root.** `json_tree_subtree(doc, token, budget DEFAULT 64)` and `json_tree_subtree_at(doc, token, selector, budget DEFAULT 64)` stop once the subtree exceeds `budget` entries and return the root entry alone, with `d = 0`, when the document's root is sealed: the root's ciphertext is the whole document at about the plaintext's size, where the walk returns some four hundred characters per entry and pays the fan-out in hashes per node, so past a dozen or so entries the whole document is cheaper on both sides. The client opens the root and takes the walked path over it, and gets what the walked rows open to: attach the path with `below(path)` for `json_tree_subtree` rows and `at(path)` for `json_tree_subtree_at` rows (`JsonSubtree` in `stack-encrypt`, `JsonTreeSubtree::new(rows)` in `eql-bindings`). Rows that fell back and carry no path refuse to open. A budget of `0` never falls back; a token that names no node still yields no row; a document whose root is not sealed is always walked out.
+- **The token-chain walk reads the document once.** `json_tree_subtree`, `json_tree_subtree_at`, `json_tree_children` and `json_tree_child_count` computed each parent tag inside a scan of `sv`, once per (node, ordinal, entry); a 244-entry subtree took ten seconds. The walk now builds one map of the document's entries keyed by parent tag and looks each computed tag up in it, so a level costs its fan-out in SHA-256 evaluations and the document is never scanned per node. The same subtree walks out in 23 ms; `children` and `child_count` in 1.6 ms. The walk is also bounded by the document's size, so a stored document whose unauthenticated chain fields claim a huge ordinal, or loop back to an ancestor, can no longer keep a query running until `statement_timeout`.
+- **A walk has a budget, and past it returns the sealed root.** `json_tree_subtree(doc, token, budget DEFAULT 64)` and `json_tree_subtree_at(doc, token, selector, budget DEFAULT 64)` stop once the subtree exceeds `budget` entries and return the root entry alone, with `d = 0`, when the document's root is sealed: the root's ciphertext is the whole document at about the plaintext's size, where the walk returns some four hundred characters per entry and pays the fan-out in hashes per node, so past a dozen or so entries the whole document is cheaper on both sides. The client opens the root and takes the walked path over it, and gets what the walked rows open to. The rows are opened through the key that keyed the walk, which carries the path it was derived from and knows from its gates which walk the rows came from: `key.open(rows)` in `stack-encrypt`, `JsonTreeSubtree::from_key(&key, rows)` in `eql-bindings`. Rows cannot be opened without it, so a document that grows past the budget does not break a caller. A budget of `0` never falls back; a token that names no node still yields no row; a document whose root is not sealed is always walked out.
 
 Also: `eql_v4.json_tree_path_query_first`, the function `->` and the ordering comparisons extract through, is costed at 1000 so a prepared statement's generic plan keeps the functional btree on `eql_v4.ord_term(col -> sel)` instead of filtering a sequential scan (16.9 ms down to 0.11 ms after the fifth execution). `eql_v4_internal.json_tree_group`, the server-side projection builder, is removed.
 
-The `eql-bindings` crate's `JsonTreeDocument` gains the `q` field and `JsonTreeSubtree` becomes a struct with `rows`, an optional `path` and `below`; `stack-encrypt`'s `JsonDocument` writes `q` on the wire and ignores it on the way in, `JsonPath::select` evaluates a path over a value, and `JsonSubtree::below`, `at`, `path` and `is_sealed_root` carry the fallback. EQL v4 only; the v3 bundle is unchanged.
+The `eql-bindings` crate's `JsonTreeDocument` gains the `q` field and `JsonTreeSubtree` becomes a struct of the rows and the key that keyed the walk; `stack-encrypt`'s `JsonDocument` writes `q` on the wire and ignores it on the way in, `JsonPath::select` evaluates a path over a value, and `JsonSubtreeKey::path` and `open`, `JsonSubtree::path` and `is_sealed_root` carry the fallback. EQL v4 only; the v3 bundle is unchanged.
+
+**The leaf ordering term `op` is opt-in per path.** A stored leaf carried the order-preserving term on every number and string, about 80 bytes each and the largest field of a nested document, whether or not anything ordered by it. Now the JSON index's options name the paths that order (`{"json": {"ordered": ["$.age", "$.orders[*].total"]}}` on the data plan; `JsonOptions::ordering([...])` in Rust), a wildcard covering every element of an array, and no path is ordered by default. A position in an ordered path (`$.scores[2]`), or in the path an ordering asks at, is refused: the comparison extracts by selector, which reads a position as the wildcard and reaches the first element, never the one named. An ordering query at a path the index does not order is refused before any key loads rather than silently matching no row. The SQL is unchanged: `eql_v4.ord_term` was already NULL for a leaf without a term. `eql-bindings` gains `v4::JsonTree`, the `json_tree` layout under options, and `stack-encrypt` gains `target::Layout` and `Plan::value().encrypt_as(layout)`, a plan laid out by a value whose queries run under its options.

.github/workflows/fuzz.yml

@@ -78,6 +78,7 @@ jobs:
           - { task: "fuzz:term-decode", slug: term-decode }
           - { task: "fuzz:check-record", slug: check-record }
           - { task: "fuzz:plan-build", slug: plan-build }
+          - { task: "fuzz:json-wire", slug: json-wire }
     steps:
       - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
         with:
@@ -126,6 +127,7 @@ jobs:
           - { task: "fuzz:term-decode",  slug: term-decode,  dir: packages/stack-encrypt, target: term_decode }
           - { task: "fuzz:check-record", slug: check-record, dir: packages/stack-encrypt, target: check_record }
           - { task: "fuzz:plan-build",   slug: plan-build,   dir: packages/stack-encrypt, target: plan_build }
+          - { task: "fuzz:json-wire",    slug: json-wire,    dir: packages/stack-encrypt, target: json_wire_decode }
     steps:
       - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
         with:

docs/fuzzing.md

@@ -28,6 +28,7 @@ Current targets:
 | `fuzz:term-decode`   | `stack-encrypt` | `term_decode`       | the SEM term decoders (`EqualityTerm`, `MatchTerms`, `OreTerm`, `OpeTerm` `from_bytes`) |
 | `fuzz:check-record`  | `stack-encrypt` | `check_record`      | `dynamic::record::check_record` over an `Arbitrary`-derived plan and tree, against a model of the record rules (structure-aware) |
 | `fuzz:plan-build`    | `stack-encrypt` | `plan_build`        | `plan::FieldsBuilder::build` over `Arbitrary`-derived untrusted names (context, fields, identities); a built field's label must render and parse back, and passthrough names are never refused (structure-aware) |
+| `fuzz:json-wire`     | `stack-encrypt` | `json_wire_decode`  | the JSON index's wire decoders (`JsonDocument`, `JsonNode`, `JsonProjection`, `JsonLeafQuery` from JSON text; `JsonHeader::from_bytes`), over base64 tags and base85 ciphertexts; an accepted document or node must round-trip through its wire form |
 
 Each target is a few lines — `libfuzzer-sys` hands a `&str` (or `&[u8]`
 for the byte decoders) to the parser via the `arbitrary` crate: