Repository navigation
Conversation
…oducer Stack Encrypt can produce one EQL type because its ciphertext, equality and order encodings disagree about which kinds exist, and the mapping from a kind to a Rust type lives in stack-encrypt (Scalar, dynamic::Value) instead of vitaminc. EQL also cannot tell a Stack Encrypt column from a cipherstash-client one, so a query from one producer against the other's column matches nothing, silently. ADR-0002 records the decisions from the value-encodings RFC review: the new kinds and their frozen tags, one canonical form per kind for terms, term derivation moving into vitaminc-prf and a new vitaminc-ore, and EQL v4 as the v3 SQL emitted under a second name with "v": 4 envelopes, so Postgres keeps the two producers apart. The plan-builder plan defined "EQL v4" as a name for a Stack Encrypt payload in the v3 envelope and eql_v3 domains; it now points at the ADR.
…m stack Starting vitaminc-ore under ADR-0002 surfaced four problems: order encodings, scheme traits and term derivation spread over three repositories; scheme crates that accept raw, hand-encoded bytes; vitaminc-prf reaching orderable-bytes through aead-value's canonical module; and one canonical form shared by equality and order, which rules out leakage-tuned domains such as an exact timestamp for equality with day resolution for order. The plan settles the replacement design: vitaminc-ore owns the order encodings and an OrderScheme trait with a sealed input, and ore-rs and cllw-ore implement it; each term crate applies a semantic baseline and per-domain transforms named in its labels; PRF has no ordering dependency; orderable-bytes is frozen. Exhaustive matching over Value is left open, with both approaches and their shared prerequisite laid out. It includes current and target dependency diagrams and the steps, and ADR-0002 now points to it. Claude-Session: https://claude.ai/code/session_012zHhtTEo968WazExKxA5oT
The ADR said Postgres refuses to compare an eql_v4 query term with an eql_v3 column at plan time. It does not. EQL's domains are over jsonb, and with no operator defined for the exact pair Postgres resolves `=` as the built-in jsonb = jsonb, so the query returns no rows silently, as a mismatched pair within v3 already does. Verified on Postgres 17 with both bundles from #1141 installed in one database. The decision stands: separate names still reject a wrong-version insert, make the function form ambiguous, and give typed clients distinct types. Consequences now says where the protection stops, and Deferred names the cross-version blocker operators that would close the gap (#1160).
The ADR deferred to a plan whose own banner disclaims being a specification, so an implementer had no authoritative text for the canonical forms, the order-scheme layering or the release order. The amended decisions now live in the ADR body, with exhaustive handling of Value recorded as the one open decision. Review fixes: pin the timestamp microsecond rounding to Postgres's (round half to even, checked on 17.5) and the text fold's step order; let unassigned code points pass through instead of refusing writes; forbid lossy order transforms on _ord domains that answer = with the order term; choose a binding's kind from the declared EQL target, with a kind-to-family table, and take a Go date in the value's own location; copy the exact leaf byte layouts from tags.rs; say why float equality is now acceptable and that text order terms change; keep cllw-ore's Bool impl; reintroduce --eql-version deliberately on install and migration; list the eql_v3_-prefix detectors, installer schema lists and single-bundle release tooling v4 needs; and name the owner of the deferred cross-version blockers.
…ic.=) as a loud path ADR-0002's host-language table claimed the seven non-EQL integer kinds and Bytes go through Stack Encrypt's own targets, but no target names them yet. Say so, and say how Go's narrow integers reach a target today. The Deferred section listed typed clients and the function form as the only things that catch a cross-producer query, while Consequences also names OPERATOR(public.=). List all three.
ADR-0002 separates the two producers of EQL payloads by Postgres type: cipherstash-client's terms stay in eql_v3, Stack Encrypt's move to eql_v4, and both come from one SQL source. This is the first step of #1141: the source stops naming its schemas and payload version literally. The hand-written SQL under src/v3, the version template, the search_path pin script and the uninstaller now write {{prefix}} where they wrote eql_v3, and {{eql_version}} where domain CHECKs compared VALUE->>'v' to '3'. eql-codegen emits the same placeholders: its schema, typname-prefix and version constants are placeholder strings, and its unit tests render output as v3 before asserting. The Rust bindings keep the literal eql_v3_ prefix because they describe v3 payloads. A new `eql-codegen render <3|4> <out> <file>...` substitutes them. It refuses an unknown or unterminated placeholder and any literal eql_v3 or eql_v4, since a literal name would render the same for both versions and let the v4 bundle reach into v3. build.sh renders the ordered surface into build/render/eql_v3/ and runs the symbol-order and installer-completeness gates on the rendered files. The docs tooling reads rendered SQL too. The committed generated surface is regenerated in the next commit, so codegen:parity fails at this one alone. Refs #1141
Output of `cargo run -p eql-codegen` after the previous commit; no hand edits. `mise run codegen:parity` reproduces it. Only v3 is rendered so far, and the installer and uninstaller built from this commit are byte-identical to the ones built before the placeholders. Refs #1141
Gate 1 of verify_installer_complete.sh compared the order against the rendered tree, which build.sh writes from that same order, so a dropped file could no longer fail it. It now lists the source tree and maps the paths into the rendered namespace (a truncated order fails again). The Doxygen filter read through a process substitution, whose exit status is lost, so an unreadable input passed docs_v3_grep.sh with empty output; it is a plain pipe under pipefail now. eql-codegen render refuses an absolute or .. path instead of writing over the source. build/ is anchored so tasks/build/ is not ignored, the codegen templates are build cache sources, a moved doc comment is back on its function, and the schema, JSON and pin-script comments name the rendered version (v3 output is byte-identical).
- `eql-codegen render` refuses an empty file list instead of exiting 0
having written nothing.
- The literal-name guard no longer matches a longer version such as
`eql_v30`, and its error says to write the placeholder, comments
included.
- `list-schemas [3|4]` prints rendered names, and the schema parity gate
renders schema.sql with the build's renderer before comparing.
- The Doxygen filter refuses any `{{` left after its substitution.
- verify_installer_complete.sh takes its rendered root from the order
file's directory by default.
- The render tests clean up their scratch directories on failure.
- pin_search_path_v3.sql's header says it is rendered, not appended
verbatim. This is the only change to the v3 installer, and it is a
comment.
…endering it The filter substituted the placeholders with its own sed, a second renderer without eql-codegen's checks. It now reads a src/ or tasks/ file's rendered copy from build/render/eql_v3/, which every caller already builds, and refuses a file with no rendered copy. Other files, such as the filter's own test fixtures, are read as given. The Doxygen XML is byte-identical (297 files). docs_v3_grep.sh counts the hidden ste_vec_contains aliases in the rendered file too, so it no longer depends on the placeholder's spelling, and now depends on build. Claude-Session: https://claude.ai/code/session_01VYKAWbd361xDLmqBdLoUcB
… traits on v4 only Stack Encrypt's output is not byte-for-byte compatible with cipherstash-client's, which writes the eql_v3 domains, so an eql_v3 struct deriving EncryptFrom or DecryptInto claims a compatibility that does not exist. ADR-0002 gives Stack Encrypt its own name, eql_v4. eql-codegen now writes the scalar domains a second time into eql_bindings::v4: eql_v4_* names, a SchemaVersion pinned to 4, a schema id under eql/v4, and an inventory. encryption_gap refuses every v3 domain, so TextEq and TextEqQuery derive their traits only in v4, and a codegen test asserts no v3 struct derives either. The Stack Encrypt target table moves to v4/targets.rs and names eql_v4 domains; the Go package encrypt/eql is regenerated from it. v4 has no SteVec bindings, payload enums, TypeScript export or JSON Schema yet. No published eql-bindings carried the stack-encrypt feature, so nothing released loses a trait. The encryption test crate's PostgreSQL test now installs the v4 bundle; the Go guest and the stack-encrypt fixtures name the v4 domains. Refs #1142
ADR-0002 installs one SQL source under two names, so a column's domain says which engine wrote its terms. The build now renders the placeholder source a second time as eql_v4 and writes release/cipherstash-encrypt-v4.sql and its uninstaller. The v3 artefacts are unchanged, byte for byte. verify_no_v3_in_v4.sh fails the build if the v4 bundle names eql_v3 or checks for payload version 3: a stray v3 reference would install cleanly and then compare v4 data with v3 terms. The symbol-order check stays on v3 only, since v4 is the same files in the same order with every name renamed. v3_v4_coexistence_tests installs both bundles into one database and pins what the separate names enforce: a wrong-version payload fails the domain CHECK, and the cross-version function form is ambiguous. It also pins the gap the corrected ADR records: a plain `=` across versions falls back to jsonb = jsonb and returns no rows, silently, until cross-version blocker operators exist (#1160). Refs #1141
The guest's eql-feature round-trip test still asserted "v": 3 after Stack Encrypt moved to writing EQL v4, failing the WASI/Go CI job. Also update two comments that named the v3 envelope and the eql_v3.query_<name> operand.
…dles verify_no_v3_in_v4.sh piped grep into head under pipefail. When the matches outgrew the pipe buffer, grep died of SIGPIPE, the condition read false, and a wholesale eql_v3 leak passed while a single stray reference failed. It now captures the matches before printing them, and a new self-test (test:v4_isolation, run in CI) feeds it a 20,000-line leak. With v3 and v4 in one database, each lints() reported 3,840 errors about the other version's blocker operators: encrypted_domain_types took every jsonb domain in public, while blockers are recognised only in the lint's own schemas. The public arm is now limited to the lint's own prefix, and the coexistence test asserts both lints are clean with both installed. The coexistence tests also pin OPERATOR(public.=) between versions (ambiguous) and the eq_term text comparison (runs, refuses nothing), and build.sh and mise run clean remove the v4 artefacts too.
…he version boundary The inventory boxes PhantomData<T> handles, and their DomainType impl did not forward schema_id, so every v4 entry reported an /eql/v3/ $id. A type-level schema_id_base_static() now carries each type's base through the handle, and the generated v4 types override it. That surfaced in a new v4 parity test, the counterpart of catalog_parity.rs: v4::all() is the scalar catalog under eql_v4 names in catalog order, each query twin lives in eql_v4, term keys match the catalog, and each generation's TextEq refuses the other's version, both directly and through the inventory. The encryption suite refuses a real v4 payload whose version alone was set back to 3. types:generate snapshots and restores src/v4 as well as src/v3, types:check diffs it, and the pre-commit hook triggers on it. The v4 module doc no longer links a feature-gated module, and the Go READMEs name the eql_v4 column and query operand the guest now writes.
- The gate accepts a payload-version read only in the form `->> 'v' = '4'`, so IN (...), swapped operands, casts and jsonb comparisons are refused without being listed. A grep fault ends the gate with 2 rather than reading as no match. - The self-test asserts the exact exit status for every case: 0 clean, 1 leak, 2 bad input. A crashed gate no longer passes the rejection cases. - test:installer_complete checks the v4 installer as well as v3. - lints() matches its own public domains with starts_with: in LIKE, the underscores inside the prefix are wildcards.
v3 and v4 each had a hand-written SchemaVersion that differed only in the constant and the schema name. They are now one type, SchemaVersion<const V: u16 = 3>: v3::SchemaVersion and v4::SchemaVersion are SchemaVersion<3> and SchemaVersion<4>, and plain SchemaVersion still means v3. The TypeScript binding is written by hand, since ts-rs's derive has no const generics, and bindings/v3/SchemaVersion.ts is byte-identical. Also: - The changeset says what moved to v4 was never released. - The Go README says how to install EQL v4 before using an eql_v4 column. - The targets test exempts only the SteVec rows from the compiled-type check, and asserts they are not producible. - packages/eql/AGENTS.md: types:check covers src/v4, and v4 shares v3's term types. - A private intra-doc link in encryption::targets no longer fails rustdoc -D warnings with the stack-encrypt feature. Claude-Session: https://claude.ai/code/session_01VYKAWbd361xDLmqBdLoUcB
…atalog operator_across_versions_returns_no_rows read the planner's choice from EXPLAIN text, whose deparse can change between Postgres majors. It now stores the comparison as a view and reads the operator it bound: an EQL operator from pg_depend, and the built-in jsonb = jsonb, which is pinned and has no dependency row, from the view's stored query tree. A control checks that a same-version comparison binds the EQL operator, so an empty list is a finding rather than a probe that sees nothing. Claude-Session: https://claude.ai/code/session_01VYKAWbd361xDLmqBdLoUcB
The hook never ran in cipherstash/stack, and would have matched nothing if it had: - `mise run install-hooks` set core.hooksPath to `tasks/githooks`. Git resolves a relative hooksPath against the root of the working tree, which here is the monorepo root, so git looked for `tasks/githooks` there and ran no hook. It now writes the path from the root, through `git rev-parse --show-prefix`. - Git runs a hook from the root of the working tree, so the staged paths carried a `packages/eql/` prefix that the hook's patterns, anchored at `crates/` and `src/`, never matched. The hook now lists only the staged paths under this package, relative to it. - `mise run` ran from the monorepo root, where this package's tasks are not defined. The hook now works from the package root. The prefix is computed from paths rather than with `cd` and `git diff --relative`, because git exports GIT_DIR to a hook, and with GIT_DIR set git takes the current directory as the top of the working tree. Checked with `git hook run pre-commit` from the monorepo root: a staged `src/v3/schema.sql` runs `test:schemas:parity`, and an unrelated staged file runs nothing. Running the hook from the package directory still works. Claude-Session: https://claude.ai/code/session_01VYKAWbd361xDLmqBdLoUcB
…text types A Go partner needs a pre-release of EQL v4 with Stack Encrypt before vitaminc 0.6 settles ordering. These are the v4 types that need no ordering term and whose value kind is already in vitaminc 0.5's value enum: Text, Integer, IntegerEq, Bigint, BigintEq, Boolean, Real and Double, beside TextEq. vitaminc 0.5 implements Encrypt for String but not for i32, i64, bool or the floats, and stack-encrypt deliberately has no combinator that re-encodes a plaintext. So eql_bindings::encryption::plaintext adds Int32, Int64, Bool, Float32 and Float64: each seals vitaminc's tagged leaf, the bytes FfiValue seals for the same value, opens only its own tag, and the integers derive the equality term of the integer inside. A value written through these types and one the Go SDK writes as a plain field open each other, and IntegerEq's term is a plain i32's. The derive reasons now have one source, encryption_gap: ordering terms wait for vitaminc-ore, float equality for vitaminc 0.6's canonical float, smallint/date/timestamp/numeric for their value kinds. Json stays unproducible: a JSON value seals as a tree of leaves and EQL's c holds one, and how a document becomes one leaf is a wire decision not made here. The encryption test crate checks each type through both entry points, against the value enum's leaf and the plain integer's term, and in PostgreSQL on an eql_v4 install: equality queries on IntegerEq and BigintEq return the right rows and every stored value opens. Refs #1130, #1133
Text became producible in this PR, so it is now the live storage-only type the no-query rule describes, and TextOrdOre's refusal names vitaminc-ore rather than CLLW. Both unit tests still asserted the old table and failed under --all-features.
Their equality terms came from vitaminc 0.5's per-width PRF domains (i32-le/v1, i64-le/v1). vitaminc main replaces those with one integer-orderable/v1 domain for every width, so shipping them now would either break the promise that IntegerEq matches a Go int32 equality field or leave every stored hm unmatchable after the upgrade. Both leave ENCRYPTION_DOMAINS with a reason that names the domain, the generated dispatch and the Go table mark them unproducible, and the PRF impls on Int32 and Int64 are removed so nothing derives a term under the old domains. The storage-only types, TextEq and the plaintexts stay. v4_types.rs now covers what the review found missing: a leaf the value enum sealed opens as each typed plaintext, every producible row goes through all three by-name entry points (a swapped type or plaintext in the generator fails), Float64 is checked against the value enum, and the Postgres test reads every column back. The width test that pinned the per-width design is gone. The guest's producibility assertions and the eql-bindings unit tests use the final producible set, and the stashgen README lists each producible type with the Go field type it takes.
… bits, check widths - plaintext.rs: Float32 and Float64 compare by bit pattern, so -0.0 is not 0.0 and a NaN equals itself, the equality a round trip preserves. Unit tests pin each leaf's tag, width and little-endian order through the same reader Decrypt uses, and refuse another tag or width. - The generated plaintext doc line shows without the stack-encrypt feature again; only its link to crate::encryption is feature-gated. - stashgen refuses a field of the right kind and the wrong width (int64 into Integer, float64 into Real) itself, naming both widths, as its README says. - enginetest.Static lists the seven types the engine produces, with their widths. - The eql-bindings README no longer says TextEq is the only producible type. Claude-Session: https://claude.ai/code/session_01VYKAWbd361xDLmqBdLoUcB
…ch terms TextMatch needs no ordering term, and the engine already derives match terms. Under ADR-0002 the terms only have to be ones EQL's SQL can compare, not cipherstash-client's bytes: both the stored value and the query come from Stack Encrypt, and cipherstash-client never writes an eql_v4 column. EQL's bloom filter is a smallint[] of set bit positions, and @@ tests containment. The new BloomFilter adapter writes the engine's positions under its default match configuration (3-grams, lower-cased, k = 3, a 256-bit filter), so every position fits a smallint, and refuses one that does not rather than wrapping it. It names the match term MatchTerm, the spelling that compiles against both the tree and the crates.io stack-encrypt 0.2.0 the published crate builds against. Proven in PostgreSQL on an eql_v4 install: @@ returns every row for a shared substring, one row for a unique one and none for a miss. Text too short to make a 3-gram is refused rather than stored with a filter no query could match. Refs #1131
…age against crates.io The adapter's turbofish named four type parameters for target::matching, and stack-encrypt 0.2.0 on crates.io, which eql-bindings publishes against, takes three, so cargo publish --dry-run --all-features failed. Naming the Encryption type instead lets inference fill the rest under either version. The match configuration was Stack Encrypt's default (k = 3, 256 bits), taken from another crate and recorded nowhere. A 256-bit filter saturates on a few hundred characters, so @@ returned most of a table, and a change to that default would have left stored filters disagreeing with new queries. eql-bindings now pins its own EqlMatch (3-grams, lower-cased, k = 6, 2048 bits); a unit test pins the values and an encryption test pins the filter positions for one query, so a change has to be deliberate. Also: TextMatchQuery's generated docs say match query terms, the Postgres test pins the reverse operand order (query @@ stored) as the known SQL gap it is, the Go guest lists TextMatch as producible, and the READMEs and changeset say that short text cannot be stored yet. Storing an empty filter for it needs a stack-encrypt release first: 0.2.0 refuses token-less text inside the term derivation, before anything here can see it.
…he test engine TextMatch - The eql-bindings README, the stashgen README and the changeset say that only `stored @@ query` matches until #1171 is fixed, and that a TextMatchQuery for text shorter than three characters is refused too. - The PostgreSQL test's reversed-order assertion names #1171, and the filter test says its negative case is a property of one key source and configuration, not of Bloom filters. - enginetest.Static produces TextMatch; the refusal case for an EQL type the engine cannot produce now names IntegerEq; the users golden case stores Name as TextMatch, so generation of its field methods is exercised. - encryption_gap drops its Bloom branch, which no catalog domain reached once text_match became producible. - The generated TextMatchQuery doc keeps the plaintext line without the stack-encrypt feature. Claude-Session: https://claude.ai/code/session_01VYKAWbd361xDLmqBdLoUcB
…eSQL stashgen needed no change: it maps int32, int64, bool, float32, float64 and string to their kinds and accepts any type the engine's se_targets table marks producible, so every type the previous commits made producible already generates typed Encrypt and Query methods. A Profile fixture stores one field in each new type. The hermetic test seals it through generated code over the deterministic EQL guest, checks every value is a v4 envelope with the type's keys, and opens it back. A cross-language fixture, v4_queries.json, holds the IntegerEq, BigintEq and TextMatch query operands the Rust dispatch derives under the shared test index key; the Go fields derive them byte for byte. Checked by hand against PostgreSQL: rows sealed by this Go code, loaded into an eql_v4 install, answer =, = and @@ with exactly the expected ids. The Go module has no PostgreSQL driver, and adding one for a test would change every consumer's dependency graph, so that check is not committed. The encrypt and stashgen READMEs name eql_v4 and list the producible types with the Go field type each takes. Refs #1144
… withdrawn integer equality types Nothing in the repository put a value the generated Go code sealed into an eql_v4 domain or operator; the PostgreSQL check was manual. The Go test now writes v4_rows.json, the Profile rows the generated code seals over the deterministic EQL guest. On every later run it checks that file against today's output, envelopes and match terms byte for byte and ciphertexts by opening them (their AEAD nonce is random). The EQL suite opens those rows with stack-kms's DeterministicSource over the same seed, and its PostgreSQL test, which test:encryption:postgres runs in CI, inserts them into eql_v4_* columns and queries name @@ with the Go query operand from v4_queries.json. With IntegerEq and BigintEq held back, Profile drops Age and stores Balance as Bigint, and v4_queries.json carries TextMatch only. The one stored-versus-query check had used Age, so it is replaced by a TextMatch check that a contained needle's filter bits are a subset of the stored filter and an absent one's are not. Every stored field's i is checked against its table and column. The Go workflow now runs when the fixtures or EQL's lockfile change, and biome leaves the generated rows file alone.
…re stable - TestV4RowsMatchTheCommittedFixture writes the rows it seals to EQL_V4_ROWS_OUT, and the Rust suite opens the file EQL_V4_ROWS names. The Go job runs both, so the ciphertexts the Go SDK seals in that run are opened by native Rust, not only the committed ones. - EQL_UPDATE_FIXTURES rewrites v4_rows.json only when something besides the ciphertexts moved, so a regeneration that changes nothing leaves no diff of fresh nonces. - The Go and Rust TextMatch checks say that their negative cases hold for one seed and configuration, the PostgreSQL expectation folds case as `@@` does, and a needle shorter than three characters is asserted to be refused. - tests-golang.yml runs on any change under packages/eql/tests/encryption. Claude-Session: https://claude.ai/code/session_01VYKAWbd361xDLmqBdLoUcB
The encrypt README and package doc opened with the index= grammar (Ciphertext plus term byte columns, Fields.Email.Equality), while the stashgen README and the EQL v4 alpha setup guide use encrypt_into= and Fields.Email.Query. A reader starting from go doc wrote tags the guide does not describe, for a storage model it does not cover. Both quickstarts now declare encrypt_into fields, build the query with Query, and store each field as one eql_v4 column, with the install step and the query cast. The index= form stays documented and is named as the alternative; the example program still uses it, and the README now says so rather than calling it the eight steps.
…seconds "Host languages" said int8, int16, uint8 and uint16 get their own kinds only when a target names one. A field with no EQL target has no target to pick a kind from; #1144 has Go pick it from the Go type, and an amendment says so, with the mapping for each integer type. The timestamp leaf row allowed chrono's leap-second representation. The vitaminc 0.6 codec refuses it (nanoseconds of 1,000,000,000 or more have no count in any unit; vitaminc-aead's leaf.rs and tags.rs) and requires the instant to lie in chrono's NaiveDateTime range. The row now states that rule. Claude-Session: https://claude.ai/code/session_01BZhkvKLpNtKuzj3HCfMXoL
`PlanError::FieldType` printed "is not a int8" and "is not a object". The article now follows how the type's name is read: "an" before a, e, i or o, and "a" otherwise, including a name starting with u, which is read as "you" (a uint8, a u32). A unit test pins both kinds of name. Claude-Session: https://claude.ai/code/session_01BZhkvKLpNtKuzj3HCfMXoL
…e value-kinds fixture `int(math.MinInt32) - 1` overflows `int` on 32-bit targets, so the 386 job could not build the gensupport tests; the out-of-range case uses `int64`. The value-kinds fixture is generated by its own test, like the record and query fixtures beside it, so Biome no longer reformats it. Claude-Session: https://claude.ai/code/session_01BZhkvKLpNtKuzj3HCfMXoL
… a query's operand as a write's asKind converted every integer to a built-in type before a seal, through reflect, so a defined integer type with its own EncryptValue lost it: a write stored the bare value while a query, which passed the value as it was, asked for the encoded one, and the encoder's own refusals were skipped on writes. A value that writes itself now crosses as it is. Field.query now runs each operand through asKind too, so a query and a write of one field take the same conversion and the same range check, and a query outside the field's range is refused naming the field.
…n the code is generated gensupport converts a built-in integer to a kind of 64 bits or fewer only, and intoKind took any integer target, so a field of a 128-bit EQL type would generate, build, and fail on every Encrypt. No producible EQL type has a 128-bit kind today; intoKind now refuses one, and lowering names the field and both kinds.
…sed when it seals too The breaking note named the open path only. A host that sends a narrow leaf for a field declared int32 is now refused with FieldType at seal, where it was widened; Go code generated before still seals, because gensupport converts to the declared width. ValueFrame::of's errors no longer name a 128-bit integer and a date as outside the model.
feat(stack-encrypt)!: hold a dynamic plaintext as its frame, not a value tree
feat(stack-encrypt,golang): give every Go integer width, dates, timestamps, decimals and 128-bit integers their own kinds
…rough fields unauthenticated A passthrough field comes back beside the decrypted ones in one map, but nothing authenticates it: a writer to the store can change it and decrypt succeeds. The decrypt docs now say so.
… as OpenedType, not Aead A typed decrypt collapsed every failure vitaminc reports into Error::Aead, the error for a wrong key, a wrong context or a tampered ciphertext. That included a leaf that authenticated and only held another type than the target reads: an integer opened as String, say. The Go guest then reported a value sealed as another type as STATUS_AUTH, i.e. tampering. vitaminc's DecryptError already tells the sink's refusal apart from the cipher's, so open_keyed now maps DecryptError::Sink to the new Error::OpenedType, naming only the type the target reads (never the map key a DecodeError can carry). Every other failure stays Aead. Telling a wrong-type reader apart reveals nothing, because the sink runs only after every leaf has authenticated. The guest maps OpenedType to STATUS_ENCODING, as it already maps a typed record field that opens to another kind. Closes #1118.
fix(deps): take chacha20 0.10.2, since 0.10.0 and 0.10.1 were yanked
…m Aead A container's leaves authenticate one by one, but its membership does not: a map entry's AAD binds only its key, and a sequence element's binds no index. So an entry removed or repeated, or elements reordered or dropped, still authenticate, and the target's refusal of the result (MissingField, DuplicateKey, or Expected on a reshaped container) can be tampering. open_keyed now reports OpenedType only when the tree is one leaf and the target refused its kind. Every other failure, every refusal of a container's value included, stays Aead. Found by review on #1256.
…ed-passthrough note A plan's context field is stored as a passthrough, but a change to it is caught: every other field was sealed under it, so decrypt fails with ContextMismatch against the expected context, or the sealed fields do not open. Found by review on #1255.
…part from Aead", and say why This reverts commit 8e904f8. A container's membership and order are deliberately not authenticated: a stored sequence is a set of rows and a stored map a row's columns, and a database returns rows in any order and a query selects some columns, not all. Authenticating position or presence would break exactly those reads. So a target refusing a container's shape is a shape mismatch, OpenedType, not tampering. The rationale now sits on open_keyed, on Error::OpenedType and in the CHANGELOG, and a test opens a Vec<u32> as u32 and expects OpenedType, so a later review that reads the missing membership check as a defect meets the reason first.
…cated docs(stack-encrypt): say that dynamic::record::decrypt returns passthrough fields unauthenticated
…ild may break The record fixtures prove that two authors of one record agree today, and are regenerated when the encoding moves, so a change that breaks rows already stored can be regenerated away. stored_rows_v1.json is the opposite: rows the next build stored under the deterministic key source, written once by an ignored test that refuses when the file exists, and never regenerated. STACK_ENCRYPT_UPDATE_FIXTURES does not touch it. Every case must open to the frame it opened to, and sealing its source again must give the same tree apart from the leaves' nonces: the same shape and keys, terms and passthroughs. The cases cover every value kind with every index it admits, the non-default match options, composites with empty containers, an untyped field, a renamed field, a passthrough, three-segment labels, one and two extension parts, a context field, and a batch. A one-byte change to a stored term or leaf fails the test. Refs #1119.
…ust and Go v4_rows.json proves the Go SDK and Rust agree today and is regenerated with EQL_UPDATE_FIXTURES=1, so a change that breaks rows already stored can be regenerated away. stored_v4_rows_v1.json is a frozen copy of the Profile rows the Go SDK's generated code stored with the next build: one value of every producible EQL v4 type but TextEq. Nothing writes it. Rust opens every frozen value as its v4 type (frozen_rows_the_go_sdk_stored_still_open). Go seals the frozen plaintexts again and requires the same envelopes and terms, and opens every committed ciphertext (TestFrozenV4RowsStillOpen). A one-byte change to a stored term fails the Go test. Refs #1119.
…ver every producible type Profile stored one value of every EQL v4 type the engine produces except TextEq, so neither v4_rows.json, nor v4_queries.json, nor the frozen stored_v4_rows_v1.json held a TextEq payload. Profile gains Email (encrypt_into=TextEq), and every reader follows: the generated code, the Go tests' row and query helpers, the Rust rows reader, and the Postgres test's table, insert and = query. v4_queries.json gains the email query and is otherwise byte-identical. v4_rows.json is regenerated: the email column is new, and the other ciphertexts carry fresh nonces. stored_v4_rows_v1.json is not merged yet, so it is copied again from the new rows before it is frozen. Refs #1119.
…ot-aead fix(stack-encrypt)!: report a ciphertext that opens as the wrong type as OpenedType, not Aead
…fiValue
eql-bindings' API takes a ValueFrame since the plaintext moved to frames,
but its tests still built every input as an FfiValue and read every
opened value back into one. That kept FfiValue looking like a plaintext
type eql-bindings depends on, which it no longer is.
The tests now write a Rust value straight into a frame
(ValueFrame::of(-7i32), ValueFrame::of("café")) and compare an opened
frame with the frame of the value expected, byte for byte, which checks
the kind and a float's bit pattern too. A null is the frame of
None::<i32>. The README's dispatch example, a Cargo.toml comment and an
unreleased CHANGELOG line still named FfiValue as the dispatch's plaintext
and now say ValueFrame.
No test changed what it asserts. eql-bindings and its encryption tests
no longer name FfiValue anywhere.
Refs #1252.
In a_value_of_another_kind_is_refused_not_converted each case is already built as a ValueFrame. Wrapping it in frame() again ran ValueFrame::of over a frame and wrote the same bytes, which read as if the value were still an FfiValue. Pass the frame straight to targets::encrypt. Refs #1252
…c frame The typed-path paragraph named stack_encrypt::dynamic::ValueFrame to say which leaf each type seals. A typed-only build does not build Stack Encrypt's dynamic module, so point at the Go SDK instead, which is the compatibility the sentence is about. The sentence also read badly. The interoperability matrix still called the other direction "the value enum". The tests now build value frames, so say that. Refs #1252
…, and keep dynamic out of stack-encrypt eql-bindings' stack-encrypt feature turned on Stack Encrypt's dynamic layer for every user, although only the by-name dispatch a language binding needs uses it: encryption::targets and the generated v4::targets take a ValueFrame and name an EQL type at run time. A Rust program that encrypts an IntegerEq through encrypt_as compiled the whole dynamic layer for nothing. stack-encrypt now gives the typed EQL v4 types alone, with Stack Encrypt's default features off and no dynamic. A new targets feature turns on stack-encrypt, Stack Encrypt's dynamic feature, and zeroize and thiserror, which only the dispatch uses, and compiles the two targets modules. The Go guest's eql build and the encryption tests enable targets. test:encryption tests eql-bindings under both feature sets, and the wasm32 check builds targets, as the guest does. docs.rs documents targets. The comment on the encryption::targets declaration is a plain comment: a doc comment there would resolve the module's own intra-doc links from the parent scope. Refs #1252.
Before the targets split, stack-encrypt was the only feature, so clippy with -D warnings and the wasm32-wasip1 check covered the typed path. After it, both ran only with targets on, and the typed set was built only by a host cargo test without -D warnings. A lint that fires only without targets, or a wasm32 break in the typed path, would have passed CI. test:encryption now also runs clippy on eql-bindings with --features stack-encrypt alone, and check:encryption:wasi also checks that set for wasm32-wasip1. Both are clean. Refs #1252
The encryption module docs lost their link to `targets` when the module went behind the feature, because a link to a missing module fails the doc build. Link it with cfg_attr when the feature is on, as lib.rs does, and keep the plain text otherwise. docs.rs builds with targets and the crate has no docsrs cfg, so no feature badge shows. Say in the targets module docs that it needs the feature. Refs #1252
test: freeze fixtures of stored rows that no later build may break
test(eql-bindings): build test values as frames of Rust values, not FfiValue
…eature feat(eql-bindings): put the by-name dispatch behind a targets feature, and keep dynamic out of stack-encrypt
…pes compiles on next #1257 added a TextEq query to the v4 query fixture with an FfiValue input, and #1259 removed FfiValue from v4_types.rs. Each passed on its own; merged together on next, the test file does not compile. The input is now frame("alice@example.com"), as every other query there is, and the fixture's bytes are unchanged.
fix(eql): build the TextEq query fixture's input as a frame, so v4_types compiles on next
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This is the integration branch for a release candidate of EQL v4 and of Stack Encrypt's Rust and Go support. EQL is the SQL that lets PostgreSQL store and search encrypted values; EQL v4 is a second copy of it for values written by Stack Encrypt, the Rust encryption engine behind the Go SDK. The open pull requests for both are merged here first, so they can be tested together before any of them reaches
main.The release version is not decided yet.
Why a branch, not main
Some of this work depends on vitaminc and ore.rs code that is not yet on crates.io, so it pins those crates to a git commit.
stack-encryptandeql-bindingsare published crates, andcargo publishdrops those git pins, so nothing on this branch can be released until vitaminc 0.6 is published (cipherstash/vitaminc#389, which waits on cipherstash/vitaminc#463). Once it is, the git pins become crates.io versions and this branch merges tomainas a normal release.What lands here, in order
feat/aead-remove-legacy-traitsto one commit on vitamincmain, and takes vitaminc's breaking changes since then.Rules for this branch
rev, neverbranch. A branch is deleted when its stack merges.packages/eqlandpackages/stack-encrypt/fuzz. Two commits mean two copies of vitaminc's traits, and the code stops compiling.Changes
Verification
nextwas created frommainat 2a70a47. Nothing has been built yet, because nothing has changed.Related