feat(go): bind a jsonpath predicate on a JsonSearch field as one parameter

languages/golang/CHANGELOG.md

@@ -12,6 +12,43 @@ listed under **Breaking**. Releases are tagged `languages/golang/vX.Y.Z`.
 
 ### Added
 
+- **Bind a whole jsonpath filter on a `JsonSearch` field as one parameter
+  (#1443).** Every `JsonSearch` field gets
+  `Fields.<Field>.Match(ctx, cipher, predicate, vars, opts)`, which takes
+  what plaintext `jsonb_path_match(col, $1, $2)` takes: a jsonpath predicate
+  and its variables (any value `encoding/json` writes as an object, or a
+  `json.RawMessage` of one; an empty map when there are none). A nil `vars`
+  (nil, or a nil map, slice or pointer) answers as pgx and `database/sql`
+  bind it over plaintext, as SQL `NULL`: the operand is empty, binds as
+  `NULL`, and matches no row. `vars` that write the JSON `null` are refused
+  as not an object, as plaintext refuses them. It returns an
+  `eql.JSONSearchPredicateQuery` for
+  `col @@ $1::eql_v4.query_json_predicate` or
+  `eql_v4.json_tree_match(col, $1::eql_v4.query_json_predicate)`. The engine
+  parses the predicate and compiles it into a tree of the operands `EqAt`,
+  `Selector`, `At.<Path>` and `Elements.<Array>.Match` derive; the Go side
+  reads none of it. The grammar is paths of `.key`, `."quoted key"` and
+  `[*]`; `==`, `!=`, `<`, `<=`, `>`, `>=` against a variable or a JSON
+  literal; `exists(path)` and `exists(path ? (@.a == $x && @.b > $y))`; and
+  `&&`, `||`, `!(…)`. A construct it cannot answer is refused as an
+  `*encrypt.Diagnostic` wrapping `encrypt.ErrEncoding`, never compiled to a
+  tree that drops a leaf; an ordering at a path the tag does not order is
+  `encrypt.ErrTerm`. Each row answers as `jsonb_path_match` does over its
+  plaintext, save five documented cases (lax array unwrapping and wrapping,
+  a string compared with a number, folded ordering terms, and an ordering
+  of a `null` that is NULL rather than false outside an odd number of `!`,
+  which returns the same rows). The tree is
+  evaluated inside a function, so no index engages: `EqAt`, `Contains` and
+  `At.<Path>` stay the indexed form. The embedded EQL guest names the
+  `eql_v4.query_json_predicate` domain, so `Checker.Operators` lists `match`
+  (input `predicate`) on every `JsonSearch` field, and the guest reports a
+  refused predicate as a malformed input rather than an internal failure.
+
+- **`Diagnostic.Construct` and `Diagnostic.At` (#1443).** A refused `Match`
+  predicate names the construct in the grammar's words (`` `like_regex` ``,
+  `a position (`[2]`)`) and the byte offset in the predicate where it
+  starts. Neither carries text from the predicate.
+
 - **Test several members of one array element of a `JsonSearch` field
   (#1442).** A field whose `ordered=` paths name members beneath an array's
   wildcard (`$.contacts[*].relationship;$.contacts[*].calls`) gets
@@ -158,6 +195,19 @@ listed under **Breaking**. Releases are tagged `languages/golang/vX.Y.Z`.
 
 ### Breaking
 
+- **Every `encrypt/eql` type prints and logs redacted (#1443).** `String`,
+  `Format` (every verb, `%#v` included) and `LogValue` write the type's
+  name and `[redacted]` (`eql.TextEq{[redacted]}`, `eql.TextEq{}` when
+  empty), never the bytes: a stored value holds ciphertext and terms, and a
+  query value terms. Before, each type was a bare `[]byte` to `fmt` and
+  `slog`: `%s` and `%q` gave its JSON, and `fmt.Sprint(v)`, `%v` and `%x`
+  its bytes as numbers or hex. They now give the redacted text, with no
+  error. A caller that printed or logged the JSON with `%s` calls the new
+  `Reveal()` at that call (`slog.Debug("stored", "email", v.Reveal())`),
+  which returns the JSON as a string; code that reads the JSON converts the
+  value, `string(v)` or `json.RawMessage(v)`, or calls `v.MarshalJSON()`.
+  `Value` is unchanged. Generated by eql-codegen for every type.
+
 - **`eql.SteVecDocument` and `eql.SteVecQuery` are gone.** They named
   EQL v4's SteVec domains, which the engine never produced and the v4
   bundle no longer carries: in v4, `public.eql_v4_json_search` is the

languages/golang/cmd/stashgen/README.md

@@ -179,6 +179,7 @@ The field's `Fields` entry has a method per operation:
 | `Subtree(ctx, c, path, opts)` | `encrypt.JSONRead`: `Tokens()`, `Open` and `OpenOne` | `eql_v4.json_tree_subtree(col, $1::text[], 64)` |
 | `Children(ctx, c, path)`, `ChildCount(ctx, c, path)` | `encrypt.JSONToken` | `eql_v4.json_tree_children(col, $1::text)`, `eql_v4.json_tree_child_count(col, $1::text)` |
 | `Elements.Visits.Match(ctx, c, opts, predicates...)` | `eql.JSONSearchElementsQuery` | `eql_v4.json_tree_element_match(col, $1::eql_v4.query_json_elements)` |
+| `Match(ctx, c, predicate, vars, opts)` | `eql.JSONSearchPredicateQuery` | `col @@ $1::eql_v4.query_json_predicate`, or `eql_v4.json_tree_match(col, $1::eql_v4.query_json_predicate)` |
 
 `At` has one accessor per ordered path, named for its keys (`$.visits[*].at` is `At.VisitsAt`, `$.a.1` is `At.A1`); two paths with one name, or one path given twice, stop `stashgen`.
 An ordering compares the value `->` extracts at its path, and EQL's `<` compares ordering terms alone, so take the selector from the same accessor (`At.Age.Selector`) rather than spelling the path again.
@@ -238,6 +239,66 @@ Each comparison is of the member's ordering term, so `Eq` is an equality of fold
 `EqAt` at the member's full path is exact, of any element.
 `Match` reads each array to `opts.WildcardBound` elements, 64 when it is 0, and a row whose array is longer fails the query with SQLSTATE 22023 rather than answer over part of it: set the bound past the longest array.
 The function runs on each row a scan gives it, so narrow the rows first with an indexed predicate, such as `@>` on the same values.
+
+`Match` on the field itself takes a whole filter as one parameter: a jsonpath predicate and its variables, the two things plaintext `jsonb_path_match(col, $1, $2)` takes.
+It is generated on every `JsonSearch` field, ordered paths or none: the engine reads the predicate when `Match` is called, and refuses what the field cannot answer then.
+
+```go
+q, err := entities.Fields.CustomFieldValues.Match(ctx, cipher,
+	`$."extra"."country" == $val && $."weight" < $max`,
+	map[string]any{"val": "nz", "max": 9.5}, encrypt.JSONReadOptions{})
+```
+
+```sql
+SELECT id FROM entity WHERE custom_field_values @@ $1::eql_v4.query_json_predicate
+-- or, the same rows
+SELECT id FROM entity WHERE eql_v4.json_tree_match(custom_field_values, $1::eql_v4.query_json_predicate)
+```
+
+`vars` are any value `encoding/json` writes as a JSON object (a map, a struct, or a `json.RawMessage` of one), and an empty map when the predicate has none; a Go string is refused, as it is for a `Contains` needle.
+They answer as the same Go value bound as `jsonb_path_match`'s `vars` through pgx or `database/sql` answers over plaintext. A nil value (`nil`, or a nil map, slice or pointer) is SQL `NULL` there, and `jsonb_path_match` is `NULL` for every row: `Match` returns an empty operand, which binds as `NULL`, so the query matches no row. The predicate is still compiled, so text the grammar does not read is still refused. `vars` that write the JSON `null`, such as `json.RawMessage("null")` or a pointer to a nil map, are refused as `json_input`, as plaintext refuses vars that are not an object.
+The Go side reads nothing of the predicate: the engine parses it and compiles it into a tree of the operands the methods above derive, one leaf per comparison.
+The grammar is the subset of SQL/JSON path that the stored terms can answer, as PostgreSQL reads it:
+
+- paths: `$`, then `.key`, `."quoted key"` and `[*]` steps;
+- values: a variable (`$v`, `$"v"`) or a JSON literal (a string, a number, `true`, `false`, `null`);
+- comparisons: `==`, `!=` (or `<>`), `<`, `<=`, `>`, `>=` between a path and a value, either side first;
+- `exists(path)`, and `exists(path ? (@.m == $v && @.n > $w))`, conditions on `@` or `@.member` joined by `&&`;
+- `&&`, `||`, `!(…)`, parentheses and `/* */` comments. Keywords are read in any case.
+
+An ordering (`<` and the rest) needs its path in `ordered=`, and so does every member of a filter of more than one condition, which compiles to the element match above.
+Anything else is refused before any term is derived, as an `*encrypt.Diagnostic` wrapping `encrypt.ErrEncoding`.
+`Construct()` names the construct in the grammar's own words (`` `like_regex` ``, `a position (`[2]`)`), and `At()` is the byte offset in the predicate where it starts; neither says anything of the predicate's text or the variables' values.
+`Reason()` says why:
+
+- `json_predicate_syntax`: text the grammar does not read.
+- `json_predicate_unsupported`: a jsonpath construct the stored terms cannot answer, such as `like_regex`, arithmetic, a method, `.**`, or a filter outside `exists`.
+- `json_predicate_unknown_variable`: a `$name` that `vars` does not hold.
+- `json_predicate_value_type`: an object or array as a value, or anything but a number or a string under an ordering or in a filter of more than one condition.
+- `json_predicate_not_equal_at_wildcard`: `!=` at a path with `[*]`.
+- `json_predicate_too_deep`: more than 32 levels of nesting, or a path deeper than a document.
+- `json_path_position`: a position (`[2]`) in a path.
+- `json_path_no_wildcard`: a filter on a path with no `[*]`.
+- `json_path_wildcard`: a `[*]` in a filter's member.
+- `json_syntax`: JSON text that does not parse, a literal in the predicate (`1e999`) or `vars` (`{`, or an empty or whitespace-only `json.RawMessage`); with `At()` for a literal, without it for `vars`.
+- `json_input`: `vars` that are not a JSON object (`[1]`, `1`, `null`).
+
+An ordering at a path the tag does not order is `encrypt.ErrTerm` with the code `stack_encrypt::json_path_not_ordered`.
+
+A query returns the rows `jsonb_path_match` returns over the plaintext, and each row answers as it does, true, false or NULL, save five cases:
+
+- Lax mode's unwrapping of an array the path names no `[*]` for is not applied: `$.tags == "a"` over `{"tags": ["a"]}` is false here. Write `[*]`.
+- Nor is its wrapping of a value that is not an array under `[*]`: `$.tags[*] == "a"` over `{"tags": "a"}` is false here. Keep an array an array.
+- A string compared with a number, which jsonpath calls unknown, is false under `==` and true under `!=`. Keep one kind of value at a path.
+- Orderings, and filters of more than one condition, compare the folded ordering terms: text case- and accent-folded, numbers as float64. So `exists($.c[*] ? (@.r == "sibling" && @.n > 5))` finds a `"Sibling"`; `==` anywhere else is exact.
+- An ordering of a JSON `null` is NULL where jsonpath says false, unless it sits under an odd number of `!`. Both pick the same rows, so a `WHERE` is unaffected; a `SELECT` of the boolean itself can show NULL. Under an odd number of `!`, where the rows would differ, the operand also asks that the node is not `null` (a `null` needle, which tells the database which rows hold `null` at that path); elsewhere it sends none.
+
+A filter of more than one condition reads each array to `opts.WildcardBound` elements, 64 when it is 0; a row whose array is longer fails the query with SQLSTATE 22023, as the element match does.
+
+`Match` costs an index: `eql_v4.json_tree_match` evaluates the tree inside a function, so it filters every row a scan gives it and no index engages.
+The per-operator form engages the GIN index for `@>` (`EqAt`, `Contains`) and a btree on an ordered path's term (`At.<Path>`).
+Where a query needs an index, write those leaves with the per-operator methods, or put one beside `Match` to narrow the rows: `WHERE col @> $1::eql_v4.query_json AND col @@ $2::eql_v4.query_json_predicate`.
+
 `OpenOne` reads one value and refuses a read whose path has a wildcard before anything is opened; open a wildcard read with `Open`.
 A needle and a value are Go values: a `json.RawMessage` of JSON text crosses as it is, and anything else is marshalled with `encoding/json`.
 `EqAt` and the orderings take a scalar: a string, a number, a bool, or `nil` for JSON `null`.

languages/golang/cmd/stashgen/main_test.go

@@ -468,6 +468,18 @@ func TestRunGeneratesJSONSearchFields(t *testing.T) {
 	if strings.Contains(written, "PrefsFieldElements") {
 		t.Error("Prefs orders no member, and got Elements")
 	}
+	// Every document field has Match, ordered paths or none: what a
+	// predicate may use is the engine's to decide when it compiles it.
+	for _, field := range []string{"ProfileField", "PrefsField"} {
+		for _, want := range []string{
+			"func (f " + field + ") Match(ctx context.Context, c *encrypt.Cipher, predicate string, vars any, opts encrypt.JSONReadOptions) (eql.JSONSearchPredicateQuery, error)",
+			"gensupport.JSONMatch[eql.JSONSearchPredicateQuery](ctx, f.field, c, predicate, vars, opts)",
+		} {
+			if !strings.Contains(written, want) {
+				t.Errorf("the generated file lacks %q", want)
+			}
+		}
+	}
 	// The members of one array, named as At names their paths; an ordered
 	// path that ends at its wildcard, or has none, is no array's member.
 	written, stderr, code = generate("\tProfile json.RawMessage `stash:\"profile,encrypt_into=JsonSearch(ordered=$.extra.emergency_contacts[*].relationship;$.extra.emergency_contacts[*].previous_call_count;$.tags[*];$.age)\"`")
@@ -598,6 +610,19 @@ func TestRunGeneratesJSONSearchFields(t *testing.T) {
 	if _, err := os.Stat(filepath.Join(dir, "customer_stash.go")); !os.IsNotExist(err) {
 		t.Fatal("a file was written after the refusal")
 	}
+	// The same for any document field, ordered or not, and an engine that
+	// answers no match.
+	dir = writeModule(t, map[string]string{"model.go": source("\tProfile json.RawMessage `stash:\"profile,encrypt_into=JsonSearch\"`")})
+	stderrBuf.Reset()
+	if code := run([]string{"-type", "Customer"}, dir, &bytes.Buffer{}, &stderrBuf, withoutOperator("match")); code != 1 {
+		t.Fatalf("exit %d, want 1\n%s", code, stderrBuf.String())
+	}
+	if got := stderrBuf.String(); !strings.Contains(got, "Customer.Profile") || !strings.Contains(got, "the engine answers no match on JsonSearch") {
+		t.Fatalf("stderr = %q", got)
+	}
+	if _, err := os.Stat(filepath.Join(dir, "customer_stash.go")); !os.IsNotExist(err) {
+		t.Fatal("a file was written after the refusal")
+	}
 	for _, c := range []struct{ name, fields, want string }{
 		{"a sealed null", "\tPrefs *json.RawMessage `stash:\"prefs,encrypt_into=JsonSearch,null=sealed\"`", "Customer.Prefs: a nil has an equality term and no other, so another index's column would show which values are absent; write null=sql"},
 		{"a nil document that does not say null=sql", "\tPrefs *json.RawMessage `stash:\"prefs,encrypt_into=JsonSearch\"`", "null=sql"},

languages/golang/encrypt/README.md

@@ -687,6 +687,38 @@ A row whose array is longer than
 `JSONReadOptions.WildcardBound` (64 by default) fails the query with
 SQLSTATE 22023.
 
+A whole filter binds as one parameter with the field's `Match`: a jsonpath
+predicate and its variables, as plaintext `jsonb_path_match(col, $1, $2)`
+takes them. The engine reads the predicate and compiles it into a tree of
+the operands above; the Go side reads none of it.
+
+```go
+q, err := users.Fields.Profile.Match(ctx, cipher,
+	`$."plan" == $plan && $."age" < $max`,
+	map[string]any{"plan": "pro", "max": 40}, encrypt.JSONReadOptions{})
+rows, err := db.QueryContext(ctx, `SELECT id FROM users WHERE profile @@ $1::eql_v4.query_json_predicate`, q)
+```
+
+`eql_v4.json_tree_match(profile, $1::eql_v4.query_json_predicate)` returns
+the same rows. The variables are any value `encoding/json` writes as an
+object, or `nil` for none. The grammar is paths of `.key`, `."key"` and
+`[*]`; `==`, `!=`, `<`, `<=`, `>`, `>=` with a variable or a JSON literal;
+`exists(path)` and `exists(path ? (@.a == $x && @.b > $y))`; `&&`, `||` and
+`!(…)`. An ordering, and each member of a filter of more than one condition,
+needs its path in `ordered=`. Anything else is refused as a `*Diagnostic`
+wrapping `ErrEncoding`, whose `Reason()`, `Construct()` and `At()` give the
+reason, the construct in the grammar's words and the byte offset in the
+predicate, never its text or the values. Each row answers as
+`jsonb_path_match` does over its plaintext, save five cases the stashgen
+README lists: lax unwrapping and wrapping of arrays are not applied, a
+string compared with a number is false under `==`, orderings and
+multi-condition filters compare folded terms, and an ordering of a `null`
+is NULL rather than false outside an odd number of `!`, which returns the
+same rows. The tree is evaluated inside a
+function, so no index engages: where a query needs one, write its leaves
+with `EqAt`, `Contains` and `At.<Path>`, whose `@>` and `->` comparisons
+use the GIN and btree indexes, or put one beside `Match` to narrow the rows.
+
 ## Building
 
 The package embeds `wasm/stack_encrypt_guest.wasm`, a build artefact of the

languages/golang/encrypt/checker.go

@@ -258,7 +258,7 @@ func parseOperators(value any) ([]record.Operator, error) {
 				op.Index, err = parseOperatorIndex(s)
 			case "input":
 				switch s {
-				case "value", "pair", "json", "path_json", "path_pair", "path", "path_bound", "path_predicates":
+				case "value", "pair", "json", "path_json", "path_pair", "path", "path_bound", "path_predicates", "predicate":
 					op.Input = s
 				default:
 					return nil, fmt.Errorf("operator %d: unknown input %q", i, s)
@@ -282,14 +282,14 @@ func parseOperators(value any) ([]record.Operator, error) {
 // of the names the engine's OpSpec::parse accepts, the scalar operators
 // (eq, lt, lte, gt, gte, between, matches) and a document's (contains,
 // contained_by, eq_at, the orderings at a path, between_at, selector,
-// subtree, children, child_count, element_match). The catalog (se_targets) and se_query_term are
-// read through this, parseFidelity and parseOperatorIndex, so the two
-// readers refuse the same values.
+// subtree, children, child_count, element_match, match). The catalog
+// (se_targets) and se_query_term are read through this, parseFidelity and
+// parseOperatorIndex, so the two readers refuse the same values.
 func parseOperatorName(s string) (string, error) {
 	switch s {
 	case "eq", "lt", "lte", "gt", "gte", "between", "matches",
 		"contains", "contained_by", "eq_at", "lt_at", "lte_at", "gt_at", "gte_at", "between_at",
-		"selector", "subtree", "children", "child_count", "element_match":
+		"selector", "subtree", "children", "child_count", "element_match", "match":
 		return s, nil
 	}
 	return "", fmt.Errorf("unknown operator %q", s)

languages/golang/encrypt/doc.go

@@ -265,6 +265,22 @@
 // array is longer than [JSONReadOptions.WildcardBound] (64 by default)
 // fails the query with SQLSTATE 22023.
 //
+// A whole filter binds as one parameter with the field's Match: a jsonpath
+// predicate and its variables, as plaintext jsonb_path_match takes them.
+// The engine parses the predicate and compiles it into a tree of the
+// operands above; nothing in Go reads it.
+//
+//	q, err := customers.Fields.Profile.Match(ctx, cipher,
+//		`$."name" == $name && $."age" < $max`,
+//		map[string]any{"name": "Alice", "max": 40}, encrypt.JSONReadOptions{})
+//	// WHERE profile @@ $1::eql_v4.query_json_predicate
+//
+// A predicate the engine cannot answer is refused as a [Diagnostic]
+// wrapping [ErrEncoding], whose Reason, Construct and At name the refusal
+// and the byte offset in the predicate where it starts. The tree is
+// evaluated inside eql_v4.json_tree_match, so no index engages: a query
+// that needs one writes its leaves with EqAt, Contains and At.<Path>.
+//
 // # Cancellation
 //
 // Every call takes a [context.Context], and a call whose context is