feat: kosli evaluate policy --identifier, evaluate a published Rego policy

[jumboduck]

Implements the CLI side of kosli-dev/server#7081: kosli evaluate policy can run a Rego policy published with kosli publish rego-policy (#1256), by its identifier.

kosli evaluate policy --context trail=my-flow/my-trail --identifier pr-approved
kosli evaluate policy --context trail=my-flow/my-trail --path policies/pr-approved --path lib/ergo.rego

What it does

--path replaces --policy (commit 1)

• --path (-p, repeatable) sends the policy's files, as --policy did.
• --policy stays for one release as a deprecated alias: hidden from the help, and Flag --policy has been deprecated, use --path instead on stderr (so --output json stays clean). sdlc-policies runs the latest CLI and passes --policy.
• Help, examples and the bundle errors say --path.

--identifier (commit 2)

• Sends policy: {"identifier": "pr-approved"}. The server evaluates the policy's current version, the highest valid one. No files are read.
• Exactly one of --identifier or --path is required, refused before any request: only one of --identifier or --path can be given / one of --identifier or --path is required.
• The check uses resolved values, so a KOSLI_IDENTIFIER exported by a pipeline is refused next to --path rather than silently winning or losing.
• The server's refusals (unknown policy, no valid version, wrong trail count, params off the schema) are shown in its words.

The policy that ran is printed (commits 3 and 5)

• --output json prints the evaluation as the server gave it: id, status, policy, result: {allow, outputs}, decision_attestation_id. The policy's outputs stay under result.outputs, so no output name can clash with the fields beside them. --assert reads result.allow.
• The table is a summary, with a POLICY: row: a published policy as pr-approved, version 3 (sha256:…), an inline policy by its digest only.
• A server that does not cite the policy yet prints no POLICY row.

RESULT:      DENIED
VIOLATIONS:  change is not approved
POLICY:      pr-approved, version 3 (sha256:abc123)
DECISION:    01DECISION

Decisions worth a reviewer's attention

• Flag names as in the ticket. The CLI binds every flag to KOSLI_<FLAG>, so KOSLI_IDENTIFIER and KOSLI_PATH are read too. A conflict is refused (tested), not resolved silently.
• evaluate policy --output json changes shape. It no longer copies the flat evaluate trail shape (allow, violations and outputs at the top level); it echoes the server's evaluation (review feedback). This removes the reserved output names from its help. sdlc-policies, which reads the old shape, is updated after the release.
• evaluate trail / evaluate trails are unchanged, including --server-side, which still prints the same bytes as the local path (tested with a verdict that cites a policy). runServerEvaluation is split: awaitServerEvaluation creates and waits, and each command prints.
• No local identifier check. The identifier travels in the body, not the path; the server validates it, and an empty value is already refused by the CLI's general empty-flag check.

Depends on

• kosli-dev/server#7373 (open) for the identifier variant and the policy field in the response. Until it is merged, --identifier gets a 400 from the server, and no POLICY row is printed.

Not in this PR

• sdlc-policies moves from --policy to --path, and reads the new JSON shape (that repo).
• A later CLI release removes --policy.

Testing

• New tests in cmd/kosli/evaluatePolicy_test.go (14), internal/evaluations/client_test.go (3), and a cited-policy case in evaluateServerSide_test.go, all against stubbed servers.
• Empty-flag audit: path and identifier in spec.json and the regenerated coverage.json.
• make test_integration passes (3609 tests, 53 skipped). make lint: 0 issues.
• Tried the built binary: dry runs of each refusal, and the table/JSON output against a local stub.
• Not yet done: a run against staging, once server#7373 is merged.

🤖 Generated with Claude Code

[claude]

Claude encountered an error after 3m 12s —— View job

---

Follow-up review of commits 0a928c4..bdb3b94

• Fetch existing review comments and diff between commits
• Assess status of each previous finding touched by new commits
• Check changed lines for new bugs/regressions
• Close resolved inline threads (all 5 were already marked resolved on GitHub)
• Post summary comment

Changed in this push: cmd/kosli/evaluateHelpers.go, evaluateHelpers_test.go, evaluatePolicy.go, evaluatePolicy_test.go, internal/evaluations/client.go, client_test.go — this is the "evaluate policy --output json echoes the evaluation as the server gave it" commit (bdb3b94).

Posted the findings summary as a PR comment. No new bugs found in the changed lines — the refactor (runServerEvaluation split into awaitServerEvaluation + printing, Evaluation.Raw added, printEvaluation in evaluatePolicy.go) traces correctly end-to-end, and the new/updated tests (TestItPrintsTheEvaluationAsTheServerGaveIt, TestAnOutputNamedPolicyStaysAnOutput, TestGetKeepsTheBodyAsTheServerGaveIt) match the new behavior.

[claude]

Follow-up review: commits 0a928c4..bdb3b94

Reviewed only the lines touched by the newest commit (bdb3b94, "evaluate policy --output json echoes the evaluation as the server gave it"). Status of each previously-raised finding whose location this commit touched:

1. citedPolicyText robustness nit (evaluateHelpers.go:786, claude) - not addressed; function unchanged by this commit. Already explicitly skipped by jumboduck with rationale (server guarantees digest and version >= 1 for a cited identifier). No regression.
2. --policy/--identifier error message wording (evaluatePolicy.go:200, claude) - not addressed; the refusal line is unchanged. Already explicitly skipped by jumboduck (deprecation warning on stderr explains the link). No regression.
3. "Do we need this, echo back the server's own reference" (evaluateHelpers.go, pbeckham) - resolved. evaluate policy --output json now prints evaluation.Raw (the server's own body) via the new printEvaluation in evaluatePolicy.go, instead of a hand-built auditResult with an injected policy field. runServerEvaluation no longer carries citePolicy/cited at all.
4. Warn when both --path and --identifier are set (evaluateHelpers.go:338, pbeckham) - unchanged behavior (both are still refused outright in evaluatePolicy.go's run(), stricter than a warning), as explained by jumboduck's reply. No regression.
5. "Ugly" taken-names caveat in help text (evaluatePolicy.go, pbeckham) - resolved. The taken-names sentence is gone from the help; a rule named policy is now printed normally under result.outputs in the JSON (verified by the new TestAnOutputNamedPolicyStaysAnOutput), since it can no longer clash with the top-level policy/decision_attestation_id fields.

All 5 original inline review threads are already marked resolved on GitHub.

New findings in the changed lines: none. Traced the awaitServerEvaluation/printEvaluateResult split, Evaluation.Raw plumbing (Get -> WaitForTerminal -> decodeEvaluation), and the new printEvaluation table/JSON branches - all consistent with existing conventions, and covered by the updated tests.

Generated with Claude Code