[cipherstash-agent]

Summary

Before: there was no tool that tells a reviewer how risky a pull request is to release, as opposed to line-by-line review comments.

After: change-risk-research/ holds a change-risk agent skill and the evaluation behind it. The skill is a set of instructions plus small scripts that Claude Code follows when asked to assess a PR. It writes a short report: how much damage the change could do, whether it needs a closer look, and the specific ways it could break. The skill was tested on 41 historical PRs from this repository and compared with CodeRabbit and a rule-based baseline.

How: each historical PR was rebuilt as reviewers saw it before merge, in an isolated sandbox with no later history. The skill assessed it, and its findings were scored against bugs this repository fixed later. All findings were then scored a second time, blind.

Nothing here runs anywhere. It adds no CI workflow, changes no product code, and ships nothing to customers. The skill sits in change-risk-research/.agents/skills/, not in the root skills/ directory that ships in the stash npm package, so no changeset is needed.

Changes

• Skill: change-risk-research/.agents/skills/change-risk/. Contains SKILL.md, references, a JSON schema for the report, an evidence collector, a validator, a renderer and unit tests.
• Results and recommendation: docs/final-recommendation.md and eval/comparison.md.
• Method: research/methodology.md, fixed before scoring. Research on PR-Agent (Qodo) and CodeScene is in research/.
• Proposed CI integration: docs/ci-integration.md, with an inactive example workflow under docs/examples/. It is deliberately not in .github/workflows.
• Evidence: eval/. Holds the corpus, ground truth, every run's assessment and gzipped transcript, and both sets of labels.
• Harness: harness/, the Python scripts that build cases, run them and score them.

Headline results (31 holdout PRs)

• The skill asked for a closer look on 21 of the 22 PRs that warranted one, and on 1 of the 9 low-risk ones.
• Between 77% and 90% of its non-trivial findings were sound, depending on which adjudicator is used.
• It found 3 or 4 of the 11 known bugs. It missed the most severe one in every run.
• Adding history signals, the PR description or CI results did not measurably help.
• A run costs about $1.50 and takes about 3 minutes.

The recommendation is to use it on demand alongside CodeRabbit, and not to gate merges on it.

Verification

• pnpm run code:check: no errors (warnings and infos only, which CI allows).
• pnpm run test:scripts: 1415 passed, 1 skipped. One earlier run failed once, in go-module-path.test.mjs; three reruns of that file and two full reruns passed. I could not reproduce the failure.
• Skill unit tests: cd change-risk-research/.agents/skills/change-risk/scripts && python3 -m unittest discover -s tests.
• Package builds and tests were not run, because no package changed.

Review notes

• Size: about 1,150 files and 52 MB. About 46 MB of that is gzipped evaluator transcripts in eval/change-risk/results/ and eval/adjudication-blind/raw/. They are the audit trail showing that no run saw later fixes. They could be dropped before merge if repository size matters more.
• Start with docs/final-recommendation.md, then eval/comparison.md.
• All of this was produced and scored by agents. A human review of the 12 PRs with known bugs is still outstanding, and the report says so.

---

Generated by Claude Code

[changeset-bot]

⚠️ No Changeset found

Latest commit: 7144a8d

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}