Skip to content

docs(jules): narrow the lane to verifiable work and cut Claude's share - #12453

Open
MarkusNeusinger wants to merge 6 commits into
mainfrom
docs/jules-skill-round3
Open

MarkusNeusinger wants to merge 6 commits into
mainfrom
docs/jules-skill-round3

Conversation

@MarkusNeusinger

Copy link
Copy Markdown
Owner

Summary

  • Jules lane narrowed after three test rounds (2026-10-27 to 10-29). Jules now takes mechanically verifiable work (codemods, migrations, tests, lint fixes); behaviour-preserving refactors go to /gemini-task, because only three of nine page-split variants kept their page unchanged and steering all nine took many Claude turns. CLAUDE.md, .claude/guardrails.md and the gemini-task skill follow (owner's agreement of 2026-10-29).
  • Claude's share cut. Every Monitor event is a Claude turn, so the skill watches a round with one low-noise Monitor: watch-sessions.py gains --changes-only and repeatable --allow <prefix> (flags scratch files such as fix_*.py or get_lines.sh), settles only after its PR checks ran, and rejects unknown flags. Prompts now forbid confirmation questions and string-sliced code edits; stuck sessions get a budget (one nudge, about 60 minutes in the provisioning queue); a REST fallback covers the MCP send failures.
  • How to prove unchanged behaviour for refactors: a jsdom snapshot with the existing test's mocks (0 vs 206 differing lines decided DebugPage), a browser comparison built against the production API, a verbatim check of canvas paint callbacks, and the Flash check on the PR into main for loading and error states. Also recorded: Jules names branches <base>-<session id>, and workflow triggers come from the pushed commit, so branches cut before a CI change keep old triggers.

Plan

N/A

Test plan

  • tests/unit/test_agent_instructions.py: 36 passed
  • ruff check and ruff format --check on watch-sessions.py
  • Watcher against real sessions: --allow flagged get_lines.sh on refactor(app): split PalettePage into hooks and sections #12451, --changes-only stayed quiet for running sessions, -h and unknown flags exit 2
  • REST :sendMessage fallback from the skill returned HTTP 200
  • uv run python -m tools.changelog check --base origin/main
  • Gemini 3.8 Flash pre-PR check: its five findings fixed (unset $D, settle before PR check, grep dropping PR lines, single-dash flags, missing kill)

MarkusNeusinger and others added 3 commits October 29, 2026 15:26
Three test rounds (2026-10-27 to 10-29): watcher events dominated Claude's
cost, sessions asked despite interactive=false, and behaviour-preserving
refactors mostly failed. The skill now limits Jules to mechanically verifiable
work, watches with --changes-only/--allow, forbids confirmation questions and
string-sliced edits in prompts, budgets stuck sessions, adds a REST fallback
for messages, and records how to prove unchanged UI behaviour.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…fixed

Gemini Flash pre-PR check: a GitHub hiccup could end the watch before the PR
was checked, single-dash flags were taken as session IDs, the skill's grep
dropped the PR lines, the REST fallback used an unset $D, and stopping a
preview server by port lacked the kill.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copilot AI lite review requested due to automatic review settings October 29, 2026 17:08
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Active critical and moderate findings remain in the watcher’s error-status handling, option parsing, and synchronized routing documentation, so the change is not ready for approval.

3 open issues
What changed in this PR

This PR narrows Jules assignments to mechanically verifiable work, routes larger behavior-preserving refactors to Gemini, and reduces monitoring overhead through watcher, prompt, fallback, and verification updates.

Changes:

  • Updates Jules/Gemini routing guidance, guardrails, and operational documentation.
  • Adds quieter session monitoring, path allowlists, stricter argument handling, and PR validation.
  • Documents branch conventions, fallback messaging, and unchanged-behavior verification procedures.
  • Final review findings remain unresolved: credential exposure in the REST fallback (moderate, 1 vote), lost watcher exit status (critical, 2 votes), option-value parsing accepting unknown flags (moderate, 3 votes), and an outdated synchronized routing guide (moderate, 2 votes).
File Description
CLAUDE.md Updates agent-selection guidance to restrict Jules to mechanically verifiable work.
changelog.d/​jules-skill-round3.md Documents the Jules lane, watcher updates, and verification results.
.claude/​skills/​jules-task/​SKILL.md Defines Jules task boundaries, monitoring, prompts, fallback handling, and review procedures.
.claude/​skills/​jules-task/​scripts/​watch-sessions.py Adds quiet monitoring, path allowlists, argument parsing, and PR validation.
.claude/​skills/​gemini-task/​SKILL.md Routes larger behavior-preserving refactors to Gemini.
.claude/​guardrails.md Aligns delegation rationale and cost guidance with the revised Jules/Gemini boundaries.

Review effort: Lite

Comment thread .claude/skills/jules-task/SKILL.md
Comment thread .claude/skills/jules-task/scripts/watch-sessions.py
Comment thread CLAUDE.md
@codecov

codecov Bot commented Oct 29, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

MarkusNeusinger and others added 2 commits October 29, 2026 18:24
… code behind grep

Copilot review of #12453.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants