[zwanski-start]

Summary

Testing agents only knew how to verify features through the browser, so projects that are run from bash (CLIs, scripts, data pipelines, libraries, API-only services) got weak regression testing. This PR teaches the testing agent a command-line verification workflow that follows the steps proposed in the issue.

.claude/templates/testing_prompt.template.md

STEP 2 now starts by deciding how the feature is used:

• 2A: Browser Automation: the existing Playwright CLI instructions, unchanged.
• 2B: Command-Line Verification, for features without a browser UI:
  0. Plan success criteria before running anything: command, simulated input, expected exit code, stdout/stderr, side effects, and at least one invalid or edge-case input.
  1. Make errors and successes visible: a program that hides failures (exits 0 after an error, stays silent) is treated as a bug, and the fix must add clear stderr messages and correct exit codes.
  2. Run the program from bash through a repeatable scenario script tests/feature_<id>_test.sh, started with bash …. The default allowlist includes bash, but not printf, diff or timeout as top-level commands.
  3. Simulate user input with fixtures, stdin piping, arguments and env vars, all inside the script. Heredocs are avoided because they fail in the sandbox.
  4. Collect the error or success: exit code, stdout, stderr and produced files are captured separately and checked one by one. The script prints PASS/FAIL per check, and the agent judges strictly ("ran without crashing" is not a pass).
• STEP 3, AVAILABLE TOOLS and the reminders now cover both methods.

.claude/templates/coding_prompt.template.md

A short "No browser UI?" note in STEP 5 tells coding agents to verify the same way and save the scenario script, so testing agents can rerun it later. YOLO mode still removes the whole of STEP 5 as before.

prompts.py: v1 → v2 migration

Existing projects keep their copied testing_prompt.md, so CURRENT_MIGRATION_VERSION goes up to 2 and a new _migrate_v1_to_v2 runs on agent start:

• It replaces STEP 2 with the template's new STEP 2 only if STEP 2 is still the unmodified v1 text (_CLI_TESTING_STEP2). Customized prompts are not touched.
• It updates a few exact v1 phrases (workflow line, quality bar, regression reminder).
• Running it again changes nothing, and the v0 → v1 → v2 chain works because v0→v1 writes exactly the v1 STEP 2.

Verification

• New test_prompts.py (7 tests): template content, migration of an unmodified v1 prompt, no-op on a second run, current template left unchanged, customized STEP 2 preserved, and the end-to-end migrate_project_to_current version bump.
• python -m pytest test_prompts.py → 7 passed. ruff check . → clean. python test_security.py → all pass.
• Took the example scenario script out of the prompt and ran it against a small bash CLI: all 6 checks PASS and the script exits 0. After breaking the CLI (exit 0 on empty input), the matching check prints FAIL and the script exits 1.
• No UI changes, so I did not run a dev server or a browser check.

Assumptions / notes

• I didn't add a separate agent type or project-type detection in code. The testing agent chooses 2A or 2B per feature from app_spec.txt and the feature's steps, which also covers mixed projects (for example a CLI that also serves a dashboard). Doing it in the prompt avoids guessing the project type wrongly.
• I left the bash allowlist (security.py) alone. Running the scenario through bash script.sh uses the existing allowed bash entry. Projects still need their interpreter (python, cargo, …) available as they already do today.

Closes #136

🤖 Generated with Claude Code

[zwanski-start]

🔍 QA report: ✅ Passed

Tester: Dennis (cubefarm agent, in a fresh session) · Round: 1 · Author: Guido

The testing prompt now has a command-line verification path (STEP 2B) that covers all five steps from issue #136: plan success criteria, make errors visible, run from bash, simulate user input, collect exit code/stdout/stderr/side effects. The v1→v2 migration correctly upgrades real v1 project prompts (and the v0→v1→v2 chain), is idempotent and leaves customised prompts alone. Lint, the new prompt tests and the security tests pass; the mypy errors and pytest failures are the same on master. Gap that doesn't block the merge: migrated v1 projects keep browser-only wording in STEP 3 and don't get the new AVAILABLE TOOLS bash section.

	Check	Details
✅	Issue #136 acceptance: plan criteria / error reporting / bash run / simulated input / collect result	Read the new STEP 2B in testing_prompt.template.md: sub-steps 1–5 map directly onto the issue's five steps, with strict pass/fail rules ("ran without crashing" is not a pass). The coding prompt tells coding agents to save tests/feature__test.sh for later reruns.
✅	Example scenario script from the prompt works	Extracted the script from the template and ran it against a toy bash CLI: 6/6 PASS, exit 0. With the CLI broken (exit 0 on empty input): 'rejects empty input' printed FAIL and the script exited 1.
✅	Security allowlist intact, prompt commands allowed	security.py unchanged. bash_security_hook allows 'bash tests/feature_3_test.sh' and blocks top-level printf/timeout/diff, which matches the prompt's advice to run them inside the script.
✅	v1→v2 migration on a real v1 prompt (master template)	_CLI_TESTING_STEP2 matches master's template exactly; after migration STEP 2 and the Quality Bar match the new template, the version file is set to 2, get_testing_prompt returns 2B, and a second run returns [].
✅	v0→v1→v2 chain	An MCP-era prompt migrates through both steps: no mcp__playwright text left, 2B present, version 2.
✅	Customised STEP 2 preserved / idempotency	Covered by the test_prompts.py unit tests, which pass; the template is unchanged by the migration.
✅	YOLO mode still strips STEP 5 from the coding prompt	get_coding_prompt(yolo_mode=True) has the YOLO STEP 5 header and no 'No browser UI?' note; the normal prompt keeps the note.
⏭️	Migrated v1 prompts get full parity with the new template	Doesn't block the merge: migrated v1 prompts don't get the STEP 3 CLI regression-fix wording or the AVAILABLE TOOLS 'Command-Line Verification (Bash)' section (49-line diff against the new template). New projects get the full template.
⏭️	UI / browser check	The PR doesn't touch ui/ or the server; there is nothing to render, so no browser session or screenshots.

🧪 Commands run

Command	Result
ruff check .	All checks passed
mypy .	24 errors in 9 files, none in prompts.py; master shows the same 24 errors
python -m pytest test_prompts.py -q	7 passed
python test_security.py	ALL TESTS PASSED
python test_security_integration.py	ALL INTEGRATION TESTS PASSED
python -m pytest -q (full suite, PR vs origin/master worktree)	151 passed, 21 failed; exactly the same 21 failures on master (e.g. test_devserver_security), so none come from this PR
probe.py: migrate master's testing_prompt template and full project via migrate_project_to_current	STEP 2 replaced, version 2, rerun no-op; 49-line remaining diff in STEP 3 / AVAILABLE TOOLS
bash tests/feature_7_test.sh (example from the prompt, toy CLI; then broken CLI)	6/6 PASS exit 0; broken: 1 FAIL exit 1
bash_security_hook probe for bash/printf/timeout/diff	bash script allowed; printf/timeout/diff blocked at top level as the prompt says
get_coding_prompt(yolo_mode=True) + v0→v1→v2 chain probe	YOLO strip OK; chain migrates to version 2 with 2B

No browser screenshots were taken in this round.

_{Posted by cubefarm · merges automatically once GitHub's checks pass}