Testing agent: command-line verification for non-web projects

.claude/templates/coding_prompt.template.md

@@ -86,6 +86,13 @@ Implement the chosen feature thoroughly:
 
 **CRITICAL:** You MUST verify features through the actual UI.
 
+**No browser UI?** For CLI tools, scripts, libraries, and API-only services, verify through bash
+instead of the browser: plan the expected exit codes, output, and side effects first, run the
+program with simulated user input (stdin, arguments, input files), and check stdout, stderr, the
+exit code, and produced files against that plan -- including at least one invalid input. Save the
+scenario as `tests/feature_<id>_test.sh` (run with `bash tests/feature_<id>_test.sh`) so testing
+agents can re-run it for regression checks.
+
 Use `playwright-cli` for browser automation:
 
 - Open the browser: `playwright-cli open http://localhost:PORT`

.claude/templates/testing_prompt.template.md

@@ -9,7 +9,7 @@ You are assigned to test the following features: {{TESTING_FEATURE_IDS}}
 ### Workflow for EACH feature:
 1. Call `feature_get_by_id` with the feature ID
 2. Read the feature's verification steps
-3. Test the feature in the browser
+3. Test the feature the way a real user would (browser for web UIs, bash for CLIs/scripts/libraries)
 4. Call `feature_mark_passing` or `feature_mark_failing`
 5. Move to the next feature
 
@@ -25,7 +25,16 @@ Use the feature_get_by_id tool with feature_id=<ID>
 
 ### STEP 2: VERIFY THE FEATURE
 
-**CRITICAL:** You MUST verify the feature through the actual UI using browser automation.
+**CRITICAL:** You MUST verify the feature by actually running the application the way a real user would. Reading the code is NOT verification.
+
+First, decide how this feature is exercised. Check `app_spec.txt`, the feature's verification steps, and the project layout:
+
+- **Web UI** (pages, forms, buttons, anything opened in a browser) -> follow **2A: Browser Automation**
+- **No browser UI** (CLI tool, bash/Python/Node script, data pipeline, library/SDK, background worker, API-only service) -> follow **2B: Command-Line Verification**
+
+If a feature has both (e.g. a CLI that also serves a dashboard), verify each part with the matching method.
+
+#### 2A: Browser Automation (Web UI features)
 
 For the feature returned:
 1. Read and understand the feature's verification steps
@@ -58,6 +67,59 @@ For the feature returned:
 
 **Note:** Screenshots and snapshots save to files. Read the file to see the content.
 
+
+#### 2B: Command-Line Verification (CLI / script / library / non-UI features)
+
+Do NOT open a browser for these features. Exercise the program through bash, exactly as a user or calling program would, and judge the result only from observable evidence: **exit code, stdout, stderr, and the files/data it produces.**
+
+1. **Plan the success criteria BEFORE running anything.** Turn each verification step into concrete, checkable test cases:
+   - The exact command and arguments to run
+   - The simulated user input (stdin answers, arguments, environment variables, input files)
+   - The expected exit code (`0` on success, non-zero for documented error cases)
+   - The expected output (exact text, key lines, or a pattern) and the stream it goes to (stdout vs stderr)
+   - The expected side effects (files created/modified, database rows, generated output contents)
+   - At least one **invalid or edge-case input** (empty input, malformed data, missing file, bad argument) and the expected error behavior
+
+2. **Create input fixtures that mimic real user data** with the file Write tool, e.g. `tests/fixtures/feature_<id>_input.txt` and, where useful, `tests/fixtures/feature_<id>_expected.txt`.
+
+3. **Write a scenario script** (with the Write tool) such as `tests/feature_<id>_test.sh` that runs each planned test case and checks it. Keep it self-contained and repeatable:
+   ```bash
+   #!/usr/bin/env bash
+   # Feature #<id>: <feature name>
+   set -u
+   FAIL=0
+   WORK=$(mktemp -d)   # isolate side effects between runs
+   check() { if eval "$2"; then echo "PASS: $1"; else echo "FAIL: $1"; FAIL=1; fi; }
+
+   # Case 1: valid input supplied on stdin
+   printf 'alice\n42\ny\n' | timeout 30 ./app --out "$WORK/result.txt" > "$WORK/out" 2> "$WORK/err"; CODE=$?
+   check "exits 0"              '[ "$CODE" -eq 0 ]'
+   check "prints greeting"      'grep -q "Hello, alice" "$WORK/out"'
+   check "no errors on stderr"  '[ ! -s "$WORK/err" ]'
+   check "writes result file"   'diff -u tests/fixtures/feature_<id>_expected.txt "$WORK/result.txt"'
+
+   # Case 2: invalid input fails cleanly
+   timeout 30 ./app --out "$WORK/result.txt" < /dev/null > "$WORK/out" 2> "$WORK/err"; CODE=$?
+   check "rejects empty input"  '[ "$CODE" -ne 0 ]'
+   check "explains the error"   'grep -qi "input" "$WORK/err"'
+
+   rm -rf "$WORK"
+   exit $FAIL
+   ```
+   Adapt the command to the project (`python main.py`, `node cli.js`, `cargo run --`, a library driver script that imports the module and calls its public API, etc.). Wrap anything that could hang waiting for input in `timeout`. For a long-running worker or service, start it in the background inside the script, exercise it (send a request, drop a file in a watched folder, enqueue a job), check its logs, then stop it.
+
+4. **Run it and read ALL of the output:** `bash tests/feature_<id>_test.sh`. Do not rely on the exit code alone -- read the PASS/FAIL lines and look at the captured stdout/stderr for any test that fails.
+
+5. **Judge the result strictly.** The feature FAILS if any of these happen:
+   - An exit code, output, or side effect differs from what you planned
+   - A traceback, stack trace, panic, or unexpected warning appears on the success path
+   - An invalid input crashes the program instead of failing with a clear message and a non-zero exit code
+   - The program hides failures (exits `0` after an error, prints nothing, or only logs somewhere invisible). Missing error reporting is itself a bug: when you fix the feature, add clear stderr messages and correct exit codes to the program
+
+6. **Keep the evidence.** Commit the scenario script and fixtures so later regression runs can re-use them, and also run the project's own test suite (pytest, `npm test`, `go test`, etc.) if it has one.
+
+"It ran without crashing" is NOT a pass. A feature only passes when every planned criterion is met.
+
 ### STEP 3: HANDLE RESULTS
 
 #### If the feature PASSES:
@@ -74,18 +136,19 @@ A regression has been introduced. You MUST fix it:
    ```
 
 2. **Investigate the root cause:**
-   - Check console errors
-   - Review network requests
+   - Web UI: check console errors and review network requests
+   - CLI / non-UI: read the captured stderr, tracebacks, and exit codes from your scenario script
    - Examine recent git commits that might have caused the regression
 
 3. **Fix the regression:**
    - Make the necessary code changes
-   - Test your fix using browser automation
+   - Test your fix with the same method you used in Step 2 (browser automation or your scenario script)
    - Ensure the feature works correctly again
 
 4. **Verify the fix:**
    - Run through all verification steps again
-   - Take screenshots and read them to confirm the fix
+   - Web UI: take screenshots and read them to confirm the fix
+   - CLI / non-UI: re-run `bash tests/feature_<id>_test.sh` and confirm every case prints PASS
 
 5. **Mark as passing after fix:**
    ```
@@ -99,7 +162,7 @@ A regression has been introduced. You MUST fix it:
 
    - [Describe what was broken]
    - [Describe the fix]
-   - Verified with browser automation"
+   - Verified with browser automation / scenario script"
    ```
 
 ---
@@ -124,22 +187,28 @@ Use `playwright-cli` commands for browser interaction. Key commands:
 - `playwright-cli console` - Check for JS errors
 - `playwright-cli close` - Close browser (always do this when done)
 
+### Command-Line Verification (Bash)
+For CLIs, scripts, libraries, and other non-UI features:
+- Write a scenario script with the Write tool (e.g. `tests/feature_<id>_test.sh`) plus input/expected-output fixtures
+- Run it with `bash tests/feature_<id>_test.sh`
+- In the script, feed simulated user input via stdin (`printf '...' | ./app`, `./app < input.txt`), arguments, and environment variables
+- Capture stdout, stderr, and the exit code separately and check each against your planned criteria
+
 ---
 
 ## IMPORTANT REMINDERS
 
 **Your Goal:** Test each assigned feature thoroughly. Verify it still works, and fix any regression found. Process ALL features in your list before ending your session.
 
 **Quality Bar:**
-- Zero console errors
 - All verification steps pass
-- Visual appearance correct
-- API calls succeed
+- Web UI: zero console errors, visual appearance correct, API calls succeed
+- CLI / non-UI: expected exit codes, expected stdout/stderr, correct side effects, clean failure on invalid input
 
 **If you find a regression:**
 1. Mark the feature as failing immediately
 2. Fix the issue
-3. Verify the fix with browser automation
+3. Verify the fix with browser automation (web UI) or your scenario script (CLI / non-UI)
 4. Mark as passing only after thorough verification
 5. Commit the fix
 

CLAUDE.md

@@ -120,6 +120,7 @@ python test_security_integration.py   # Integration tests (9 tests)
 python -m pytest test_client.py       # Client tests (20 tests)
 python -m pytest test_dependency_resolver.py  # Dependency resolver tests (12 tests)
 python -m pytest test_rate_limit_utils.py     # Rate limit tests (22 tests)
+python -m pytest test_prompts.py             # Prompt template + migration tests (7 tests)
 ```
 
 ### React UI

prompts.py

@@ -17,7 +17,7 @@
 TEMPLATES_DIR = Path(__file__).parent / ".claude" / "templates"
 
 # Migration version — bump when adding new migration steps
-CURRENT_MIGRATION_VERSION = 1
+CURRENT_MIGRATION_VERSION = 2
 
 
 def get_project_prompts_dir(project_dir: Path) -> Path:
@@ -821,6 +821,73 @@ def _migrate_v0_to_v1(project_dir: Path) -> list[str]:
     return migrated
 
 
+# Phrases from the v1 testing prompt (browser-only) and their v2 replacements
+# (browser or command-line verification). Only exact matches are replaced, so
+# user-customized wording is left untouched.
+_V2_TESTING_PHRASES = (
+    (
+        "3. Test the feature in the browser\n",
+        "3. Test the feature the way a real user would (browser for web UIs, bash for CLIs/scripts/libraries)\n",
+    ),
+    (
+        "**Quality Bar:**\n- Zero console errors\n- All verification steps pass\n"
+        "- Visual appearance correct\n- API calls succeed\n",
+        "**Quality Bar:**\n- All verification steps pass\n"
+        "- Web UI: zero console errors, visual appearance correct, API calls succeed\n"
+        "- CLI / non-UI: expected exit codes, expected stdout/stderr, correct side effects, "
+        "clean failure on invalid input\n",
+    ),
+    (
+        "3. Verify the fix with browser automation\n",
+        "3. Verify the fix with browser automation (web UI) or your scenario script (CLI / non-UI)\n",
+    ),
+)
+
+
+def _get_template_section(name: str, start_pattern: str, end_pattern: str) -> str | None:
+    """Return the section of a base template from start_pattern up to (not including) end_pattern."""
+    template_path = TEMPLATES_DIR / f"{name}.template.md"
+    try:
+        template = template_path.read_text(encoding="utf-8")
+    except (OSError, PermissionError):
+        return None
+    match = re.search(f"{start_pattern}.*?(?={end_pattern})", template, flags=re.DOTALL)
+    return match.group(0) if match else None
+
+
+def _migrate_testing_prompt_to_v2(content: str) -> str:
+    """Add command-line verification guidance for non-web projects to a testing prompt.
+
+    The STEP 2 section is only replaced when it is the unmodified v1 (browser-only)
+    text, so project-specific customizations are preserved.
+    """
+    new_step2 = _get_template_section("testing_prompt", r"### STEP 2: VERIFY THE FEATURE", r"### STEP 3:")
+    if new_step2 and _CLI_TESTING_STEP2 in content:
+        content = content.replace(_CLI_TESTING_STEP2, new_step2, 1)
+
+    for old_phrase, new_phrase in _V2_TESTING_PHRASES:
+        content = content.replace(old_phrase, new_phrase, 1)
+
+    return content
+
+
+def _migrate_v1_to_v2(project_dir: Path) -> list[str]:
+    """Migrate testing_prompt.md to support CLI / non-web verification (issue #136)."""
+    migrated: list[str] = []
+    testing_prompt_path = get_project_prompts_dir(project_dir) / "testing_prompt.md"
+    if not testing_prompt_path.exists():
+        return migrated
+    try:
+        content = testing_prompt_path.read_text(encoding="utf-8")
+        updated = _migrate_testing_prompt_to_v2(content)
+        if updated != content:
+            testing_prompt_path.write_text(updated, encoding="utf-8")
+            migrated.append("Updated testing_prompt.md with command-line verification for non-web projects")
+    except (OSError, PermissionError) as e:
+        print(f"  Warning: Could not update testing_prompt.md: {e}")
+    return migrated
+
+
 def migrate_project_to_current(project_dir: Path) -> list[str]:
     """Migrate an existing project to the current AutoForge version.
 
@@ -836,7 +903,8 @@ def migrate_project_to_current(project_dir: Path) -> list[str]:
     if current < 1:
         migrated.extend(_migrate_v0_to_v1(project_dir))
 
-    # Future: if current < 2: migrated.extend(_migrate_v1_to_v2(project_dir))
+    if current < 2:
+        migrated.extend(_migrate_v1_to_v2(project_dir))
 
     _set_migration_version(project_dir, CURRENT_MIGRATION_VERSION)
     return migrated

test_prompts.py

@@ -0,0 +1,111 @@
+"""
+Prompt Tests
+============
+
+Tests for testing-agent prompt content and the v1 -> v2 prompt migration
+that adds command-line verification for non-web projects (issue #136).
+
+Run with: python -m pytest test_prompts.py -v
+"""
+
+import tempfile
+import unittest
+from pathlib import Path
+
+import prompts
+from prompts import (
+    _CLI_TESTING_STEP2,
+    CURRENT_MIGRATION_VERSION,
+    _migrate_testing_prompt_to_v2,
+    get_testing_prompt,
+    migrate_project_to_current,
+)
+
+# Minimal reproduction of a project testing_prompt.md scaffolded from the v1 template
+V1_TESTING_PROMPT = (
+    "## YOUR ROLE - TESTING AGENT\n\n"
+    "### Workflow for EACH feature:\n"
+    "1. Call `feature_get_by_id` with the feature ID\n"
+    "2. Read the feature's verification steps\n"
+    "3. Test the feature in the browser\n"
+    "4. Call `feature_mark_passing` or `feature_mark_failing`\n\n"
+    "---\n\n"
+    + _CLI_TESTING_STEP2
+    + "### STEP 3: HANDLE RESULTS\n\n"
+    "**Quality Bar:**\n- Zero console errors\n- All verification steps pass\n"
+    "- Visual appearance correct\n- API calls succeed\n\n"
+    "**If you find a regression:**\n"
+    "3. Verify the fix with browser automation\n"
+)
+
+
+class TestTestingPromptTemplate(unittest.TestCase):
+    def test_template_covers_browser_and_cli_verification(self):
+        prompt = get_testing_prompt(testing_feature_ids=[1, 2])
+        self.assertIn("1, 2", prompt)
+        self.assertIn("#### 2A: Browser Automation", prompt)
+        self.assertIn("#### 2B: Command-Line Verification", prompt)
+        # Key steps requested in issue #136
+        self.assertIn("Plan the success criteria BEFORE running anything", prompt)
+        self.assertIn("simulated user input", prompt)
+        self.assertIn("exit code", prompt)
+        self.assertIn("bash tests/feature_<id>_test.sh", prompt)
+
+    def test_template_step2_is_bounded_by_step3(self):
+        section = prompts._get_template_section(
+            "testing_prompt", r"### STEP 2: VERIFY THE FEATURE", r"### STEP 3:"
+        )
+        self.assertIsNotNone(section)
+        assert section is not None
+        self.assertIn("#### 2B:", section)
+        self.assertNotIn("### STEP 3:", section)
+
+
+class TestTestingPromptMigrationV2(unittest.TestCase):
+    def test_migrates_unmodified_v1_prompt(self):
+        updated = _migrate_testing_prompt_to_v2(V1_TESTING_PROMPT)
+        self.assertNotIn(_CLI_TESTING_STEP2, updated)
+        self.assertIn("#### 2B: Command-Line Verification", updated)
+        self.assertIn("### STEP 3: HANDLE RESULTS", updated)
+        self.assertNotIn("3. Test the feature in the browser\n", updated)
+        self.assertIn("CLI / non-UI: expected exit codes", updated)
+        self.assertIn("or your scenario script (CLI / non-UI)", updated)
+
+    def test_migration_is_idempotent(self):
+        once = _migrate_testing_prompt_to_v2(V1_TESTING_PROMPT)
+        self.assertEqual(once, _migrate_testing_prompt_to_v2(once))
+
+    def test_current_template_is_unchanged_by_migration(self):
+        template = (prompts.TEMPLATES_DIR / "testing_prompt.template.md").read_text(encoding="utf-8")
+        self.assertEqual(template, _migrate_testing_prompt_to_v2(template))
+
+    def test_customized_step2_is_preserved(self):
+        custom = V1_TESTING_PROMPT.replace(
+            "**CRITICAL:** You MUST verify the feature through the actual UI using browser automation.",
+            "**CRITICAL:** Run `make e2e` to verify.",
+        )
+        updated = _migrate_testing_prompt_to_v2(custom)
+        self.assertIn("Run `make e2e` to verify.", updated)
+        self.assertNotIn("#### 2B:", updated)
+
+    def test_migrate_project_to_current_updates_testing_prompt(self):
+        with tempfile.TemporaryDirectory() as tmp:
+            project_dir = Path(tmp)
+            prompts_dir = project_dir / ".autoforge" / "prompts"
+            prompts_dir.mkdir(parents=True)
+            (project_dir / ".autoforge" / ".migration_version").write_text("1")
+            testing_prompt = prompts_dir / "testing_prompt.md"
+            testing_prompt.write_text(V1_TESTING_PROMPT, encoding="utf-8")
+
+            migrated = migrate_project_to_current(project_dir)
+
+            self.assertEqual(len(migrated), 1)
+            self.assertIn("#### 2B:", testing_prompt.read_text(encoding="utf-8"))
+            version = (project_dir / ".autoforge" / ".migration_version").read_text()
+            self.assertEqual(version, str(CURRENT_MIGRATION_VERSION))
+            # Second run is a no-op
+            self.assertEqual(migrate_project_to_current(project_dir), [])
+
+
+if __name__ == "__main__":
+    unittest.main()