feat(tests): opt-in NATIVEAPPTEMPLATE_TESTS=1 runs the generated apps' test suites

[dadachi]

Closes #150

Stacked on #153 (#149, formatters): it reuses that PR's runStep helper, StepResult type and report statusCard. Base is feat/autoformat-after-rename; merge #153 first and GitHub retargets this one to main.

What

SPEC §4 Operation 3 says bin/rails test, xcodebuild test and ./gradlew test must pass, but Layer 2 only builds. I took the preferred option from the issue: an opt-in flag, plus docs saying the suites don't run by default.

NATIVEAPPTEMPLATE_TESTS=1 (also on / true):

• Puts Layer 2 in build mode (as VISUAL>=1 does).
• After the visual phase (so xcodebuild test doesn't compete with Stage 1/2 for the simulator) and after self-repair (so the suites test the final code), runs these in parallel:
  • rails: bin/rails db:test:prepare then bin/rails test (mise exec -- when mise is available)
  • ios: xcodebuild -project <P>.xcodeproj -scheme <P> -destination $NATIVEAPPTEMPLATE_IOS_DESTINATION -configuration Debug test (same as the Layer 2 build, so the build is reused)
  • android: ./gradlew testDebugUnitTest --no-daemon (JVM, no emulator)
• New src/validation/test-suites.ts: testsEnabled, runTestSuite, extractSummary (Minitest / XCTest / Swift Testing / xcodebuild / Gradle summary lines), applyTestResults.
• A failing suite fails the run (overallPass false → verdict FAIL, exit code non-zero), since you opted in and the SPEC says they must pass. Skipped suites (e.g. no .xcodeproj) are shown but don't count. The summary gets · tests x/y pass.
• Report: platforms[].tests and meta.tests in report.json; a "Test suites" chip in the gate strip, a "Test suites" section (command, exit, duration, the runner's summary line, output tail on failure), tests: on in the header, and NATIVEAPPTEMPLATE_TESTS=1 in the reproduce command. Trace: dispatch.log per-platform line, output tail in <platform>.log on failure.
• runStep now keeps only the last ~1 MB of output (xcodebuild test prints a lot).
• Docs: SPEC §4 Operation 3 ("as shipped: don't run by default…"), README (operation 5 + flag), AGENTS.md, docs/validation-report.md, docs/OUTPUT-CHECKLIST.md §3, generate-app skill (also mentions Run the generated apps' linters (with autocorrect) after the rename #149's formatter results).

Not done / choices to check

• Test failures are not fed to the self-repair loop (the loop targets Layer 1/2). The SPEC line "Failures feed into the self-repair loop" is still the target; the "as shipped" note says so.
• Timeouts: rails 30 min, iOS 45 min, android 30 min, db:test:prepare 5 min.

Tests

• Fake bin/rails / gradlew / xcodebuild: the exact commands and args (including NATIVEAPPTEMPLATE_IOS_DESTINATION), db:test:prepare before the suite and a failing prepare stopping it, summary extraction, skipped when the project files are missing. Flag parsing. applyTestResults gating + summary, and the report section, gate chip and reproduce prefix (absent when off). runStep output cap.
• Saw all 5 new tests red by breaking the targets.
• Real: runTestSuite("android", …) on a copy of the paid Android substrate → pass, BUILD SUCCESSFUL in 36s. I didn't run the real Rails suite (on an un-renamed copy, db:test:prepare would hit the substrate developer's own nativeapptemplateapi_test DB) or the real iOS suite (needs the simulator, takes minutes).
• npm run ci: 170/170 pass.

🤖 Generated with Claude Code

https://claude.ai/code/session_01UZJxUoubntzu7Ec7DPVfAr