Skip to content

Show live agent browser in the feature modal - #321

Open
zwanski-start wants to merge 1 commit into
AutoForgeAI:masterfrom
zwanski-start:swarm/issue-70-guido
Open

zwanski-start wants to merge 1 commit into
AutoForgeAI:masterfrom
zwanski-start:swarm/issue-70-guido

Conversation

@zwanski-start

Copy link
Copy Markdown

Closes #70

Context

Most of what the issue asks for is already on master. Agents always run Playwright headless (server/routers/agent.py, settings.py), so desktop browser windows no longer pop up. BrowserViewService streams playwright-cli screenshot frames over the project WebSocket to the Browsers view (sidebar, shortcut B) and to the debug panel's Browsers tab, which covers the "dedicated agent page" option.

The part still missing was the issue's other option: getting to the browser from the feature card. This PR adds that and fixes two subscription bugs that showed up once a third consumer was added.

Changes

  • FeatureBrowserPreview (new): a "Live Browser" section in the feature detail modal. It shows the latest frame from each agent (coding or testing) working on that feature, including batched agents matched through featureIds. While an agent is on the feature but hasn't opened a browser yet, it shows a placeholder instead. An All browsers button closes the modal and opens the Browsers view. Screenshots stream only while the modal is open.
  • FeatureModal: renders the preview and now shows an IN PROGRESS status. Before this, in-progress features showed "PENDING".
  • FeatureCard: a small "Live" hint on cards that have an active agent, so users know clicking the card shows the browser.
  • useProjectWebSocket:
    • Subscriptions are now ref-counted. The server keeps a single subscribed flag per socket, so before this, closing one consumer (for example the modal) stopped streaming for every other consumer.
    • Subscriptions are restored on reconnect. Before this, a reconnect, or subscribing before the socket was open, silently stopped all frames.
    • When an agent finishes (agent_update success/error), its last frame is removed so a finished browser no longer looks live.

No backend changes.

Verification

  • npm run lint: 0 errors. The 6 warnings were already on master.
  • npm run build: passes.
  • Manual check in Playwright against the Vite dev server (port 5820) and a backend using a throwaway HOME and registry. The project WebSocket was mocked with page.routeWebSocket to send an agent_update for feature fix: auto-activate venv in start_ui.sh #1 and browser_screenshot frames every 2s after browser_view_subscribe:
    • the kanban card shows the "Live" hint
    • opening the card shows IN PROGRESS and the live frame in the modal
    • All browsers closes the modal, opens the Browsers view, and the frames keep updating there (timestamps advanced), which confirms the ref-counted subscribe/unsubscribe hand-off

I did not test this against a real running agent.

Assumptions

  • An actual iframe isn't possible: the agent's browser is a separate headless Chromium controlled by playwright-cli, not a URL the UI can load. Streaming screenshots, which the existing service already does, is the practical in-UI equivalent.
  • Testing sessions are still only cleaned up server-side when the orchestrator stops. This PR only clears their stale frames on the client.

🤖 Generated with Claude Code

Agents already run Playwright headless and stream screenshots to the
Browsers view. This makes the live browser reachable from the feature
card as the issue asks:

- FeatureModal embeds a Live Browser preview for any agent working on or
  testing the feature, with a shortcut to the full Browsers view
- FeatureCard shows a Live hint when an agent is on the feature
- Ref-count browser view subscriptions in useProjectWebSocket so one
  consumer unmounting no longer stops streaming for the others, and
  resubscribe after a reconnect
- Drop a finished agent's last frame so it doesn't linger as live

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@zwanski-start

Copy link
Copy Markdown
Author

🔍 QA report: ✅ Passed

Tester: Dennis (cubefarm agent, in a fresh session) · Round: 1 · Author: Guido

The feature detail modal now has a Live Browser section showing each coding/testing agent's latest frame for that feature, including batched agents matched through featureIds. It shows a placeholder while the agent hasn't opened a browser yet, and the section is hidden when no agent is on the feature. In-progress features now read IN PROGRESS instead of PENDING, cards with an active agent get a 'Live' hint, and 'All browsers' closes the modal and opens the Browsers view with frames still streaming. In a browser test with a mocked project WebSocket, I confirmed the subscription counting, the resubscribe after reconnect, and the removal of finished agents' frames. Lint and build pass, and the change is frontend-only (5 UI files, no backend or schema changes).

Check Details
✅ Code review of the diff (5 UI files) FeatureBrowserPreview subscribes on mount and unsubscribes on unmount. Subscriptions are counted, and the hook only sends a subscribe/unsubscribe message when the count goes between 0 and 1, which matches the server's single subscribed flag per socket (server/websocket.py:997-1005). Agent indexes are global on the server, so matching on agentIndex+agentType is safe. The frame cleanup on agent_update success/error handles the agentIndex=-1 case by matching featureId. No debug code left in, no backend or security changes.
✅ Card 'Live' hint Kanban cards with an active agent show the Monitor icon and 'Live' with a tooltip. The card with no agent (#2, after re-mocking) has no hint.
✅ Modal shows IN PROGRESS and the live frame Opened feature #1 (in_progress=1). The status reads IN PROGRESS and the Spark/coding frame renders. The frame timestamp advanced from 21:04:30 to 21:04:32 while the modal was open.
✅ Testing agent and batched agent frames in the modal Feature #3's modal shows both the testing agent frame (Fizz, featureId 3) and the batched coding agent frame (Octo, primary feature 2, featureIds [2,3]).
✅ Placeholder before the agent opens a browser / no section without an agent With an agent on feature #1 and no frames, the modal shows 'The browser view will appear here once the agent opens a browser.' Feature #2 with no agent shows no Live Browser section and status PENDING.
✅ 'All browsers' hand-off Clicking it closes the modal and switches to the Browsers view. The client sends unsubscribe then subscribe in the same tick, and all 3 frames keep updating there (timestamps advanced).
✅ Modal close stops streaming Closing the modal with no other consumer sends a single browser_view_unsubscribe. Reopening sends subscribe.
✅ Resubscribe after reconnect With the Browsers view open, I force-closed the project socket. On reconnect the client sends browser_view_subscribe again and frames resume (timestamps advanced 21:05:53 to 21:05:54).
✅ Finished agent's frame removed Sending agent_update state=success for coding agent 0 dropped its frame (2 to 1 visible). An untracked testing success (agentIndex -1, featureId 3) dropped the testing frame (1 to 0).
✅ Phone layout (390x844) The modal with the Live Browser placeholder fits the width and the page has no horizontal scroll.
✅ Console errors 0 errors over the session. The only warnings are the existing Radix 'Missing Description for DialogContent' ones.
✅ Graph view still renders 3 react-flow nodes rendered.
✅ Works with no API key or Claude CLI The backend ran with a throwaway HOME and no credentials. All agent activity was simulated through the mocked WebSocket. No real agent was run.
⏭️ UI e2e suite All 15 specs failed at browserType.launch because the expected chromium_headless_shell-1200 binary is missing on this machine, and npx playwright install fetched a different build. Environment problem, not related to the PR.
🧪 Commands run
Command Result
npm --prefix ui run lint 0 errors, 6 warnings (same warnings as on master)
npm --prefix ui run build tsc -b && vite build succeeded
venv/bin/python -m ruff check . All checks passed
venv/bin/python -m mypy . 24 errors in 9 files; no Python files changed, so these are already on master
venv/bin/python -m pytest -q 144 passed, 21 failed; no Python files changed, so these are already on master
HOME=<scratch> uvicorn server.main:app --port 5247; POST /api/projects and /api/projects/qa321/features Server healthy; project and 3 features created
Playwright (MCP) with page.routeWebSocket proxying to the real backend and injecting agent_update / browser_screenshot All browser scenarios passed
VITE_API_PORT=5247 npx playwright test (ui) 15 failed: headless Chromium binary missing (environment)

No browser screenshots were taken in this round.

Posted by cubefarm · merges automatically once GitHub's checks pass

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Launch browser window within UI

1 participant