Skip to content

Commit 7664ad7

Browse files
authored
feat(cli): restore cursor overlay through page.evaluate (#2869)
## Summary Restore Browse's visible cursor as a CLI-owned DOM overlay, without adding a cursor API to core Stagehand V4. - Keep the overlay implementation in one dedicated `cursor-overlay.ts` file. - Install it idempotently for the current document through `page.evaluate(CURSOR_OVERLAY_SCRIPT)` and for future navigations through `page.addInitScript(...)`. - Retry installation on `DOMContentLoaded` when the init script runs before the document root exists. - Keep injection in the top frame and update the marker from coordinate input, including when input lands inside a child frame. - Treat visual position updates as best-effort so they cannot block or invalidate real mouse input. - Preserve the V3 `browse cursor` JSON response: `{ "cursor": "enabled" }`. ## Stack (#2872) 1. #2833 — exact Browse V3 baseline import 2. #2834 — Stagehand V4 runtime and standard command parity 3. **#2869 — CLI-owned cursor overlay** 4. #2849 — CLI-private CDP sidecar; V3 network parity 5. #2835 — remove `--return-xpath`; supported V3 parity/release checkpoint 6. #2838 — eval and packaging integration 7. #2839 — managed Context names (fast-follow) 8. #2701 — shared Functions core consumer (fast-follow) ## Why this is separate The cursor is a self-contained optional visual feature with different review concerns from the combined V4 runtime/command migration: injected DOM/CSS, idempotency, event handling, and screenshot behavior. Keeping it additive on #2834 lets this feature be reviewed or reverted without disturbing browser lifecycle or commands. ## E2E Test Matrix Review-feedback verification compared the exact prior head `6a9d6aa09` with fixed implementation head `1035fbbf5` through the built CLI and real Browserbase browsers. Final head `68f6fcb2c` only expands automated coverage and does not change runtime code. Targets were the public `example.com` and `example.org` pages. | Command / flow | Observed output | Confidence / sufficiency | | --- | --- | --- | | Prior head: enable cursor, alternate 20 cross-origin navigations, inspect `#__browse_cursor_overlay__` before any mouse input | Overlay count was `0` after 20/20 navigations | Reproduces the DOM-readiness bug raised in review | | Fixed head: repeat the same 20-navigation flow | Overlay count was `1` after 20/20 navigations (0 misses) | Proves the `DOMContentLoaded` retry restores the overlay after navigation in the real browser path | | Prior head: replace the page's cursor-position callback with a throwing function, then run `browse mouse click 200 200` against an oversized synthetic button | Command exited `1`; the page's click state remained `null` | Reproduces the visual-update failure blocking real mouse input | | Fixed head: repeat the same forced overlay failure and click | CLI returned `{ "clicked": true }`; page click state became `"yes"` | Proves overlay rendering is best-effort while real input still executes | | Built CLI: `browse cursor` | `{ "cursor": "enabled" }` (prior head returned `{ "enabled": true }`) | Confirms V3-compatible output for existing scripts | | `pnpm --filter browse lint` | Passed formatting, ESLint, and TypeScript checks | Static validation on the final head | | `pnpm --filter browse test:cli` | 26 files / 393 tests passed | Full Browse suite, including DOM readiness, safe styling, idempotency, top-frame isolation, cursor positioning/clamping, and all four coordinate input commands | | `browse stop` after each live run | Completed successfully | Covers Browserbase session and daemon cleanup | The already-uploaded screenshot below remains representative visual proof of the same overlay behavior. ![Browse V4 CLI cursor overlay surviving navigation and pointing inside an iframe](https://raw.githubusercontent.com/browserbase/stagehand/agent/browse-v4-e2e-assets/e2e-proof/browse-v4-cli-cursor-navigation-iframe-20260901.png) No LLM path or customer data was involved.
1 parent aaf421d commit 7664ad7

7 files changed

Lines changed: 390 additions & 14 deletions

File tree

packages/cli/src/lib/driver/commands/mouse.ts

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,7 @@
11
import { z } from "zod";
22

3+
import { updateCursorOverlayPosition } from "../cursor-overlay.js";
4+
import type { DriverPage, DriverSessionManager } from "../session-manager.js";
35
import type { DriverCommandHandlers } from "./types.js";
46

57
const ButtonSchema = z.enum(["left", "right", "middle"]).optional();
@@ -17,6 +19,7 @@ export const mouseHandlers: DriverCommandHandlers = {
1719
.parse(params);
1820
assertXPathUnavailable(returnXPath);
1921
const page = await manager.activePage();
22+
await positionCursorOverlay(manager, page, x, y);
2023
await page.click(x, y, {
2124
...(button === undefined ? {} : { button }),
2225
...(clickCount === undefined ? {} : { clickCount }),
@@ -34,6 +37,7 @@ export const mouseHandlers: DriverCommandHandlers = {
3437
.parse(params);
3538
assertXPathUnavailable(returnXPath);
3639
const page = await manager.activePage();
40+
await positionCursorOverlay(manager, page, x, y);
3741
await page.hover(x, y);
3842
return { hovered: true };
3943
},
@@ -50,6 +54,7 @@ export const mouseHandlers: DriverCommandHandlers = {
5054
.parse(params);
5155
assertXPathUnavailable(returnXPath);
5256
const page = await manager.activePage();
57+
await positionCursorOverlay(manager, page, x, y);
5358
await page.scroll(x, y, deltaX, deltaY);
5459
return { scrolled: true };
5560
},
@@ -69,15 +74,33 @@ export const mouseHandlers: DriverCommandHandlers = {
6974
.parse(params);
7075
assertXPathUnavailable(returnXPath);
7176
const page = await manager.activePage();
77+
await positionCursorOverlay(manager, page, fromX, fromY);
7278
await page.dragAndDrop(fromX, fromY, toX, toY, {
7379
...(button === undefined ? {} : { button }),
7480
...(delay === undefined ? {} : { delay }),
7581
...(steps === undefined ? {} : { steps }),
7682
});
83+
await positionCursorOverlay(manager, page, toX, toY);
7784
return { dragged: true };
7885
},
7986
};
8087

88+
async function positionCursorOverlay(
89+
manager: DriverSessionManager,
90+
page: DriverPage,
91+
x: number,
92+
y: number,
93+
): Promise<void> {
94+
if (!manager.isCursorOverlayEnabled(page)) return;
95+
// The overlay is visual-only. A navigation can destroy its execution
96+
// context, but that must not prevent or invalidate the real mouse action.
97+
try {
98+
await page.evaluate(updateCursorOverlayPosition, { x, y });
99+
} catch {
100+
// Best-effort parity with V3's cursor updates.
101+
}
102+
}
103+
81104
function assertXPathUnavailable(returnXPath: boolean | undefined): void {
82105
if (returnXPath) {
83106
throw new Error("Coordinate XPath lookup is not exposed by Stagehand V4");

packages/cli/src/lib/driver/commands/runtime.ts

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2,8 +2,8 @@ import { promises as fs } from "node:fs";
22

33
import { z } from "zod";
44

5+
import { CURSOR_OVERLAY_SCRIPT } from "../cursor-overlay.js";
56
import type { DriverCommandHandlers } from "./types.js";
6-
import { unavailableCursorOverlay } from "./unavailable.js";
77

88
export const runtimeHandlers: DriverCommandHandlers = {
99
async screenshot(manager, params) {
@@ -89,7 +89,13 @@ export const runtimeHandlers: DriverCommandHandlers = {
8989
return { waited: true };
9090
},
9191

92-
cursor: unavailableCursorOverlay,
92+
async cursor(manager) {
93+
const page = await manager.activePage();
94+
await page.addInitScript(CURSOR_OVERLAY_SCRIPT);
95+
await page.evaluate(CURSOR_OVERLAY_SCRIPT);
96+
manager.markCursorOverlayEnabled(page);
97+
return { cursor: "enabled" };
98+
},
9399
};
94100

95101
function parseTimeoutMs(value: string | undefined): number {

packages/cli/src/lib/driver/commands/unavailable.ts

Lines changed: 0 additions & 9 deletions
This file was deleted.
Lines changed: 80 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,80 @@
1+
export const CURSOR_OVERLAY_SCRIPT = `(() => {
2+
if (globalThis !== globalThis.top) return;
3+
4+
const cursorId = "__browse_cursor_overlay__";
5+
const ensureCursor = () => {
6+
const existing = document.getElementById(cursorId);
7+
if (existing instanceof HTMLDivElement) return existing;
8+
9+
const root = document.documentElement || document.body;
10+
if (!root) return null;
11+
12+
const cursor = document.createElement("div");
13+
cursor.id = cursorId;
14+
cursor.setAttribute("aria-hidden", "true");
15+
Object.assign(cursor.style, {
16+
contain: "layout style paint",
17+
height: "24px",
18+
left: "0px",
19+
mixBlendMode: "normal",
20+
pointerEvents: "none",
21+
position: "fixed",
22+
top: "0px",
23+
userSelect: "none",
24+
width: "16px",
25+
willChange: "left,top",
26+
zIndex: "2147483647",
27+
});
28+
cursor.innerHTML =
29+
'<svg xmlns="http://www.w3.org/2000/svg" width="16" height="24" viewBox="0 0 16 24"><path d="M1 0 L1 22 L6 14 L15 14 Z" fill="black" stroke="white" stroke-width="0.7"/></svg>';
30+
root.appendChild(cursor);
31+
return cursor;
32+
};
33+
34+
const moveCursor = (x, y) => {
35+
const cursor = ensureCursor();
36+
if (!cursor) return;
37+
cursor.style.left = Math.max(0, x) + "px";
38+
cursor.style.top = Math.max(0, y) + "px";
39+
};
40+
41+
const installCursor = () => {
42+
if (ensureCursor()) return;
43+
if (globalThis.__browseCursorOverlayDomReadyListenerInstalled__) return;
44+
45+
document.addEventListener(
46+
"DOMContentLoaded",
47+
() => {
48+
globalThis.__browseCursorOverlayDomReadyListenerInstalled__ = false;
49+
ensureCursor();
50+
},
51+
{ once: true },
52+
);
53+
globalThis.__browseCursorOverlayDomReadyListenerInstalled__ = true;
54+
};
55+
56+
globalThis.__browseMoveCursorOverlay__ = moveCursor;
57+
installCursor();
58+
if (!globalThis.__browseCursorOverlayListenerInstalled__) {
59+
document.addEventListener(
60+
"mousemove",
61+
(event) => {
62+
moveCursor(event.clientX, event.clientY);
63+
},
64+
{ capture: true },
65+
);
66+
globalThis.__browseCursorOverlayListenerInstalled__ = true;
67+
}
68+
})()`;
69+
70+
export function updateCursorOverlayPosition(position: {
71+
x: number;
72+
y: number;
73+
}): void {
74+
const moveCursor = (
75+
globalThis as typeof globalThis & {
76+
__browseMoveCursorOverlay__?: (x: number, y: number) => void;
77+
}
78+
).__browseMoveCursorOverlay__;
79+
moveCursor?.(position.x, position.y);
80+
}

packages/cli/src/lib/driver/session-manager.ts

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -80,6 +80,7 @@ export class DriverSessionManager {
8080
private browserbaseIdentityValue: BrowserbaseIdentity = {};
8181
private consecutiveInitFailures = 0;
8282
private context: DriverContext | null = null;
83+
private cursorOverlayPageIds = new Set<string>();
8384
private lastForwardedEnvSignature: string | null = null;
8485
private pendingEnv: ForwardedEnv | undefined;
8586
private initFailure: InitFailure | null = null;
@@ -207,6 +208,7 @@ export class DriverSessionManager {
207208
this.stagehand = null;
208209
this.browser = null;
209210
this.context = null;
211+
this.cursorOverlayPageIds.clear();
210212
this.browserbaseIdentityValue = {};
211213
this.initFailure = null;
212214
this.consecutiveInitFailures = 0;
@@ -223,6 +225,14 @@ export class DriverSessionManager {
223225
return resolveCachedSelector(selector, this.refMaps);
224226
}
225227

228+
markCursorOverlayEnabled(page: DriverPage): void {
229+
this.cursorOverlayPageIds.add(page.pageId);
230+
}
231+
232+
isCursorOverlayEnabled(page: DriverPage): boolean {
233+
return this.cursorOverlayPageIds.has(page.pageId);
234+
}
235+
226236
setRefMaps(refMaps: RefMaps): void {
227237
this.refMaps = refMaps;
228238
}
Lines changed: 123 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,123 @@
1+
import { createContext, runInContext } from "node:vm";
2+
3+
import { describe, expect, it, vi } from "vitest";
4+
5+
import { CURSOR_OVERLAY_SCRIPT } from "../src/lib/driver/cursor-overlay.js";
6+
7+
describe("cursor overlay", () => {
8+
it("installs after DOMContentLoaded when the document root is not ready", () => {
9+
const harness = createCursorHarness({ ready: false });
10+
11+
harness.install();
12+
13+
expect(harness.elements.size).toBe(0);
14+
expect(harness.listeners.has("DOMContentLoaded")).toBe(true);
15+
16+
harness.makeDocumentReady();
17+
harness.listeners.get("DOMContentLoaded")!();
18+
19+
expect(harness.cursor()).toBeInstanceOf(FakeDiv);
20+
});
21+
22+
it("installs a click-through cursor once in an already-ready document", () => {
23+
const harness = createCursorHarness();
24+
25+
harness.install();
26+
27+
expect(harness.cursor()?.style).toMatchObject({
28+
left: "0px",
29+
pointerEvents: "none",
30+
position: "fixed",
31+
top: "0px",
32+
zIndex: "2147483647",
33+
});
34+
expect(harness.listeners.has("DOMContentLoaded")).toBe(false);
35+
expect(harness.listeners.has("mousemove")).toBe(true);
36+
37+
harness.install();
38+
39+
expect(harness.document.createElement).toHaveBeenCalledOnce();
40+
expect(harness.document.addEventListener).toHaveBeenCalledOnce();
41+
expect(harness.elements.size).toBe(1);
42+
});
43+
44+
it("moves and clamps the cursor from top-document mouse events", () => {
45+
const harness = createCursorHarness();
46+
harness.install();
47+
48+
const mousemove = harness.listeners.get("mousemove")!;
49+
mousemove({ clientX: -25, clientY: 80 });
50+
expect(harness.cursor()?.style).toMatchObject({
51+
left: "0px",
52+
top: "80px",
53+
});
54+
55+
mousemove({ clientX: 140, clientY: -10 });
56+
expect(harness.cursor()?.style).toMatchObject({
57+
left: "140px",
58+
top: "0px",
59+
});
60+
});
61+
62+
it("does not install inside a child frame", () => {
63+
const harness = createCursorHarness({ topFrame: false });
64+
65+
harness.install();
66+
67+
expect(harness.elements.size).toBe(0);
68+
expect(harness.document.createElement).not.toHaveBeenCalled();
69+
expect(harness.document.addEventListener).not.toHaveBeenCalled();
70+
});
71+
});
72+
73+
class FakeDiv {
74+
id = "";
75+
innerHTML = "";
76+
style: Record<string, string> = {};
77+
78+
setAttribute(): void {}
79+
}
80+
81+
type CursorEvent = { clientX: number; clientY: number };
82+
type CursorListener = (event?: CursorEvent) => void;
83+
84+
function createCursorHarness(
85+
options: { ready?: boolean; topFrame?: boolean } = {},
86+
) {
87+
const elements = new Map<string, FakeDiv>();
88+
const listeners = new Map<string, CursorListener>();
89+
const root = {
90+
appendChild(element: FakeDiv) {
91+
elements.set(element.id, element);
92+
},
93+
};
94+
let documentElement: typeof root | null =
95+
options.ready === false ? null : root;
96+
const document = {
97+
addEventListener: vi.fn((name: string, listener: CursorListener) => {
98+
listeners.set(name, listener);
99+
}),
100+
body: null,
101+
createElement: vi.fn(() => new FakeDiv()),
102+
get documentElement() {
103+
return documentElement;
104+
},
105+
getElementById: vi.fn((id: string) => elements.get(id) ?? null),
106+
};
107+
const context = createContext({ document, HTMLDivElement: FakeDiv });
108+
runInContext(
109+
`globalThis.top = ${options.topFrame === false ? "{}" : "globalThis"}`,
110+
context,
111+
);
112+
113+
return {
114+
cursor: () => elements.get("__browse_cursor_overlay__"),
115+
document,
116+
elements,
117+
install: () => runInContext(CURSOR_OVERLAY_SCRIPT, context),
118+
listeners,
119+
makeDocumentReady: () => {
120+
documentElement = root;
121+
},
122+
};
123+
}

0 commit comments

Comments
 (0)