Android phones: see and control real phones over adb

[danielehrhardt]

A new machine type next to VMs and SSH servers: Android phones. Godmode finds real phones (and emulators) connected to this computer over adb, by USB cable or Wi-Fi, and agents can fully control the ones they are given. The human watches every phone live and can take over at any time.

What's in it

Phones page (Machines → Android phones, /android)

• Every phone adb sees, with a live thumbnail, model, Android version, USB / Wi-Fi / emulator, battery, and which agents and chats use it.
• Waiting states are explained in place: "Allow USB debugging" on the phone, offline, no udev rule on Linux.
• No adb on the machine: one click installs Google's platform-tools r37.0.1 into Godmode's data folder (pinned SHA-256 per platform). An existing adb (Android SDK, PATH, Homebrew) is preferred, so Godmode shares the human's adb server instead of fighting over port 5037.
• Connect over Wi-Fi: pair with the six-digit code from Wireless debugging (Android 11+), or reconnect a paired phone.
• A phone keeps its record, name and description across USB/Wi-Fi and replugging (keyed by its own serial; emulators by AVD name). Forget only works while it is unplugged.

Remote control (/android/:id)

• Live screen as H.264: screenrecord on the phone, cut into frames on the host, decoded with WebCodecs in the app (works in WebKit, so in the Tauri webview). Still pictures as a fallback. A new viewer, a dropped frame or a rotation restarts the stream for a keyframe.
• Click = tap, drag = swipe, hold = long press, wheel = swipe, right click / Esc = back, middle click = home, typing and paste go to the focused field.
• Hardware buttons (power, volume, mute, wake, sleep, notifications, quick settings), Back / Home / Recents, a typing box, an adb shell box, assignments.

Agents

• New android MCP server for runs with phones (the chat's and the agent's): screenshot, tap, long_press, swipe, scroll, type_text, press_key, ui_elements (texts, buttons and fields with the coordinates to tap, from uiautomator), open_app, list_apps, shell, install_apk, push_file, pull_file, list_devices. Coordinates are pixels of the run's latest screenshot.
• fill_login / fill_totp when the human turns on "Logins and 2FA codes" for phones (vault passphrase, off by default), and only for runs that may reveal secrets: whatever is typed on a phone can be read through its shell, so a fill there is as good as showing the value. Fill-only agents ask the human to sign in on the phone instead. Passwords only go into fields the phone reports as password fields, values go to adb on stdin (not in process arguments on the computer), and anything Godmode typed is masked in later tool output.
• System prompt section with the rules (look, act, look again; never bypass a lock screen; no buying, posting, deleting or settings changes unless asked).
• Manager tools android_devices_list and android_assign (gives a phone to another agent; never to the caller or its own chat). Agents can't set phones through the normal agent API either.
• Chats show the phone live in a side panel while the agent works (with "Take control" in place), the composer and agent form get a phone picker.

Settings → Android phones: on/off, logins and 2FA codes, adb status and install, custom adb path. Android settings and agents' phone assignments are local to each computer (not synced to runners).

Tested

• test/android.test.ts (19 tests, fake adb): parsing, discovery and identity across USB/Wi-Fi, assignments, the human's command, every tool path incl. coordinate mapping, gesture-safe scrolling, open_app, fills on password vs. normal fields and masking, prompt, reveal-only fills via stdin, manager self-assignment refused, letterboxed input mapping, folder pulls.
• Related suites: agents, conversations, MCP, SSH, VMs, backup, runner snapshot/mirror, connect (289 pass). pnpm -r typecheck clean.
• Godmode Cloud: owners and operators can watch and control phones like VMs; phone screenshots stay hidden from viewers; Android settings stay on the computer.
• Real emulator (Android 15) through the app in WebKit: live video, swipe opened the app drawer, typed "settings" + Enter opened Settings, right click went back, Home button.
• A real agent run (Opus 5.5) on the emulator: opened Settings → About, read Android 15 and the model, returned home (80 s). It surfaced two bugs that are fixed here: monkey aborts on devices without hardware keys (now the launcher activity is resolved and started with am start), and full-length scrolls started in the gesture zone and sent the phone home (swipes now stay clear of the edges).

Not in this PR

• Phone app (Expo) screens for Android phones.
• Non-ASCII typing (input text can't type ä/é/emoji; tools and UI say so).