Engineering · Agents · Mobile AI
Giving an agent hands: driving a real Android phone
How the mobile-use plugin turns OpenCode into an operator — wireless adb for the transport, an accessibility tree for eyes, deep links for navigation, dumpsys for truth, and the one-pixel discipline that keeps a 14-tool surface honest.
Eyes first: read the tree, not the pixels
The agent's only reliable eyes are the accessibility dump. phone_dump_ui returns the screen as a flat list of nodes — each with its text, content-description, and exact pixel bounds. Coordinates are never estimated. To tap a row you tap the center of its bounds, straight from the tree.
That single decision gives the whole loop its shape: one pixel space. The dump reports bounds, the tap uses them, the screenshot confirms them. There is no “approximately here” anywhere in the pipeline.
One pixel space. The dump gives you bounds; the tap uses them; the screenshot confirms them. No estimation anywhere.
$ phone_dump_ui Focused window: com.spotify.music/com.spotify.music.MainActivity #16 [45,289,250,351] text="greedy" TextView #17 [45,295,90,340] desc="Paused" ViewGroup #68 [192,2010,533,2061] text="greedy • Tate McRae" TextView #73 [921,2005,1034,2118] desc="Play" FrameLayout
When the tree errors — canvas screens
Some screens come back sparse or not at all: a video, a Flutter app, the home feeds of Spotify or X. An erroring dump is a reading-method failure, not a navigation failure — switch to a screenshot and a vision model. The inverse rule holds too: a screenshot is only worth taking if someone reads it, and small icon state (a repeat “1” badge) is illegible downscaled — capture full-res, or verify behaviorally instead of trusting one icon read.
Deep links over taps — navigate by intent
Tap-walking a menu tree is the slow, flaky fallback. The primary move is an intent: am start -a android.intent.action.VIEW -d "spotify:track:<id>" opens the app at the exact track and starts playback. Search is spotify:search:<query>, a chat is whatsapp://chat, a call is am start -a android.intent.action.CALL.
The IDs themselves are a small knowledge base. The skill docs carry the canonical entry for common asks — greedy is spotify:track:6wCv0m87EJ6kYGUi5c0kbG — so the agent never re-resolves and risks landing on a remix or a duplicate. When it must resolve fresh, it verifies first: cmd package resolve-activitymust return the target app's activity, not ResolverActivity, before the link is fired.
$ adb shell cmd package resolve-activity --brief \
-a android.intent.action.VIEW -d "spotify:track:6wCv0m87EJ6kYGUi5c0kbG"
com.spotify.music/com.spotify.music.MainActivity # not ResolverActivity
$ adb shell am start -a android.intent.action.VIEW \
-d "spotify:track:6wCv0m87EJ6kYGUi5c0kbG" com.spotify.music
Starting: Intent { act=android.intent.action.VIEW dat=spotify: pkg=com.spotify.music }The pause is yours
Every screen capture — dump or screenshot — pauses media while it runs. So the moment you “verify” playback by capturing the screen, you read Paused, and you have invented a failure.
The rule: verify in dumpsys, never in pixels. Read dumpsys media_session --active with no capture in between. If the first read says Paused, sample a second time. Only when two clean samples both read Paused do you touch anything.
In the session: the greedy deep link auto-played, and our first verification read Paused at position 2294 ms — our own artifact, from the dump we had just run. One clean re-sample, PLAYING, done. This is the Zygote lesson in another skin: measure, don't assume — because here the measurement itself changes the thing being measured.
$ dumpsys media_session --active | grep state=PlaybackState
state=PlaybackState {state=PAUSED(2), position=2294, speed=0.0, ...}
# sample again, NO capture in between —
$ dumpsys media_session --active | grep state=PlaybackState
state=PlaybackState {state=PLAYING(3), position=3209, speed=1.0, ...}Look, act, verify
A step is done when the screen shows the expected change. Clear a field, dismiss a dialog, switch a camera — verify each one, even the small ones, or you act on stale state and pay twice. The loop is cheap precisely because every iteration is one small, verified step.
After any launch, confirm focus first: dumpsys window | grep mCurrentFocus. A cached dump from the previous app is a trap — never tap from it.
And the hard rule: if the screen is unchanged after an action, do not repeat the action. Re-check focus, re-dump, or read logcat and reconsider. The expensive failure is not a missed tap — it is a confident retry against stale state.
When there's no API, walk the UI
Some things have no shell command. Disconnecting a Bluetooth earbud on Samsung exposes no disconnectsubcommand — so we walked the settings UI: list the paired device, tap its gear, tap Disconnect, and confirm the button flips to “Connect”. Two dumps and a tap.
The same discipline covers ambiguous state. Front versus rear camera cannot be read from the viewfinder — a front camera pointed at a ceiling looks identical to a rear shot, and vision models misjudge it. The dump's button label is authoritative: “Switch to rear camera” means the front camera is active. Flip, re-dump, confirm the description flipped. Take the photo, pull it to the laptop, done.
And when the world is genuinely ambiguous, ask. The contacts database returned two “Guhan Kannan” entries with different numbers; the rule is to surface the choice to the user rather than guess. We did. No harm done, no wrong number.
Keep the link alive
The wire is the first thing to break. Samsung kills wireless adb when the display sleeps — the link dies mid-command, which is why long commands fail with no output at all.
So it gets front-loaded at the start of any multi-step session: stay awake, never idle, never sleep. Long waits happen host-side — never sleep more than ten seconds on-device. If the link does die, reconnect and re-wake with the 224+82 pair; on Samsung, WAKEUP alone can blank right back.
$ adb shell svc power stayon true $ adb shell dumpsys deviceidle disable $ adb shell settings put system screen_off_timeout 2147483647 # if the link dies — re-wake on Samsung: $ adb shell input keyevent 224 # WAKEUP $ adb shell input keyevent 82 # MENU (WAKEUP alone can blank right back)
What it adds up to
| Layer | Naive | mobile-use |
|---|---|---|
| Eyes | screenshots + guesswork | a11y dump, exact bounds |
| Coordinates | estimate from the photo | one shared pixel space |
| Navigation | tap-walk menus | deep links / intents |
| Truth | trust the screen | dumpsys, never pixels |
| Steps | assume it landed | verify every step |
| Link | dies on display sleep | stayon + deviceidle off |
None of this is clever. Each rule is a constraint discovered the hard way and encoded so the next session doesn't rediscover it — the capture-induced pause, the sleeping adb link, the vision model that can't tell a front camera from a rear one.
The result is an agent with hands. In one session it opened Spotify and played a track, disconnected an earbud, flipped to the front camera and took a selfie, pulled it to a laptop, looked up a contact, placed a call, answered one, and hung up — all on the same $150 Galaxy M17 that runs Zygote's model.
The transport is a handful of adb commands; the app skills on top — spotify, x, whatsapp, camera, phone-call, github — turn that raw surface into tasks an agent can run end to end. It is the physical layer of the agent economy, and the whole plugin — the phone_* tool surface, the app skills, and the field rules this post is built from — is open source.