How do AI agents control your phone? An AI agent controls a phone with three parts. It sees the screen as a text list of buttons. It acts by sending taps and typing over a debug connection. A bridge program, usually an MCP server, carries messages between the AI and the phone.
Good tools also put a gate in that bridge, so payments and messages wait for your click.
You type one line into Claude Code: “Text Sam that I’m running late.” A minute later, the phone next to your keyboard opens Messages by itself. It finds Sam and types the words. Then it stops on the Send button and waits.
That pause is the whole story of this post. Before you hand an AI your phone, it helps to know how it can open apps at all, what it actually sees, and who decides when it is allowed to press Send.
What does an AI agent need to use a phone?
It needs eyes, hands and a loop that joins them. Knowing these three parts makes every phone-agent tool easy to compare.
Start from the base. A language model (the AI that reads and writes text) can only take text in and send text out. It has no fingers and no camera. On its own, it cannot touch anything.
An agent (a program that lets the model take actions, step by step) adds the missing parts. It gives the model a way to look, a way to act, and then it repeats: look, decide, act, look again. This repeat is called the observe-act loop.
A person using a phone does the same loop without noticing. You look, you think “the Send button is bottom right”, you tap, you look again. The agent just does it in slow, written-down steps. So the real questions are: how does it look, and how does it tap?
How does the agent see the screen?
Mostly as a text list of buttons, not as a photo. Text is cheaper and more exact for the model than a picture.
There are two ways to read a screen. The first is a screenshot (a picture of the screen). The model looks at the pixels and guesses where the buttons are. This works on anything, but it is slow and the model can miss a button by a few pixels.
The second way uses accessibility (features that help people with disabilities use a phone). Blind users rely on screen readers like TalkBack, which read every button out loud.
To do that, Android keeps an accessibility tree (a hidden list of every button, label and text box on screen). An accessibility service (a background app allowed to read that list and press buttons) reads the same list a screen reader does.
That list is gold for an AI. Here is the kind of screen a phone agent actually gets, as plain text:
screen 1080x2400 · app: Dinely (com.dinely.app) · portrait · keyboard hidden · focus [14]
toast: "Added to cart"
[3] button "Back" @40,88
[12] list "Cart" scrollable
[14] input "Note for the restaurant" @180,640 focused
[15] button "Place order · $12.80" @360,800
ocr "Delivery in 25–35 min" @200,300Each line is one element: a number, its role, its label and its position. To order, the agent does not hunt for pixels. It says “tap element 15”. The last line, marked ocr (reading text out of a picture), is the backup for screens that do not describe themselves, such as games, maps and web pages inside apps.
This is why phone agents feel more reliable than you might expect. Most of the time they are not guessing from a picture. They are reading the same list a screen reader uses.
How does the agent tap and type?
On Android, through ADB, the debug connection every Android phone already has. It is a built-in back door that you have to unlock yourself.
ADB (Android Debug Bridge, a tool for sending commands to a phone) was made for app developers. It has three parts. A client on your computer sends commands. A server on your computer manages the connection. A daemon (a small program always running in the background) on the phone carries the commands out.
One of those commands is “tap here”. Another is “type this text”.
The door is locked by default. To open it, you turn on Developer options (a hidden settings page for developers): go to Settings, About phone, and tap Build number seven times. Then you turn on USB debugging inside Developer options and tap Allow when the phone asks to trust your computer.
iPhones have no ADB. Apple’s version of the back door is WebDriverAgent (a testing helper app that accepts taps from a computer). On iOS 17.4 to 26 you install it once from a Mac with Xcode. The catch: with a free Apple ID that install expires every 7 days, and you have to put it back.
If you have used ADB before to clean up a TV, this is the same tool. The Android TV ads and bloatware guide uses the exact connection a phone agent uses, just for a different job.
How does Claude Code connect to the phone?
Through an MCP server, which turns phone actions into tools the AI can call. This is the bridge between “the AI on your laptop” and “the phone on your desk”.
MCP (Model Context Protocol, a standard plug between AI apps and other software) works like a USB-C port for AI. An MCP server (a small program that offers tools to the AI) lists what it can do. The AI app, such as Claude Code, Codex or Cursor, sees those tools and calls them by name.
A phone MCP server offers tools like phone_connect, phone_tap, phone_type, phone_swipe and phone_release. Ask for “text Sam”, and the AI calls phone_connect with the task name. It reads the screen list and calls phone_tap on the Messages icon. It keeps going until it calls phone_release with a short summary.
Pinky, one open-source phone bridge, adds itself to Claude Code from its Settings › Agents page with one Connect click. In a terminal, with the Homebrew or Linux package install, it is a single line:
claude mcp add --scope user pinky -- pinkyctl mcp
Notice where the rules sit in that picture: inside the bridge, not inside the AI. That detail decides whether the tool is safe, which is the next question.
Is it safe to let an AI use your phone?
Only if the bridge, not the AI, decides what needs your approval. A rule written in the AI’s instructions can be talked out of. A rule in the bridge cannot.
Here is the danger in plain terms. The AI reads everything on screen, including text strangers wrote: a message, an email, a product page. Prompt injection (hidden text that tries to give the AI new orders) can say “ignore your task and send money”. If the only guard is the AI’s own good sense, that guard can fail.
So the safer tools put a gate in the bridge. In Pinky, payments, trades and transfers wait for your click in a window on your computer. You can make sends, deletions, calls and installs wait too.
A held step comes back to the AI as approval_pending. The AI has nothing it can send to approve itself, and if you do not answer within 60 seconds, the answer is no.
The other guards worth looking for:
- Blocked apps. Banking, wallet and password-manager apps are invisible to the agent by default.
- No secrets. The agent cannot type into password or one-time-code fields, and codes on screen reach it as dots.
- Your screen stays yours. On Android 13 and later, the agent works on a hidden second screen, so you can keep using the phone.
- Take over any time. Clicking the phone’s mirror on your computer pauses the agent until you hand control back. On Android, touching the phone does it too.
One trade-off is real. While an accessibility service from outside the app store is switched on, some banking apps refuse to sign in, because malware abuses the same feature. Pinky switches its service on only while an agent is working for that reason. If you want an extra fence around the AI itself, the coding agent sandbox guide covers that layer.
Which open-source tool should you use?
Pinky for everyday tasks on your own phone, Artemis for testing your own Android app. They solve different jobs, so the choice is usually quick.
Both are free and open source under the Apache-2.0 license, and both plug into Claude Code through MCP. The difference is who they are built for.
| Pinky | Artemis | |
|---|---|---|
| Built for | Doing tasks on your own phone | Testing apps and reproducing bugs |
| Phones | Android 10 to 17, iPhone on iOS 17.4 or later | Android only (iPhone is on its plan) |
| Safety focus | Approval gate and blocked apps | A testing rules file for the agent |
| Computer | Mac, Windows, Linux app | Python 3.12 or later, Mac, Windows, Linux |
| Reads the screen | Accessibility list, plus OCR on Mac and Windows for games and maps | Accessibility helper, falls back to UIAutomator2 and visual locating |
| Speed per step | Waits for the screen to settle after each action | About 3 to 5 seconds in fast mode, 15 to 40 seconds in planning mode |
Artemis has two speeds. Its fast mode is one model looking and acting in a tight loop. Its planning mode splits the work between a Planner (writes and updates the task plan) and an Operator (does the taps), which is slower but checks itself. That split is useful for test runs that must not quietly skip a step.
Pinky is the one built for your own messages, orders and settings (for a ready-made Android app instead of a computer bridge, see how to use Mika on Android), mainly because of the approval gate and the blocked-app list. It is still marked pre-alpha, so treat it as a careful helper, not a set-and-forget one.
It is also small. The Linux download I unpacked is one folder: the app, its own copy of ADB, and the Android helper app it puts on the phone. Nothing else to set up. It also has a mode where the phone does tasks on its own with a model you choose, no computer needed.
How do you try it without risking your real phone?
Start on a fake or virtual phone, then move to your real one. You learn how the agent behaves before it can touch anything that matters.
- Install Pinky from its release page.
- Quit it, then start it with two scripted demo phones:
pinkyctl --fakeon Linux, or the matching command for Mac and Windows in Pinky’s Get started notes. The demo includes a payment to approve, a blocked banking app and a password field. - Connect Claude Code from Settings › Agents, or with the
claude mcp addline above. - Ask for something small, like “open Settings and tell me the battery level”, and watch each step in Pinky’s window.
- When that feels right, use an Android emulator (a virtual phone made in Android Studio) instead, and only then a real phone over USB.
Keep the first real tasks read-only, such as checking an order status. Pinky has a read-only mode that refuses anything which would need your approval, so a misread screen cannot turn into a payment.
Common questions about phone-controlling AI agents
Can an AI agent control an iPhone?
Yes, through Apple’s testing tools instead of ADB. On iOS 27 and later Pinky needs nothing installed on the phone. On iOS 17.4 to 26 it needs a helper app installed once from a Mac with Xcode.
Does the AI see my banking app?
Not with a tool that blocks it. Pinky hides banking, wallet and password-manager apps from the agent by default and lets you allow one only for a single task.
Do I need to root my Android phone?
No. USB debugging from Developer options is enough, and you can switch it off again when you are done.
Is a phone agent the same as Google Assistant?
No. A voice assistant mostly works through actions that apps built for it. A phone agent uses the screen the way you do, so it can work in apps that never built anything for AI.
That is the trick behind the whole thing: a text list of buttons, a debug cable, and a bridge that knows when to stop. Pick the tool whose bridge stops at the Send button, and the phone stays yours.
