Agent Ultra — a local LLM that runs your phone

New in 2.4

Show it a job once. It can do it again with your details.

A stronger model does the task one time. Something outside the app checks the phone to see whether it really worked. If it did, the steps are kept. After that a cheaper model gets the same job with different details — another file name, another time — and the app plays the steps itself.

Scored by someone else

AndroidWorld is Google's public phone benchmark. It sets the task and checks the phone afterwards. 33 of its simpler tasks, picked by rule, with details it had never seen: 8 done without the kept steps, 13 with them. The stronger model that taught it scores 13 and 14.

No retraining

No model was changed. What it learned is a short list of steps in a plain file. On most jobs it has steps for, the model picks nothing — the app does the taps, in 14 to 26 seconds.

Not in your download yet

The memory that keeps the steps lives on my laptop, beside an emulator. The app you download can play steps; nothing on your phone writes them yet. And 20 of those 33 tasks still fail either way.

What is different

Your phone is a drawer full of apps, and you are the glue.

You read a screen here, retype it there, tap through five flows to do one thing you meant. Agent Ultra is the layer that turns those apps into tools. You stop operating screens. You state intent.

The brain is on the handset

Pick a model that fits your phone and it downloads on demand. Nothing is bundled — the app itself is 8.4 MB. With no network it keeps working.

It reads a page properly

Not the first forty labels. It scrolls the whole thing and returns structured items, so a price stays attached to its own product rather than the nearest line.

It stops before it commits

Pay, buy, send, delete, confirm — those pause and ask you, showing the exact control. An unanswered prompt is a refusal, never an approval. In 2.4 the question sits on top of the app you're in, so the app keeps its place while you decide.

Architecture

Four layers, and a rule about trust.

Deliberately shown at the level of what each layer is responsible for. The model is never trusted to be the thing that keeps you safe — a program is.

1 · Perception

Reads the screen through Android's accessibility tree, scrolls a page to its end, and groups what it finds into items rather than a flat list of words.

↓

2 · Reasoning

A model decides the next single step. On-device for offline and quick work; a cloud model when the task is long or ambiguous. The engine — not the model — is responsible for the shape of every tool call.

↓

3 · Policy gate

Every tool call is checked before it executes, against a declared manifest. A tool that is not declared cannot run at all. Arguments carry where they came from, so an action aimed at something you never mentioned is refused and handed back to you.

↓

4 · Action & verification

Taps, typing, scrolling, launching, toggles. Afterwards it re-reads the live screen to confirm the thing it intended actually happened, instead of assuming.

Limits

What it will not do.

This is the part worth reading, and it is not a disclaimer — it is the design.

It only enters apps you tick. Allowlist by default, and it fails closed. Whatever it reads on screen is sent to the cloud model to decide the next step, so “don’t look” matters as much as “don’t act”.
Messages, contacts and location are off by default. Those read Android’s own databases rather than the screen, so the app list does not cover them. They have their own switch, and it starts off.
A tap that commits stops and asks. Both by-name and by-coordinate taps, so the check cannot be sidestepped.
It cannot pass your fingerprint or PIN. Anything behind one ends with the phone in your hand. That is deliberate.
Long flows in unfamiliar apps still trip it. On AndroidWorld it finishes 8 of 33 simpler tasks on its own and 13 with steps it was shown. Camera tasks and calendar grids still fail. So does picking one file out of look-alikes.
Some apps refuse accessibility outright. That is their choice and it is not worked around.

Roadmap

Where this goes.

Working now

On-device model, scored against your phone
Deep page reading with structured items
Policy gate with a confirm channel
Hands-free: hold the side button, talk, hear the answer
Routines you teach by doing them once, then replay
The amount you asked for, checked against the screen before anything commits
Task memory that recalls across rewordings
Risk-based auto-approve — safe, low-risk actions pass without a tap
The confirm question shows on top of the app, so a selection survives it
Reads a screen by its words, and finds a row in a long list by itself
Tell it “no, that's wrong” and it takes back what that run taught it

Structured fields that are labelled, not just ordered
Reliable multi-step flows inside unfamiliar apps
Learning a job from one checked run, on the phone itself — it works on the bench today
Triggers — a routine that fires on place or event
A media toolset: find, sort and tidy your photos

Later

Wake word, without an always-on microphone
Longer horizon tasks that survive interruption
A spend contract: every payment pauses, with a limit you set in advance

Provenance

Built in the open, by one person.

Agent Ultra did not appear from nowhere. Three earlier projects each solved a piece of it, and each is public and checkable.

gatellml — the policy gate
The model is never trusted; the program is. A deterministic enforcement layer with a deny-by-default manifest, measured against a public benchmark before it was ever put on a phone. Research and measurements →
gate — the resolve channel
A gate that can only refuse is half a gate. The research named that gap. It is closed here: a block a human could legitimately cure pauses the run, shows the exact target, and waits for a tap.
Mind Meld — the split
The model owns intent; the engine owns structure. A phone-sized model will not emit clean JSON, and no prompt fixes that. That one division is why a small local model can drive real tools instead of talking about them. Source →
Agent Ultra — 2026
All three, running on a handset. Free to download and use. The source is public, and so is the security research it rests on. Source →

A local LLM that runs your phone.