← back to demo
how it works
the loop
goal
↓
plan a reasoning model → ordered steps (host / cloud)
↓
┌ per step ─────────────────────────────
perceive accessibility tree + screenshot (off-runner)
ground target → exact pixel
· AX match — most elements, instant + free
· fine-tuned VLM (UI-TARS) — the rest
on-device (MLX) or cloud GPU
act actuation cascade → HID / runner / BLE / voice
verify did the screen change? → repair if not
└───────────────────────────────────────
↓
done
Two models, not one: a reasoning model plans — it's large, and runs on a host or in the cloud — while a small fine-tuned grounding model finds the exact pixel, on-device via MLX for speed and privacy, or a cloud GPU when needed. Grounding is accessibility-first: the AX tree gives exact coordinates for most elements for free; the VLM only handles the custom-drawn controls a generic model can't see.
No jailbreak, no Mac required, no iPhone modifications. The agent runs on a host (Mac / Linux / cloud); the iPhone is a stock production device.
the actuation cascade
Most iOS automation has exactly one way to touch the screen — the XCUITest runner — and when it dies under memory pressure (which it does, constantly, on heavy apps) the run stalls. clank has four independent channels with automatic failover.
01 direct HID injection — OS-level taps & keystrokes, no runner, no hardware ← default
02 XCUITest runner — privileged, element-targeted automation
03 BLE HID — external controller, independent of the host link
04 Voice Control — entitlement-free, works on the lock screen
auto-failover: lose a channel to an OS update, the next takes over.
The default path drives the device's own input system directly from the host — no test runner, no hardware in the loop — so it survives the crashes that flatten every runner-based tool. If that path ever closes, the agent degrades to the runner, then to external hardware, then to voice. It doesn't fail; it gets slower.
underneath
The loop above is the happy path. What makes it fast and reliable:
- Caching + skill packs — a known flow replays from cache in seconds with zero model calls; learned cross-app skills are reused instead of re-derived every time.
- Memory — the agent remembers what worked (and what didn't) per app and reconciles it over time.
- Self-healing — every action is verified against the resulting screen; on a miss it repairs instead of failing, and jetsam / wedges / runner deaths recover automatically.
- A trajectory flywheel — every run becomes training data, so the grounding model gets better at iOS with use.
what's verifiable from outside
Numbers are p50 ranges on physical iPhones over the local network. Simulator is faster on every axis.
what's real in the demo
Everything except the live voice commands. Real iPhone 15 Pro, real iOS 26.4, real App Store apps, real ClankDriver taps and swipes, single-take recording. The Dynamic Island is rendered by SpringBoard — iOS, not the app — so it can't be composited in post.
The voice was for presentation. At the start of the video I fed the agent the same list of tasks as a typed prompt; the spoken voiceover is just so a viewer can follow along. Replace my voice with a chat input, an API call, or a webhook payload and the rest runs identically.
what clank can do today
- Drive any iOS app on a stock iPhone or iPad — no SDK, no developer integration, no App-Intent registration.
- Single natural-language prompt → multi-step execution across apps (e.g., "find a Thai place near me, text Mark the address" → search, switch app, compose, send).
- Cross-app workflows: Messages → Photos → Mail → Safari.
- Record-and-replay with self-heal when the UI shifts — scripts survive UI churn instead of failing.
- Multi-device fleets: same task run in parallel across many phones (10+ devices in CI).
- Recovery from unexpected state — modals, permissions, auth prompts, system alerts.
- Apps without active anti-debug protections.
- Reachable over WiFi or a tunnel — host can be a local Mac, Linux box, or cloud.
what we're still validating
what's under NDA
The iOS-internals stack, the training pipeline + trace corpus, the synthetic-iOS substrate (ClankGym) and its sim2real fidelity measurements, the tether path and signing infrastructure, the runtime that closes the latency gap to milliseconds, and the Y2 hardware + multi-device-class roadmap.