cell-use

On-device iPhone automation for AI agents.

cell-use is an open-source Swift SDK for building agents that control iOS apps from the iPhone itself. Capture screenshots, tap, swipe and type through a common API. Connect your own model to decide what happens next.

Documentation · Swift API · Embed in your app · Releases

iPhone control from your app

The phone runs the screenshot and input loop. Each step gives your agent a fresh screenshot; the agent returns an action, and cell-use delivers it to the active app. Use it to build mobile assistants, prototype iOS workflows or add phone control to an existing Swift agent.

• Screen capture: PNG screenshots for your agent's vision model.
• Phone actions: tap, swipe, type text, wait and finish.
• Multi-step execution: run actions in order, with screenshots between steps.
• Model integration: implement PhoneAgent with your own decision logic.
• Embedded connection: bundle the local tunnel inside your app so users do not need a separate LocalDevVPN installation.

The native runtime uses Developer Mode, pairing, prepared developer services and a local tunnel. Follow iPhone setup for the demo or app integration for your own host.

Swift packages

Component	Purpose	Installation
CellUse	Agent API, actions and execution loop	Swift Package Manager
CellUseRuntime	iPhone screenshot and input adapter	Clone and build with native dependencies
CellUseTunnel	Embedded local connection for your own app	SwiftPM product + packet-tunnel extension target
cell-use demo	Reference app for pairing, background tasks and device experiments	Sign the release IPA, or build with Xcode

Use the portable SDK to implement your agent, and the native runtime to connect it to an iPhone. The reference app supplies the session and background-task setup.

Building your own app? Embed the tunnel to ship the connection inside your app, with no separate LocalDevVPN installation. The publishing developer signs the app and extension; users need no paid developer account. See the host example for app/extension packaging and signing.

Install with Swift Package Manager

Add https://github.com/Raynan00/cell-use in Xcode's package dependencies, selecting the exact version 0.1.0-alpha.2 and the CellUse product. Or use SwiftPM:

dependencies: [
    .package(url: "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/Raynan00/cell-use.git", exact: "0.1.0-alpha.2")
]
// In your target's dependencies:
// .product(name: "CellUse", package: "cell-use")

Implement PhoneAgent to choose actions:

import CellUse

struct MyAgent: PhoneAgent {
    let choose: @Sendable (PhoneObservation) async throws -> PhoneAction

    func nextAction(for observation: PhoneObservation) async throws -> PhoneDecision {
        let action = try await choose(observation)
        return PhoneDecision(observation: observation, action: action)
    }
}

Pass observation.screenshotPNG to your model and return an action. The runner handles action order, screenshot timing, cancellation and command results.

Actions: tap · swipe · typeText · wait · finish.

Read the API, native integration and iPhone setup.

Try the iPhone demo

1. Follow setup to prepare developer services, sign the app, enable Developer Mode, start LocalDevVPN and pair the phone.
2. Use Connect and Arm swipe test with Settings, or Arm text test with an empty Notes draft. Switch to the target app and leave it untouched.
3. Return to cell-use and inspect the final image and run report.

The included scripted clients make it easy to try the action loop before connecting your own model. The demo guide also covers ordered Calculator taps (0 → 7 → 77) and background inputs after 60 seconds.

The full-control demo IPA is available in 0.1.0-alpha.1. The SDK and embedded tunnel templates are available in 0.1.0-alpha.2.

Common questions

Does cell-use need a computer while an agent runs? The native control loop runs on the iPhone. Initial signing and developer-service preparation are described in the setup guide.

Can I use an on-device model or a cloud model? Yes. Implement PhoneAgent with either. Your provider determines where inference runs and whether screenshots leave the phone.

Is cell-use an app or a library? It is a developer library with an iPhone runtime, an optional embedded tunnel and example apps. You build and distribute your own app around it.

Which iOS version does it target? The current native phone-control runtime requires iOS 27. The agent API and embedded tunnel are separate components with an iOS 17 deployment minimum:

Component	iOS minimum	What it provides
CellUse	17	Agent interface, actions and execution loop
CellUseTunnel	17	Local network route for the native connection
CellUseRuntime and full-control demo	27	Screenshots and input through the pinned DeviceHub transport

For an app using the supplied phone-control runtime, use iOS 27 as the minimum. The iOS 17 package declarations alone do not provide cross-app control on iOS 17. See version requirements and initial setup versus running an agent.

Read the FAQ for integration, signing and model setup.

Build and contribute

git clone https://github.com/Raynan00/cell-use.git
cd cell-use
swift test                         # portable SDK; Swift 6+
swift test --package-path Runtime  # core on Linux; native tests after bootstrap on Mac

For the iPhone app, use macOS with Xcode 27, Python 3, Rust/rustup, XcodeGen and ripgrep, then run bash Scripts/build-macos.sh. The script prepares pinned native dependencies, builds the unsigned app and runs the SDK and native integration suites. Windows users can sign a release IPA.

See contributing and release procedure.

License and credits

Created by Raynan Wuyep. Original cell-use code is licensed under Apache-2.0.

The native transport uses DeviceHub iOS and its patched idevice dependency under MIT licenses. See third-party notices.