AI in mobile apps: choosing between on-device and cloud inference
Mobile AI forces a decision desktop and web products can avoid: the device is capable enough to run real models, and sometimes that is clearly right and sometimes it is clearly wrong.
By Quality AboveAll · · 8 min read
- On-device wins for latency, offline use and privacy. Cloud wins for capability, updatability and app size.
- Battery and thermal behaviour are real product constraints, not footnotes, for sustained on-device inference.
- Most shipping apps end up hybrid: small models locally, escalation to the cloud when needed.
What on-device buys you
Latency collapses to milliseconds because there is no network round trip, which changes what interactions are possible. Live camera analysis, real-time suggestions and instant transcription feel different when they are genuinely immediate rather than merely quick.
It also works offline and keeps data on the device. For health, finance and anything involving photographs or audio of people, being able to state truthfully that the data never left the phone is a meaningful product and compliance position rather than a marketing line.
What it costs
Model files inflate app size, which measurably affects install conversion, so bundling several large models is a product decision with a real cost. Downloading models after install mitigates it and introduces its own first-run experience problem.
Sustained inference drains battery and generates heat, and thermal throttling means performance degrades exactly during heavy use. Capability is also capped by the weakest device you support, which is usually far behind the flagship the demo was built on.
The device that runs your demo is not the device your median user has. Test on the oldest phone you claim to support.
What the cloud buys you
Access to models far larger than any phone can hold, the ability to update behaviour without shipping a release or waiting for users to upgrade, and consistent performance regardless of device. For anything involving broad reasoning or long context, this is not a preference, it is a requirement.
The costs are network dependence, per-request expense, and the fact that user data leaves the device, which brings the handling questions in data privacy in AI systems into scope.
The hybrid pattern
In practice most apps run something small locally for the immediate, high-frequency work, then call the cloud for the harder cases. Wake-word detection, image pre-filtering, and quick classification happen on device; complex generation happens on the server.
The design requirement is graceful behaviour when the network is absent: features degrade rather than break, and anything queued syncs later. Deciding this deliberately is what separates an app that works on a train from one that shows a spinner. Model size choices are covered in small language models versus LLMs.
Testing across the fleet
Device fragmentation makes this harder than server-side AI. Performance, memory ceilings and available accelerators vary enormously, so measurement has to happen on real hardware across the range you support, not on simulators.
Track inference time, memory peak, battery draw and thermal state as ordinary product metrics, and set budgets for each. Our mobile testing practice runs these on physical device matrices, because this is exactly the class of problem simulators hide.
Frequently asked questions
Can phones really run useful models?
Modern devices run compact models well, and platform frameworks use dedicated neural hardware. Capability is genuine but bounded, and it drops sharply on older devices.
Does on-device AI mean no privacy policy changes?
It removes a category of transmission, which is significant, but the app may still store or later sync derived data. Describe what actually happens rather than assuming local processing ends the question.
How do we update an on-device model?
Either through an app release or a model download mechanism. The second is faster but adds versioning, storage and rollback concerns you have to design deliberately.
Planning AI in a mobile product and weighing the architecture? A free 30-minute consultation will work through the trade-offs against your device base.
