"On-device AI" has become a marketing phrase, which means it's started to mean everything and nothing. This post is a plain-language tour of what's actually inside an app like our Offline AI Notebook when it does AI on your iPhone. No hype, just the stack.
The hardware: a Neural Engine in your pocket
Every iPhone since the A11 Bionic (2017) ships with a Neural Engine: dedicated silicon designed for the specific math neural networks need: matrix multiplications, enormous numbers of them, done in parallel. Recent chips run tens of trillions of these operations per second.
This is the reason on-device AI went from a demo to a product in the last few years. The hardware was always there; the models had to shrink to fit it.
The models: distilled, quantized, and still capable
A frontier model like the ones behind major chatbots needs data-center GPUs. But there is a whole family of open-weight models: Qwen, Llama, Phi, Gemma. That are trained to be small enough to run on consumer hardware while staying surprisingly capable. Developers then apply tricks:
- Quantization: storing weights at 4 or 8 bits instead of 16 or 32. Smaller, slightly less precise, usually fine.
- Distillation: training a small model to imitate a big one.
- Architecture tuning: attention variants and layer designs that trade a little capability for a lot of speed.
The result: a model of a few billion parameters that fits in a few gigabytes and answers questions fluently on hardware that runs on a battery.
Where Apple Intelligence fits
On newer iPhones, Apple exposes its own on-device foundation model through the Foundation Models framework, and our apps use it when available. On older devices, the app falls back to downloading an open-weights model. In our case Qwen 3. With your explicit consent.
Either way, the defining property is the same: the weights are on your phone, the inference happens in your phone's memory, and the output never transits a network.
The plumbing around the model
A chat model alone doesn't make a notebook app. A complete on-device AI system looks more like this:
- Embeddings: a small model that converts text chunks into numeric vectors so the app can find the passages relevant to your question. Ours run on-device.
- Retrieval: searching your documents by meaning, not just keywords.
- Speech transcription: converting your recordings to text with an on-device open speech model, which is how the notebook transcribes audio even with the phone locked.
- OCR: Apple's Vision framework reading scanned PDFs, entirely locally.
Each piece is its own model download, each gated behind explicit consent, each running offline once installed.
The honest trade-offs
On-device AI is not better at everything:
- Smaller models know less. They're genuinely smart, but a 3-billion-parameter model won't match a frontier model on obscure trivia or long, multi-step reasoning.
- Your phone does the work. Sustained generation warms the device and uses battery.
- Downloads are real. A capable model is a couple of gigabytes.
What you get in exchange is architectural: there is no server that can be breached, no provider that can change terms, no log that can be subpoenaed. When we say the notebook's answers cite your actual passages: no hallucinated sources. That's a property of deterministic retrieval on your own documents, not a promise about a cloud pipeline.
Privacy compliance by construction beats privacy compliance by contract. That's the whole idea.