Edge & Mobile Deployment
Run where the user is: phones, browsers, sensors
Edge models run on phones, browsers and devices: small, quantized, offline-capable. Trade-offs: per-device resources, model update logistics, privacy — and the tools: TFLite/ONNX Runtime/WebGL/WASM, INT8 quantization and on-device eval.
What you will learn
- Choose edge runtimes (TFLite/ONNX/Web) and quantization
- Design update loops (bundled vs remote model delivery)
- Evaluate edge quality/latency on-device, not in the cloud
What is this?
Edge deployment shrinks the model to device constraints: small enough to ship and load fast, quantized for CPU/NPU, and able to run offline. Runtimes: TensorFlow Lite/CoreML for mobile, ONNX Runtime/WebAssembly on the web (sometimes NPU APIs). Updates flow through app bundles or remote model delivery with device-side versioning. The wins are privacy (data stays local), latency (no round-trip) and cost (zero serving infra).
Why do you need it?
- Privacy/compliance: sensitive inferences never leave the device.
- Latency: no network hop — camera/typing-input decisions are instant.
- Cost: the fleet's compute is free vs dedicated GPU serving.
- Offline robustness: works in planes, tunnels, remote sites.
Mental model
A pocket calculator vs a mainframe
Edge vs cloud inference
The calculator handles 99% of daily math instantly, no cable, no electricity bill — but can't run a statistical bootstrap. The mainframe can do anything but needs the desk, the plug and the user's ID. Edge = ship the calculator that covers the common cases; route exotic work to the cloud.
How it works
- Choose runtime by platform: TFLite (Android/mobile), CoreML (iOS), ONNX/WASM (web), vendor NPUs.
- Quantize: INT8 4-bit-aware distillation to fit memory and speed (precision band checked on-device).
- Bundle or deliver: ship the model in the app or push model updates via a remote-config endpoint.
- Device eval: benchmark load time, first-inference latency and accuracy on REAL devices (emulators lie).
- Version and rollback: device-side model version + kill-switch for remote-pushed models.
Ship small, run local, update remotely.
In production
- Mobile keyboards/autocorrect: on-device, instant, offline.
- Browsers: WASM machine translation/OCR with no server.
- IoT/cameras: edge anomaly detection with a cloud escape hatch.
Best practices
- Set a size + first-load latency budget and enforce it in CI for every model rev.
- Quantize AND validate on-device (CPU/INT8 differences show up on real hardware).
- Design update enrollment: phased rollout with device-side version tracking.
- Keep a kill-switch/hot-rollback path for remotely-delivered models.
Common mistakes
- Skipping device testing — emulator/cloud benchmarks and phone behavior diverge.
- INT8 without on-device quality check — the accuracy cliff is device-dependent.
- Blocking startup on model download; offline-first means a bundled default.
- No device version reporting → you can't see fleet model drift.
Common issues
Model loads but predicts slowly on low-end phonesFP32/unoptimized kernel path
INT8 quantize + target the runtime's optimized ops (check op-palette coverage).
Update rollout degrades a subset of devicesFirmware/NPU mismatch in that cohort
Rollback the remote model for the cohort; gate rollout by device capability flags.
Device offline but 'no model' errorBundled default missing
Always bundle a baseline model; remote update improves, never gates, cold-start.
Debugging tips
- Profile load-time vs inference-time separately; these need different fixes.
- Time first inference after install (cold) vs after warm-up — cold is what users feel.
- Cross-check CPU vs NPU: a model that needs the NPU is fragile across devices.
- Compare remote-pushed model quality across a versioned device cohort.
Interview checkpoint
Quick quiz
The primary edge-deployment accuracy risk comes from:
Small bundled models let edge features work offline from install.