Edge Lab: Building a Mobile AI Instrument
From desktop benchmarks to an open iOS app — deploying Gemma 4 on iPhone with four presets, versioned JSON manifests, and no cloud.
The desktop side — 403 tests on an iMac via Vulkan — proved Gemma 4 could run on consumer hardware. Edge Lab asks the next question: how fast is the same model family on a phone, with settings anyone can reproduce? Not a benchmark suite (no LLM judge, no scoring grid). A lab instrument: one matrix run, one manifest, ship the numbers.
Timeline
The Gemma 4 benchmark suite lives on the iMac — 403 tests, LLM judges, the model everyone said couldn't run. Today I shipped the other half: Edge Lab, an open iOS app that runs a four-preset experiment matrix on any BYOM .litertlm and exports a versioned JSON manifest. On iPhone 16 Pro Max with gemma-4-E2B-it: Sampled GPU hit 39.8 tok/s decode, Greedy CPU stayed honest at 4.7 tok/s on real CPU weights — not the Gallery app's opaque toggles. Posted the thread to X; the durable home is the repo plus a project page on this site. Same obsession (show your work), different instrument (throughput manifests, not quality scores). If you're comparing edge devices, you shouldn't need Google's closed Gallery app to read the numbers.