Running Gemma 4 on Consumer Hardware

Started building an autonomous benchmark orchestration pipeline. The goal: let the iMac run all 36 remaining tests overnight without me touching it. Sounds simple until you deal with model timeouts, OOM kills, partial results, and the machine going to sleep. SSH tunnels, pmset nosleep, and a lot of defensive scripting. This is the unglamorous side of 'local AI'.

Everyone benchmarks on H100s. Nobody tells you what a model actually feels like on the hardware you own. I wanted to know: can a 31B model run on consumer hardware? The answer is yes, with caveats. The latency is real, but the capability ceiling is higher than you'd expect. That's the gap I'm trying to fill — practical, honest numbers for people who build with what they have.

Every Ollama guide says you need 64GB+ unified memory for a 31B model. Every Reddit thread says don't bother. My own AI agent said 'expected to thrash memory.' I loaded the model anyway. First test: AIME competition math, Think Mode ON. It took 30 minutes. It got the right answer. That's the moment I knew the conventional wisdom was about interactive use — sub-second responses for chatbots. I don't need sub-second. I need correct. Different question, different answer.

Gemma 4 crossed 10 million downloads today. My mentions are filling up with people asking about local performance. Posted some of our benchmark numbers and the response has been wild — turns out nobody else is publishing consumer hardware results. The gap between 'runs on an H100' and 'runs on your machine' is real, and people are hungry for honest data. Feels good to be filling that gap, even if our testing is still in progress.

Ran Think Mode A/B tests today and the results are not what I expected. On the E2B model, enabling Think Mode on a math task dropped the score from 100% to 20%. On the 26B, it dropped logic from 60% to 20%. My hypothesis: on memory-constrained hardware, the extra tokens consumed by 'thinking' crowd out the tokens needed for a good answer. The model literally thinks itself into a worse response. This is the kind of finding you only get from testing on real hardware with real constraints.

We ran the complete E2B quantization ladder — Q4, Q8, and BF16 — through 39 identical tests on the same hardware. Q4 scored 92.2%. Q8 scored 91.8%. F16 scored 91.2%. The relationship is perfectly inverted: lower precision = higher score. This shouldn't happen. Quantization is lossy compression. But on DDR4 bandwidth-constrained hardware, smaller weights mean more of the model stays in CPU cache. Fewer cache misses. More consistent throughput. And the precision you 'lose' at Q4? It's apparently noise, not signal — at least for the tasks that matter. The daily driver isn't the most precise model. It's the smallest one. And it's also the fastest.

**Ran the E2B at full BF16 precision — 10.**3 GB, 149 minutes, 1.72 tok/s. The result? 91.2%. The Q8 quantized version? 91.8% in 36 minutes at 6.95 tok/s. Read that again: the quantized model scored higher than full precision while running 4× faster. This isn't supposed to happen. Quantization is lossy compression — it should always lose something. But on this hardware, the Q8 model actually benefits from its smaller memory footprint: fewer cache misses, more consistent throughput, and the 'lost' precision apparently doesn't matter for these tasks. The ceiling is a floor. Stop chasing precision. Chase speed.

Ran 4 models through 39 core tests each: E2B Q8, E4B Q8, Unsloth DQ4, and 31B Q4. The E4B — Google's 'recommended' mid-range model — is the worst value proposition in the lineup. It uses 2× the memory of the E2B Q8 for marginally better scores, and it's slower. Meanwhile Unsloth's dynamic Q4 of the E2B matches the E4B's quality at half the size and 38 tok/s. The 31B Q4 is the accuracy king at 94% but runs at 7.5 tok/s. The practical daily driver is Unsloth UD-Q4_K_XL: 2.94 GB, 38 tok/s, and the efficiency curve tells a story the spec sheets don't — bigger isn't always better when the quantization is smart enough.

The 31B results just came in. 63 tests. Every single one completed. Zero crashes, zero OOM errors, zero timeouts. 94.0% average score with 43 perfect scores. This is the same model that scored 21% a week ago with 83 errors — because everyone (including my AI assistant) was convinced it couldn't run on this hardware. They were wrong. I was stubborn. The data proved it. The difference wasn't the hardware — it was the approach. Stop treating a slow model like a fast one and let it work at its own pace. Build infrastructure that respects the machine's reality instead of fighting it.

Caught a subtle bug in the grading pipeline at 1am. The multi-turn context retention test scored 20% — the 31B's only 'failure.' But the model actually answered correctly: recalled the user's name (Alex) and their project (laundry-folding robot). The LLM judge scored it low because it couldn't see the conversation context. It was evaluating a correct answer as gibberish because it lacked the question. Fixed it by routing multi-turn tests through deterministic scoring instead. Score: 100%. This is why you never trust a benchmark you don't understand inside and out.

The iMac ran the full 31B benchmark suite autonomously from 10am yesterday to midnight. I built a decoupled runner that saves raw results after every test, skips already-completed tests on restart, and separates model inference from scoring entirely. The scorer runs later on a fast model. No SSH tunnels needed. No monitoring. Just a machine doing its job while I worked on other things. This is what 'local AI infrastructure' actually looks like — boring, reliable plumbing.

Everyone says: Intel Mac GPU acceleration is dead for LLMs. Metal crashes on discrete AMD GPUs (it assumes unified memory). ROCm is Linux-only. Ollama can't talk to the Radeon Pro 575. So I compiled llama.cpp with the LunarG Vulkan SDK — MoltenVK, the layer nobody tests — and the GPU appeared instantly. E2B went from 7.3 tok/s (Ollama, CPU) to 37.6 tok/s on Vulkan. That's 5.1× faster. The E4B hits 24.4 tok/s — genuinely conversational speed on a 2017 iMac. The catch: only models under 5 GiB fit in 4 GB VRAM. The 31B at 17 GiB? Hybrid offload — 10 layers on GPU, the rest on CPU — gets 1.15 tok/s, a 17% boost over pure CPU. Not magic, but it's free performance from hardware everyone wrote off. Three hours of empirical testing later, 'worthless for LLMs' became 'the E4B runs at conversational speed.' Trust the hardware. Test the APIs. Ignore the forums.

Published the full benchmark source code to GitHub. Every prompt. Every grading function. Every raw JSON result. Because if you're going to claim '94% on consumer hardware' on a fancy website and then compare yourself to Google's published benchmarks, you better show how you got those numbers. The repo has the Python orchestrator, the Ollama client wrapper, the test definitions across 7 categories (reasoning, coding, tool calling, creative, context, agentic, performance), and the raw results from all 414 test runs. MIT licensed. Anyone can clone it, pull a Gemma model, and run the same suite on their own hardware. That's the difference between a benchmark and a blog post.

Ran two machines overnight while I tried to sleep (insomnia is a feature, not a bug). The iMac ran 5 automated experiments back-to-back: 26B GPU layer scan, three-way quantization quality showdown, E4B full test suite, multi-turn coherence, and needle-in-haystack retrieval. The MacBook M4 Pro waited for a 17.5 GB SCP transfer then auto-ran benchmarks. Everything was orchestrated with bash scripts that watched for completion signals and restarted servers between experiments. Total time: 4 hours autonomous. Total data: 15 JSON result files. One MacBook overnight script had a race condition (server wasn't ready when tests started), but everything else ran clean. The iMac is 8 years old and it ran an autonomous AI research pipeline while I slept. This is what local infrastructure looks like — boring, reliable, unsupervised.

Ran Q8_0, Google Q4_K_M, and Unsloth UD-Q4_K_XL through identical 8-question quality suites on the same hardware. Q8 (4.69 GB): 7/8 at 30.12 tok/s. Google Q4_K_M (3.21 GB): 6/8 at 35.36 tok/s. Unsloth UD (2.94 GB): 7/8 at 38.16 tok/s. The smallest model is the fastest AND ties the biggest for accuracy. Unsloth's dynamic quantization preserves the weights that matter and compresses the ones that don't — and the imatrix calibration data means it knows which is which. This isn't theoretical. 2.94 GB, 38 tok/s, 88% accuracy. Unsloth UD-Q4_K_XL is the daily driver. Full stop.

The 31B Unsloth UD-Q4_K_XL failed catastrophically on overnight tests. No-think mode: 3/8 correct. Think mode: 0/8 correct. The failure pattern is specific and repeatable: the model starts a correct answer — 'The Berlin Wall fell in 1989' — then degenerates into garbage tokens (<unused50><unused50><unused50>) until it hits max_tokens. This is a quantization artifact at low tok/s. At 0.92 tok/s, each token takes ~1 second to generate, and the model's internal state apparently drifts during that time. The same quant on smaller models (E2B, E4B) works perfectly at 30+ tok/s. Think mode makes it worse — the hidden reasoning tokens eat into the answer budget, and the garbage tokens start even earlier. The takeaway: Unsloth dynamic quant is fantastic for models that fit in VRAM. For models that spill to CPU at <2 tok/s, you need Google's standard quantization or higher precision. Speed isn't just about user experience — it's about model coherence.

Benchmarked the MacBook M4 Pro (24 GB unified, Metal 4) against the iMac (Radeon Pro 575, Vulkan). The E2B Q4 hits 90 tok/s on the M4 — that's instantaneous, faster than reading speed. The E4B at Q8 does 42 tok/s. Both feel like native apps. But the 31B? The M4 Pro loads it into unified memory just fine — no swap, no thrashing — and runs at 12.4 tok/s. Usable for batch processing, too slow for conversation. The iMac's Vulkan path actually beats it at 7.5 tok/s because it's offloading to the discrete GPU's 4 GB VRAM while the CPU handles the rest. Different architectures, different tradeoffs. The M4 is the daily driver. The iMac is the overnight workhorse.