Gemma 4 Benchmark Suite
450+ tests across 7 models on two machines — iMac 2017 (Vulkan) and MacBook M4 Pro (Metal). GPU-accelerated to 90 tok/s.
450+ tests · 7 models · 94% top score
The Gemma 4 Benchmark Suite is a comprehensive testing pipeline designed to answer a single question: what does a state-of-the-art local AI model actually feel like on consumer hardware? Instead of relying on cloud data centers or enterprise GPUs, this suite ran over 450 tests across seven different models on two machines — an 8-year-old 2017 iMac and a modern MacBook M4 Pro.
The results challenged conventional wisdom. By utilizing the LunarG Vulkan SDK to resurrect a long-forgotten Radeon Pro 575 GPU, the iMac reached conversational speeds of 37.6 tok/s. On the Apple Silicon side, the M4 Pro ripped through inference at a blazing 90 tok/s for smaller models. More importantly, the largest 31B model—widely deemed impossible to run on older hardware—achieved a remarkable 94% top score in complex reasoning tasks, running completely unsupervised overnight.
The methodology prioritized real-world capability over purely synthetic numbers. Testing covered everything from logic and coding to multi-turn coherence and tool calling, with results processed through deterministic evaluation and LLM-as-a-judge pipelines. All runs were fully autonomous, orchestrated via bash scripts that managed execution, validation, and JSON reporting without human intervention.
Key Results
| Model | Score | Speed | VRAM |
|---|---|---|---|
| Gemma 4 31B (E4B) | 94% | 7.5 tok/s (iMac Vulkan) | 4 GB |
| Gemma 4 12B (E2B) | 85% | 38 tok/s (iMac) / 90 tok/s (M4 Pro) | 2 GB |
| Gemma 4 26B-A4B (MoE) | — | Blocked by Vulkan MoE bug | 4 GB |
Visual Analysis
The Data, Visualized
Seven models tested on the same iMac. Same tests, same grading. The story is in the shapes.
0
test results
across 7 models
0
universally perfect
out of 0 tests
0
tok/s peak
E2B Q4
0
min longest test
Daily Driver
E2B Q4
Fastest + highest score
Best Value
Unsloth DQ4
17.9% per GB
Quality King
31B Q4
93% accuracy
Model Fingerprints
Each model has a shape. Click to compare. Hover to isolate.
Speed vs Accuracy
Bubble size = disk footprint. Top-right is the sweet spot.
The Runtime Race
Same 39 tests. The fastest model finishes before the slowest has loaded.
Wall-clock time for 39 identical tests · score at right
The Inverted Ladder
Less precision. Higher score. The counterintuitive finding.
Same 5.1B architecture. Same 39 tests. Same iMac.
Lower precision → higher score.
Spotlight
Reasoning Tasks — The Heavy Lifts
The longest and hardest reasoning tasks across all models. Think mode on, math hard.
Deep Dive
Experiments beyond the scores
Beyond the standard benchmarks, we ran targeted experiments to answer specific questions about how these models behave in practice. Expand each to dig in.
We ran the same tasks with Think Mode ON and OFF across E2B and 26B models. The results were surprising — thinking doesn't always help, and sometimes it actively hurts.
| Task | E2B Off | E2B Think | 26B Off | 26B Think |
|---|---|---|---|---|
| Math | 100% | 20% | 100% | 100% |
| Logic | 20% | 20% | 60% | 20% |
| Code | 70% | 70% | 70% | 70% |
| Creative | 100% | 100% | 100% | 85% |
Think mode hurt E2B on math (100% → 20%) and 26B on logic (60% → 20%). On this hardware, the extra tokens spent "thinking" can actually degrade quality.
We buried a passphrase in progressively larger documents to find the real context ceiling on this hardware. The advertised limits don't match reality.
1K tokens
E2B
✓26B
✓4K tokens
E2B
✓26B
✗8K tokens
E2B
✗26B
✗16K tokens
E2B
✗26B
✗On 40GB RAM, E2B tops out around 4K tokens reliably. The 26B can only handle ~1K before OOM pressure causes timeouts. Forget about the advertised 128K context window on consumer hardware.
Can the models maintain coherence across a 5-turn conversation? Both E2B and 26B scored 95% — dropping only one point on turn 4 (a follow-up question that required referencing context from turn 1).
Both models: turn scores [100, 100, 100, 75, 100]. Impressive coherence.
Full Test Suite
Every test, every model, every score
All 39 unique tests across 7 model configurations. Click column headers to sort. Filter by category to focus on what matters to you.
| Test | Cat | Avg ↓ |
|---|
0 tests shown · 0 total results
All benchmarks were run locally using Ollama on a 2017 iMac (i7-7700K, 40GB RAM, macOS). Nothing fancy — this is a machine I actually use. Each test was executed sequentially with cold-start measurements, no prompt caching, and real wall-clock timing.
Models tested span Google's Gemma 4 family from the smallest E2B (2 billion parameters) up to the full 31B. Tests cover 10 categories: performance profiling, mathematical reasoning (AIME-style problems), code generation, tool calling, creative writing, multimodal analysis, agentic task completion, parameter optimization, context window stress testing, and score calibration.
Scoring uses a 0-1 scale. Performance tests are scored on metrics only (latency, throughput). Reasoning and coding tests use automated verification against expected answers.
I'm not a research lab. I'm one person who wanted to know if these models are actually useful on hardware I already own. I publish the failures alongside the wins because that's what's actually helpful.
— The whole point
● 7 models testedSome tests are still running. Some results will change as I learn more. The messy build log is more useful than the polished launch post.
For the complete story behind these numbers, including the MoE bug hunt and the three-way quantization showdown, read the full Gemma 4 Benchmarks Thread.