Skip to content
Andrew Voirol
WorkLogAboutContact
HomeWorkLog

AboutContact

✦ Just one prompt away from figuring it all out.

Saturday, April 11, 2026gemma-4-benchmarks

31B quality collapse: the model that broke its own brain

Gemma 431BQuantizationDebugging

The 31B Unsloth UD-Q4_K_XL failed catastrophically on overnight tests. No-think mode: 3/8 correct. Think mode: 0/8 correct. The failure pattern is specific and repeatable: the model starts a correct answer — 'The Berlin Wall fell in 1989' — then degenerates into garbage tokens (<unused50><unused50><unused50>) until it hits max_tokens. This is a quantization artifact at low tok/s. At 0.92 tok/s, each token takes ~1 second to generate, and the model's internal state apparently drifts during that time. The same quant on smaller models (E2B, E4B) works perfectly at 30+ tok/s. Think mode makes it worse — the hidden reasoning tokens eat into the answer budget, and the garbage tokens start even earlier. The takeaway: Unsloth dynamic quant is fantastic for models that fit in VRAM. For models that spill to CPU at <2 tok/s, you need Google's standard quantization or higher precision. Speed isn't just about user experience — it's about model coherence.

← Previous

Unsloth wins the three-way quant showdown

Next →

M4 Pro: 90 tok/s and the 31B reality check


Andrew Voirol

Builder, hacker, shipper. Currently leaving localhost.

Navigate

WorkBuilder’s LogAboutContactRSS Feed

Connect

X / TwitterGitHubLinkedIn

© 2026 Andrew Voirol✦Just one prompt away from figuring it all out.