31B quality collapse: the model that broke its own brain
The 31B Unsloth UD-Q4_K_XL failed catastrophically on overnight tests. No-think mode: 3/8 correct. Think mode: 0/8 correct. The failure pattern is specific and repeatable: the model starts a correct answer — 'The Berlin Wall fell in 1989' — then degenerates into garbage tokens (<unused50><unused50><unused50>) until it hits max_tokens. This is a quantization artifact at low tok/s. At 0.92 tok/s, each token takes ~1 second to generate, and the model's internal state apparently drifts during that time. The same quant on smaller models (E2B, E4B) works perfectly at 30+ tok/s. Think mode makes it worse — the hidden reasoning tokens eat into the answer budget, and the garbage tokens start even earlier. The takeaway: Unsloth dynamic quant is fantastic for models that fit in VRAM. For models that spill to CPU at <2 tok/s, you need Google's standard quantization or higher precision. Speed isn't just about user experience — it's about model coherence.