Skip to content
Andrew Voirol
WorkLogAboutContact
HomeWorkLog

AboutContact

✦ Just one prompt away from figuring it all out.

Thursday, April 9, 2026gemma-4-benchmarks

The grading artifact that almost ruined everything

DebuggingBenchmarksReflections

Caught a subtle bug in the grading pipeline at 1am. The multi-turn context retention test scored 20% — the 31B's only 'failure.' But the model actually answered correctly: recalled the user's name (Alex) and their project (laundry-folding robot). The LLM judge scored it low because it couldn't see the conversation context. It was evaluating a correct answer as gibberish because it lacked the question. Fixed it by routing multi-turn tests through deterministic scoring instead. Score: 100%. This is why you never trust a benchmark you don't understand inside and out.

← Previous

94%. Zero errors. I told you so.

Next →

13 hours, 46 minutes, no human intervention


Andrew Voirol

Builder, hacker, shipper. Currently leaving localhost.

Navigate

WorkBuilder’s LogAboutContactRSS Feed

Connect

X / TwitterGitHubLinkedIn

© 2026 Andrew Voirol✦Just one prompt away from figuring it all out.