Skip to content
Andrew Voirol
WorkLogAboutContact
HomeWorkLog

AboutContact

✦ Just one prompt away from figuring it all out.

Monday, July 20, 2026edge-ai-lab-development

The evaluation framework that changes the benchmarking game

BenchmarksGemma 4Infrastructure

Bootstrapped the evaluation framework — 4 built-in suites (Math, Tool Calling, Reasoning, Multimodal), custom suite editor, batch "Run All" mode with time estimation. This was the session where the ModelCapabilityProfile migration paid off. The old ModelMetadata was a flat struct that knew models by name. The new system knows what models can do — which suites they can run, whether they support vision input, what their context limits are. The eval runner uses this to skip incompatible suites automatically instead of crashing on large prompts. Now we can benchmark exactly how these models degrade at high quantization levels, right on the metal. Not synthetic cloud benchmarks — real hardware, real thermal throttling, real numbers.

← Previous

You don't have to scrape HuggingFace — just ask the Hub API

Next →

ModelMetadata is dead — long live ModelCapabilityProfile


Andrew Voirol

Builder, hacker, shipper. Currently leaving localhost.

Navigate

WorkBuilder’s LogAboutContactRSS Feed

Connect

X / TwitterGitHubLinkedIn

© 2026 Andrew Voirol✦Just one prompt away from figuring it all out.