The evaluation framework that changes the benchmarking game
Bootstrapped the evaluation framework — 4 built-in suites (Math, Tool Calling, Reasoning, Multimodal), custom suite editor, batch "Run All" mode with time estimation. This was the session where the ModelCapabilityProfile migration paid off. The old ModelMetadata was a flat struct that knew models by name. The new system knows what models can do — which suites they can run, whether they support vision input, what their context limits are. The eval runner uses this to skip incompatible suites automatically instead of crashing on large prompts. Now we can benchmark exactly how these models degrade at high quantization levels, right on the metal. Not synthetic cloud benchmarks — real hardware, real thermal throttling, real numbers.