Skip to content
Andrew Voirol
WorkLogAboutContact
HomeWorkLog

AboutContact

✦ Just one prompt away from figuring it all out.

Friday, April 10, 2026gemma-4-benchmarks

37.6 tok/s on a GPU everyone said was dead

GPUVulkanHardwareGemma 4

Everyone says: Intel Mac GPU acceleration is dead for LLMs. Metal crashes on discrete AMD GPUs (it assumes unified memory). ROCm is Linux-only. Ollama can't talk to the Radeon Pro 575. So I compiled llama.cpp with the LunarG Vulkan SDK — MoltenVK, the layer nobody tests — and the GPU appeared instantly. E2B went from 7.3 tok/s (Ollama, CPU) to 37.6 tok/s on Vulkan. That's 5.1× faster. The E4B hits 24.4 tok/s — genuinely conversational speed on a 2017 iMac. The catch: only models under 5 GiB fit in 4 GB VRAM. The 31B at 17 GiB? Hybrid offload — 10 layers on GPU, the rest on CPU — gets 1.15 tok/s, a 17% boost over pure CPU. Not magic, but it's free performance from hardware everyone wrote off. Three hours of empirical testing later, 'worthless for LLMs' became 'the E4B runs at conversational speed.' Trust the hardware. Test the APIs. Ignore the forums.

← Previous

13 hours, 46 minutes, no human intervention

Next →

Show your work or it's a nothing burger


Andrew Voirol

Builder, hacker, shipper. Currently leaving localhost.

Navigate

WorkBuilder’s LogAboutContactRSS Feed

Connect

X / TwitterGitHubLinkedIn

© 2026 Andrew Voirol✦Just one prompt away from figuring it all out.