Running Gemma 4, Qwen 3 and Phi-4 on CPU in pure Rust

On October 25, 2026, the AI section of DEV Community published a technical article introducing a command line inference engine written in pure Rust. The engine runs quantized GGUF versions of several small language models entirely on a standard CPU, covering Gemma 4, Qwen 3/3.6 and Phi-4-mini.

The project builds on the author's earlier experiment, which used pure Rust together with the Hugging Face Candle framework to run Microsoft's Phi-3 on an ordinary consumer CPU. That first attempt set out to show that local AI inference does not require heavy Python runtimes, complicated CUDA setups, or Docker images larger than 5GB. In the current iteration the author has scaled the work from a single model into a command line tool that can load multiple models, and the article reports measured numbers for memory footprint, cold start latency and token generation speed across the supported models. The whole setup runs without a Python runtime and without any CUDA environment. As of the published article the project exists as this benchmark write up, and the piece does not mention any further release or open sourcing plans.