arXiv 2604.18788·2026-04-20 — views
NPUMoE: Efficient MoE LLM Inference with Apple Silicon NPUs
Afsara Benazir, Felix Xiaozhu Lin · University of Virginia (per author affiliations)
Runtime that handles MoE on Apple NPUs via offline calibration, static capacity tiers, and load-aware graph residency. 1.32-5.55× lower latency on M-series chips.
NPUMoE is a runtime that handles MoE’s awkward fit to NPU hardware (dynamic routing produces dynamic shapes; tiny expert kernels create high launch overhead). The pipeline: offline calibration of expert capacity and popularity, static capacity tiers, grouped expert execution, load-aware graph residency.
Reported numbers across three MoE LLMs and four long-context workloads on Apple M-series: 1.32-5.55× lower latency, 1.81-7.37× better energy efficiency, 1.78-5.54× lower CPU cycles vs prior approaches.
Practitioner note (mine)
If you run local LLMs on Apple Silicon — and many readers of this site do — this is concretely useful. A typical Mac mini + LiteLLM-routed Qwen LAN setup sits exactly in NPUMoE’s target zone.
The practical question is when (or if) NPUMoE’s techniques land in mainstream runtimes (mlx-lm, llama.cpp Metal backend, Ollama). Watch the next few releases of those projects — the speedups here are large enough that competitive runtimes will absorb them quickly.