2026-09-23 — views
Flash-dLLM: IO-Aware KV Caching and Parallel Decoding for Fast, Memory-Efficient Diffusion LLMs
Read this because Researchers and engineers working with diffusion-based language models can significantly reduce inference latency and memory overhead by adopting this training-free acceleration framework.
Flash-dLLM introduces an I/O-aware fused KV-cache kernel and parallel decoding strategy to accelerate diffusion LLMs without auxiliary models.
The Challenge of Diffusion LLM Inference
Diffusion Large Language Models (dLLMs) have recently emerged as a promising alternative to autoregressive LLMs by enabling non-autoregressive text generation. Despite their potential, their practical deployment remains limited by inefficient inference processes. This inefficiency is largely due to the absence of effective Key-Value (KV) caching and scalable parallel decoding mechanisms. Existing acceleration methods typically study KV caching and parallel decoding in isolation, overlooking the I/O bottlenecks that arise when cache reuse and parallel token verification are jointly applied.
Introducing Flash-dLLM
In this work, researchers introduce Flash-dLLM, a training-free inference acceleration framework designed for fast and memory-efficient dLLMs. The framework first identifies GPU memory I/O as a dominant bottleneck in KV-cache-enabled dLLM inference. To address this, it employs an I/O-aware fused KV-cache kernel that reduces redundant memory movement. This optimization forms the foundation for the subsequent decoding strategies.
Parallel Decoding Strategy
Building on this optimized cache mechanism, Flash-dLLM further proposes an efficient KV-cache-driven draft-and-verify decoding strategy. A key innovation in this design is that the dLLM itself serves as both the drafter and the verifier, eliminating the need for an auxiliary model. This unified design enables faster decoding while preserving generation quality. Additionally, it improves scalability to longer sequences and larger batch sizes, addressing critical limitations in current deployment scenarios.
Performance Results
Extensive experiments on mathematical reasoning and code-generation benchmarks demonstrate that Flash-dLLM consistently outperforms existing state-of-the-art dLLM acceleration methods in both inference speed and memory efficiency. The framework achieves significant performance gains over prior baselines. In particular, it achieves a 5.1x speedup over the prior strongest baseline, Elastic-Cache, on the GSM8K benchmark. Furthermore, it delivers an 11.0x speedup on the HumanEval benchmark. These results highlight the effectiveness of addressing I/O bottlenecks and integrating caching with parallel decoding in a single framework.
Practitioner note
Developers should consider integrating Flash-dLLM into their inference pipelines to reduce GPU memory I/O bottlenecks without adding auxiliary models. This training-free approach allows for immediate speedups in diffusion LLM deployments, particularly for tasks involving mathematical reasoning and code generation.