2026-09-30 — views
WUSH-KV: Data-Adaptive Transforms for Low-Bit KV Cache Quantization
Read this because Optimizes long-context inference by reducing memory bandwidth bottlenecks through novel quantization techniques.
WUSH-KV reduces KV cache memory costs using data-adaptive transforms, achieving low perplexity at 2-bit quantization.
Addressing KV Cache Bottlenecks
The memory and bandwidth costs associated with Key-Value (KV) caches increase significantly as context length and batch size grow. This scaling behavior creates a substantial bottleneck that limits the efficiency of long-context inference in large language models. To mitigate this issue, researchers have introduced WUSH-KV, a method designed for low-bit KV-cache quantization. This approach aims to reduce the overhead while maintaining model accuracy.
Data-Adaptive Transformations
WUSH-KV adapts the WUSH framework, which constructs a data-aware transform based on the second-order statistics of both factors in a matrix product. The primary goal of this transformation is to reduce quantization error. The method utilizes calibration data to construct separate transforms for the key and value components. A key innovation in this architecture is that the value transform is folded directly into the model weights. In contrast, the key transform is applied after the Rotary Positional Embedding (RoPE) operation. These transforms are designed to work in conjunction with clipped quantizers.
Performance and Optimality
Theoretical analysis suggests that under mild assumptions, the WUSH transform is near-optimal for specific quantizers, such as QuEST INT. Empirical evaluations demonstrate that WUSH-KV reduces layerwise reconstruction error compared to other methods. Furthermore, it achieves the lowest end-to-end perplexity among the tested transforms. This indicates that the data-adaptive approach effectively preserves model performance despite aggressive quantization.
Integration and Evaluation
For practical end-to-end evaluation, WUSH-KV was integrated into SGLang. This integration utilized OSCAR-style percentile-clipped affine quantization. The results at 2-bit quantization show that WUSH-KV performs comparably to, or outperforms, the OSCAR transform. This superiority holds across all evaluated models and downstream tasks. The findings suggest that WUSH-KV offers a robust solution for managing memory constraints in long-context scenarios without sacrificing the quality of the generated output.
Practitioner note
Developers should consider integrating WUSH-KV into their SGLang deployments to optimize memory usage for long-context inference. Monitoring layerwise reconstruction error and end-to-end perplexity will help verify that the 2-bit quantization maintains acceptable model accuracy for your specific downstream tasks.