2026-10-24 — views
BeNice! Closing Looping's Inference-Memory Gap via KV-Sharing
Read this because This method solves a critical memory bottleneck in looped architectures, allowing for longer contexts and higher throughput on constrained hardware by reusing key-value caches.
Researchers propose BeNice! to enable KV-cache sharing in looped models, reducing memory usage and improving inference throughput without quality loss.
The Memory Challenge in Looped Architectures
Looped architectures offer a compelling way to scale model quality while keeping the parameter count fixed. They achieve this by repeatedly passing activations through the same block of layers. However, this recurrence creates a significant bottleneck during inference. Each recurrence generates independent key-value (KV) caches, causing memory usage to grow linearly with the number of steps. To address this, prior work has explored KV-sharing, which involves sharing KV-caches across depth. Some studies found that the KV-cache from the final recurrence could be shared with earlier recurrences without degrading quality, a property termed KV-shareability. Despite this potential, the mechanisms behind KV-shareability remain poorly understood. Even identically trained looped architectures do not consistently exhibit this property, making it difficult to rely on for production systems.
Introducing BeNice!
To enable KV-shareability reliably, researchers have proposed a new approach called BeNice!. This method consists of two distinct stages designed to induce and enforce cache sharing. In the first stage, stochastic depth is used during pretraining. This technique induces contractive residual dynamics, which drive representations toward a common limit across recurrences. This foundational step prepares the model for sharing. The second stage is a post-training method that actively broadcasts the final recurrence’s KV-cache into the first recurrence. During this phase, the model is trained to produce compatible residuals while preserving the original model’s quality. The result is a model that enables KV-shareability while matching the unshared validation perplexity of standard fine-tuning methods.
Performance and Compatibility
BeNice! demonstrates that KV-sharing is complementary to other memory-efficient techniques. It composes effectively with alternative KV-compression methods, such as memory-efficient sequence mixers like multi-latent attention and Kimi delta attention. It also works well with low-precision KV-cache quantization methods, including fp8, mxfp8, and nvfp4. In simulated low-memory, long-context inference scenarios, the benefits are substantial. KV-sharing improved cache hit rates by up to 31.4%. Throughput increased by up to 23.6%, and the p90 time-to-first-token was reduced by up to 83.6%. These metrics highlight the potential for significant efficiency gains in resource-constrained environments.
Practitioner note
Developers should integrate stochastic depth during pretraining and apply the BeNice! post-training broadcast step to enable KV-cache sharing. This approach allows you to significantly reduce memory footprint and latency in long-context inference without sacrificing model quality.