BeNice! Closing Looping's Inference-Memory Gap via KV-Sharing
Read this because This method solves a critical memory bottleneck in looped architectures, allowing for longer contexts and higher throughput on constrained hardware by reusing key-value caches.
Researchers propose BeNice! to enable KV-cache sharing in looped models, reducing memory usage and improving inference throughput without quality loss.