Tags · #kv-cache

BeNice! Closing Looping's Inference-Memory Gap via KV-Sharing

Read this because This method solves a critical memory bottleneck in looped architectures, allowing for longer contexts and higher throughput on constrained hardware by reusing key-value caches.

Researchers propose BeNice! to enable KV-cache sharing in looped models, reducing memory usage and improving inference throughput without quality loss.

24 OCT 2 MIN READ Practitioner note llm inference memory optimization

Flash-dLLM: IO-Aware KV Caching and Parallel Decoding for Fast, Memory-Efficient Diffusion LLMs

Read this because Researchers and engineers working with diffusion-based language models can significantly reduce inference latency and memory overhead by adopting this training-free acceleration framework.

Flash-dLLM introduces an I/O-aware fused KV-cache kernel and parallel decoding strategy to accelerate diffusion LLMs without auxiliary models.

23 SEP 2 MIN READ Practitioner note llm diffusion gpu optimization

Nvidia, AMD, and CoreWeave all backed Tensormesh — KV-cache reuse becomes an inference primitive

Read this because Three rivals — Nvidia, AMD, CoreWeave — co-investing is the tell: KV-cache reuse (don't recompute what you already computed) is being treated as a foundational, neutral inference-stack layer. The economics of the inference era, in one round.

Tensormesh raised $20M from Nvidia, AMD, and CoreWeave and shipped Tensormesh Inference — productized KV-cache reuse claiming up to 10x lower latency/GPU cost.

28 MAY 3 MIN READ Practitioner note tensormesh inference kv-cache nvidia