LeanSparse: Shrinking GPU Memory for Sparse-Attention LLM Serving

This episode examines LeanSparse, a serving system that cuts the GPU memory needed to hold many long-context conversations on models with native sparse attention, such as DeepSeek-V4-Flash. It explains why sparse attention reduces reads but not storage. Even with KV offloaded to host memory, the lightning indexer's keys must stay on the GPU, and at 512K tokens they take about 7.2 times the space of the hot KV buffer. It also covers why agent-style workloads, where context grows every turn, make this memory limit bite, and why prefill needs a large temporary workspace. That workspace forces a bad choice between reserving memory that sits idle and making new requests queue behind decoders, which hurts time to first token. The proposed fixes are to stream each layer's history through two alternating GPU buffers during prefill, and to search compact indexer keys during decode, fetching only a short list of originals to rank exactly. All of this runs on a shared instance, avoiding the duplicated weights of prefill-decode disaggregation. Listeners get a clear walkthrough of where memory goes in modern sparse-attention serving, plus the reasoning behind why the obvious fix of offloading the indexer keys fails. Sources: 1. LeanSparse: Memory-Efficient Serving for Native Sparse-Attention LLMs — Zaifeng Pan, Chris Wu, Xinwei Qiang, Zhengding Hu, Zhongkai Yu, Yufei Ding, 2026 http://arxiv.org/abs/2611.04671 2. Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention — Jingyang Yuan, Huazuo Gao, Damai Dai, et al. (DeepSeek-AI, Peking University), 2025 https://scholar.google.com/scholar?q=Native+Sparse+Attention%3A+Hardware-Aligned+and+Natively+Trainable+Sparse+Attention 3. DeepSeek-V3.2-Exp technical report (DeepSeek Sparse Attention, DSA) — DeepSeek-AI, 2025 https://scholar.google.com/scholar?q=DeepSeek-V3.2-Exp+technical+report+%28DeepSeek+Sparse+Attention%2C+DSA%29 4. MoBA: Mixture of Block Attention for Long-Context LLMs — Enzhe Lu, Zhejun Jiang, Jingyuan Liu, et al. (Moonshot AI), 2025 https://scholar.google.com/scholar?q=MoBA%3A+Mixture+of+Block+Attention+for+Long-Context+LLMs 5. Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference — Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, Song Han, 2024 https://scholar.google.com/scholar?q=Quest%3A+Query-Aware+Sparsity+for+Efficient+Long-Context+LLM+Inference 6. InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management — Wonbeom Lee, Jungi Lee, Junghwan Seo, Jaewoong Sim, 2024 https://scholar.google.com/scholar?q=InfiniGen%3A+Efficient+Generative+Inference+of+Large+Language+Models+with+Dynamic+KV+Cache+Management 7. ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference — Hanshi Sun, Li-Wen Chang, Wenlei Bao, et al., 2024 https://scholar.google.com/scholar?q=ShadowKV%3A+KV+Cache+in+Shadows+for+High-Throughput+Long-Context+LLM+Inference 8. RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrieval — Di Liu, Meng Chen, Baotong Lu, et al., 2024 https://scholar.google.com/scholar?q=RetrievalAttention%3A+Accelerating+Long-Context+LLM+Inference+via+Vector+Retrieval 9. Billion-scale similarity search with GPUs (FAISS) — Jeff Johnson, Matthijs Douze, Hervé Jégou, 2017 https://scholar.google.com/scholar?q=Billion-scale+similarity+search+with+GPUs+%28FAISS%29 10. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — Ruoyu Qin, Zheming Li, Weiran He, et al. (Moonshot AI, Tsinghua), 2024 https://scholar.google.com/scholar?q=Mooncake%3A+A+KVCache-centric+Disaggregated+Architecture+for+LLM+Serving 11. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving — Yinmin Zhong, Shengyu Liu, Junda Chen, et al., 2024 https://scholar.google.com/scholar?q=DistServe%3A+Disaggregating+Prefill+and+Decoding+for+Goodput-optimized+Large+Language+Model+Serving 12. Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve — Amey Agrawal, Nitin Kedia, Ashish Panwar, et al., 2024 https://scholar.google.com/scholar?q=Taming+Throughput-Latency+Tradeoff+in+LLM+Inference+with+Sarathi-Serve 13. FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU — Ying Sheng, Lianmin Zheng, Binhang Yuan, et al., 2023 https://scholar.google.com/scholar?q=FlexGen%3A+High-Throughput+Generative+Inference+of+Large+Language+Models+with+a+Single+GPU 14. PQCache: Product Quantization-based KVCache for Long Context LLM Inference — Hailin Zhang et al., 2025 https://scholar.google.com/scholar?q=PQCache%3A+Product+Quantization-based+KVCache+for+Long+Context+LLM+Inference 15. DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models (DeepSeek Sparse Attention) — DeepSeek-AI, 2025 https://scholar.google.com/scholar?q=DeepSeek-V3.2%3A+Pushing+the+Frontier+of+Open+Large+Language+Models+%28DeepSeek+Sparse+Attention%29 16. CachedAttention (AttentionStore): Cost-Efficient LLM Serving for Multi-Turn Conversations — Bin Gao et al., 2024 https://scholar.google.com/scholar?q=CachedAttention+%28AttentionStore%29%3A+Cost-Efficient+LLM+Serving+for+Multi-Turn+Conversations 17. SparseServe: Unlocking Parallelism for Dynamic Sparse Attention in Long-Context LLM Serving — Zhou et al., 2025 https://scholar.google.com/scholar?q=SparseServe%3A+Unlocking+Parallelism+for+Dynamic+Sparse+Attention+in+Long-Context+LLM+Serving 18. Sarathi-Serve: Taming Throughput-Latency Tradeoff in LLM Inference — Agrawal et al., 2024 https://scholar.google.com/scholar?q=Sarathi-Serve%3A+Taming+Throughput-Latency+Tradeoff+in+LLM+Inference 19. Product Quantization for Nearest Neighbor Search — Hervé Jégou, Matthijs Douze, Cordelia Schmid, 2011 https://scholar.google.com/scholar?q=Product+Quantization+for+Nearest+Neighbor+Search Interactive Visualization: LeanSparse: Shrinking GPU Memory for Sparse-Attention LLM Serving