Hydragen: Shared-Prefix Attention for High-Throughput LLM Inference

This episode examines Hydragen, a technique for speeding up LLM inference when many requests share a long prompt prefix, and asks how much of its claimed 32x end-to-end throughput gain comes from the idea and how much from the baseline. It explains why decode attention is memory-bound: each sequence runs a matrix-vector product against its own KV cache, which leaves tensor cores idle and makes batching unable to help. The central argument is that vLLM's PagedAttention deduplicates the storage of shared prefixes but not the reads, so every sequence still streams the full prefix from memory. Hydragen targets those reads by splitting attention into prefix and suffix parts and combining them using log-sum-exp weights. The authors present this as exact attention built on the FlashAttention family, with no approximation. The episode ends as the hosts ask why splitting the softmax is still exact. Sources: 1. Hydragen: High-Throughput LLM Inference with Shared Prefixes — Jordan Juravsky, Bradley Brown, Ryan Ehrlich, Daniel Y. Fu, Christopher Ré, Azalia Mirhoseini, 2024 http://arxiv.org/abs/2402.05099 2. Hydragen: High-Throughput LLM Inference with Shared Prefixes — Jordan Juravsky, Bradley Brown, Ryan Ehrlich, Daniel Y. Fu, Christopher Ré, Azalia Mirhoseini, 2024 https://scholar.google.com/scholar?q=Hydragen%3A+High-Throughput+LLM+Inference+with+Shared+Prefixes 3. Cascade Inference: Memory Bandwidth Efficient Shared Prefix Batch Decoding (FlashInfer blog) and FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving — Zihao Ye, Lequn Chen, Ruihang Lai, Wuwei Lin, Yineng Zhang, Stephanie Wang, Tianqi Chen, Baris Kasikci, Vinod Grover, Arvind Krishnamurthy, Luis Ceze, 2024 (blog) / 2025 (MLSys paper) https://scholar.google.com/scholar?q=Cascade+Inference%3A+Memory+Bandwidth+Efficient+Shared+Prefix+Batch+Decoding+%28FlashInfer+blog%29+and+FlashInfer%3A+Efficient+and+Customizable+Attention+Engine+for+LLM+Inference+Serving 4. Bifurcated Attention: Accelerating Massively Parallel Decoding with Shared Prefixes in LLMs — Ben Athiwaratkun et al. (Amazon / AWS AI Labs), 2024 https://scholar.google.com/scholar?q=Bifurcated+Attention%3A+Accelerating+Massively+Parallel+Decoding+with+Shared+Prefixes+in+LLMs 5. ChunkAttention: Efficient Self-Attention with Prefix-Aware KV Cache and Two-Phase Partition — Lu Ye, Ze Tao, Yong Huang, Yang Li, 2024 https://scholar.google.com/scholar?q=ChunkAttention%3A+Efficient+Self-Attention+with+Prefix-Aware+KV+Cache+and+Two-Phase+Partition 6. Online normalizer calculation for softmax; Self-attention Does Not Need O(n^2) Memory; FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — Maxim Milakov & Natalia Gimelshein (2018); Markus Rabe & Charles Staats (2021); Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, Christopher Ré (2022), 2018 / 2021 / 2022 https://scholar.google.com/scholar?q=Online+normalizer+calculation+for+softmax%3B+Self-attention+Does+Not+Need+O%28n%5E2%29+Memory%3B+FlashAttention%3A+Fast+and+Memory-Efficient+Exact+Attention+with+IO-Awareness 7. SGLang: Efficient Execution of Structured Language Model Programs (introduces RadixAttention) — Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, Ying Sheng, 2023 (arXiv) / 2024 (NeurIPS) https://scholar.google.com/scholar?q=SGLang%3A+Efficient+Execution+of+Structured+Language+Model+Programs+%28introduces+RadixAttention%29 8. Efficient Memory Management for Large Language Model Serving with PagedAttention (vLLM) — Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, Ion Stoica, 2023 https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention+%28vLLM%29 9. DeFT: Decoding with Flash Tree-attention for Efficient Tree-structured LLM Inference — Jinwei Yao, Kaiqi Chen, Kexun Zhang, Jiaxuan You, Binhang Yuan, Zeke Wang, Tao Lin, 2024 https://scholar.google.com/scholar?q=DeFT%3A+Decoding+with+Flash+Tree-attention+for+Efficient+Tree-structured+LLM+Inference 10. Large Language Monkeys: Scaling Inference Compute with Repeated Sampling — Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V. Le, Christopher Ré, Azalia Mirhoseini, 2024 https://scholar.google.com/scholar?q=Large+Language+Monkeys%3A+Scaling+Inference+Compute+with+Repeated+Sampling 11. SpecInfer: Accelerating Generative LLM Serving with Tree-based Speculative Inference and Verification; Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads — Xupeng Miao et al. (2023); Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D. Lee, Deming Chen, Tri Dao (2024), 2023 / 2024 https://scholar.google.com/scholar?q=SpecInfer%3A+Accelerating+Generative+LLM+Serving+with+Tree-based+Speculative+Inference+and+Verification%3B+Medusa%3A+Simple+LLM+Inference+Acceleration+Framework+with+Multiple+Decoding+Heads 12. Efficient Memory Management for Large Language Model Serving with PagedAttention — Kwon, Li, Zhuang, Sheng, Zheng, Yu, Gonzalez, Zhang, Stoica, 2023 https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention 13. SGLang: Efficient Execution of Structured Language Model Programs (RadixAttention) — Zheng, Yin, Xie, Huang, Sun, Yu, Cao, Kozyrakis, Stoica, Gonzalez, Barrett, Sheng, 2023/2024 https://scholar.google.com/scholar?q=SGLang%3A+Efficient+Execution+of+Structured+Language+Model+Programs+%28RadixAttention%29 14. FlashAttention / FlashAttention-2 — Dao, Fu, Ermon, Rudra, Re; Dao, 2022/2023 https://scholar.google.com/scholar?q=FlashAttention+%2F+FlashAttention-2 15. Online normalizer calculation for softmax; Self-attention Does Not Need O(n^2) Memory — Milakov and Gimelshein; Rabe and Staats, 2018; 2021 https://scholar.google.com/scholar?q=Online+normalizer+calculation+for+softmax%3B+Self-attention+Does+Not+Need+O%28n%5E2%29+Memory 16. Flash-Decoding for long-context inference — Dao, Haziza, Massa, Sizov (PyTorch/Together blog), 2023 https://scholar.google.com/scholar?q=Flash-Decoding+for+long-context+inference 17. Cascade Inference: Memory Bandwidth Efficient Shared Prefix Batch Decoding (FlashInfer) — Ye, Chen, Lai, et al., 2024 https://scholar.google.com/scholar?q=Cascade+Inference%3A+Memory+Bandwidth+Efficient+Shared+Prefix+Batch+Decoding+%28FlashInfer%29 18. Efficiently Scaling Transformer Inference — Pope, Douglas, Chowdhery, Devlin, Bradbury, Levskaya, Heek, Xiao, Agrawal, Dean, 2022 https://scholar.google.com/scholar?q=Efficiently+Scaling+Transformer+Inference 19. CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion — Yao et al., 2024/2025 https://scholar.google.com/scholar?q=CacheBlend%3A+Fast+Large+Language+Model+Serving+for+RAG+with+Cached+Knowledge+Fusion 20. AlphaCode / Large Language Monkeys: Scaling Inference Compute with Repeated Sampling — Li et al.; Brown, Juravsky, Ehrlich, et al., 2022; 2024 https://scholar.google.com/scholar?q=AlphaCode+%2F+Large+Language+Monkeys%3A+Scaling+Inference+Compute+with+Repeated+Sampling Interactive Visualization: Hydragen: Shared-Prefix Attention for High-Throughput LLM Inference