Cascade: SLO-Aware Latency Budgets for Fair, High-Goodput LLM Serving

This episode examines Cascade, a serving system that uses a single "latency budget" (a request's time-to-first-token target minus its predicted service time) to drive both queue ordering and KV cache restoration decisions. It explains why first-come-first-served, shortest-job-first and earliest-deadline-first each fail. The failures are head-of-line blocking, starved long prompts, and an inability to tell a 50 ms job from a 20 s job with the same deadline. The discussion covers goodput, tiered prefix caching across GPU memory, CPU DRAM and NVMe, and chunked prefill. It also covers the paper's motivating findings: aggregate SLO attainment hides starved classes, deep-tier restores such as a 64,000-token NVMe fetch can blow the latency target, and most violations come from queueing rather than a lack of budget. Listeners get a clear walkthrough of the scheduling vocabulary, Jain's fairness index and the budget estimator. They also get a caveat that the headline cluster results come from an extended simulator fed with GB200 NVL72 data-movement profiles rather than a real deployment. Sources: 1. Cascade: Exploiting SLO-Aware latency budget for fair and high goodput LLM inference serving — Muhammad Adnan, Rohan Mahapatra, Prashant J. Nair, Daniel Berger, Pantea Zardoshti, Rodrigo Fonseca, Esha Choukse, 2026 http://arxiv.org/abs/2608.06557 2. Orca: A Distributed Serving System for Transformer-Based Generative Models — Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, Byung-Gon Chun, 2022 https://scholar.google.com/scholar?q=Orca%3A+A+Distributed+Serving+System+for+Transformer-Based+Generative+Models 3. Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve — Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav Gulavani, Alexey Tumanov, Ramachandran Ramjee, 2024 https://scholar.google.com/scholar?q=Taming+Throughput-Latency+Tradeoff+in+LLM+Inference+with+Sarathi-Serve 4. Fast Distributed Inference Serving for Large Language Models (FastServe) — Bingyang Wu, Yinmin Zhong, Zili Zhang, Shengyu Liu, Fangyue Liu, Yuanhang Sun, Gang Huang, Xuanzhe Liu, Xin Jin, 2023 https://scholar.google.com/scholar?q=Fast+Distributed+Inference+Serving+for+Large+Language+Models+%28FastServe%29 5. Efficient LLM Scheduling by Learning to Rank — Yichao Fu, Siqi Zhu, Runlong Su, Aurick Qiao, Ion Stoica, Hao Zhang, 2024 https://scholar.google.com/scholar?q=Efficient+LLM+Scheduling+by+Learning+to+Rank 6. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving — Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, Hao Zhang, 2024 https://scholar.google.com/scholar?q=DistServe%3A+Disaggregating+Prefill+and+Decoding+for+Goodput-optimized+Large+Language+Model+Serving 7. Fairness in Serving Large Language Models — Ying Sheng, Shiyi Cao, Dacheng Li, Banghua Zhu, Zhuohan Li, Danyang Zhuo, Joseph E. Gonzalez, Ion Stoica, 2024 https://scholar.google.com/scholar?q=Fairness+in+Serving+Large+Language+Models 8. A Quantitative Measure of Fairness and Discrimination for Resource Allocation in Shared Computer Systems — Raj Jain, Dah-Ming Chiu, William R. Hawe, 1984 https://scholar.google.com/scholar?q=A+Quantitative+Measure+of+Fairness+and+Discrimination+for+Resource+Allocation+in+Shared+Computer+Systems 9. Andes: Defining and Enhancing Quality-of-Experience in LLM-Based Text Streaming Services — Jiachen Liu, Zhiyu Wu, Jae-Won Chung, Fan Lai, Myungjin Lee, Mosharaf Chowdhury, 2024 https://scholar.google.com/scholar?q=Andes%3A+Defining+and+Enhancing+Quality-of-Experience+in+LLM-Based+Text+Streaming+Services 10. Sarathi-Serve: Taming Throughput-Latency Tradeoff in LLM Inference — Agrawal, Kedia, Panwar, Mohan, Kwatra, Gulavani, Tumanov, Ramjee, 2024 https://scholar.google.com/scholar?q=Sarathi-Serve%3A+Taming+Throughput-Latency+Tradeoff+in+LLM+Inference 11. Fairness in Serving Large Language Models (Virtual Token Counter, VTC) — Sheng, Cao, Li, Zhuang, Zhang, Gonzalez, Stoica et al., 2024 https://scholar.google.com/scholar?q=Fairness+in+Serving+Large+Language+Models+%28Virtual+Token+Counter%2C+VTC%29 12. QoServe: Breaking the Silos of LLM Inference Serving — Goel, Mohan, Kwatra, Anupindi, Ramjee, 2026 https://scholar.google.com/scholar?q=QoServe%3A+Breaking+the+Silos+of+LLM+Inference+Serving 13. SLOs-Serve: Optimized Serving of Multi-SLO LLMs — Chen, Jia, Khan, Krishnamurthy, Gibbons, 2025 https://scholar.google.com/scholar?q=SLOs-Serve%3A+Optimized+Serving+of+Multi-SLO+LLMs 14. JITServe: SLO-aware LLM Serving with Imprecise Request Information — Zhang, Wu, Mu, Ning, Liu, Sarda, Lee, Lai, 2025 https://scholar.google.com/scholar?q=JITServe%3A+SLO-aware+LLM+Serving+with+Imprecise+Request+Information 15. FastServe: Iteration-Level Preemptive Scheduling for LLM Inference (MLFQ) — Wu, Zhong, Zhang, Liu, Liu, Sun, Huang, Liu, Jin, 2026 (NSDI) https://scholar.google.com/scholar?q=FastServe%3A+Iteration-Level+Preemptive+Scheduling+for+LLM+Inference+%28MLFQ%29 16. Mooncake: Trading More Storage for Less Computation — Qin, Li, He, Cui, Ren, Zhang, Wu, Zheng, Xu, 2025 https://scholar.google.com/scholar?q=Mooncake%3A+Trading+More+Storage+for+Less+Computation 17. Tutti: Making SSD-backed KV Cache Practical for Long-Context LLM Serving — Qiu, Hu, Wang, Zhu, Yan, Chen, Xu, Chen, Zhang, 2026 https://scholar.google.com/scholar?q=Tutti%3A+Making+SSD-backed+KV+Cache+Practical+for+Long-Context+LLM+Serving 18. LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference — Liu, Cheng, Yao, An, Chen, Feng, Huang, Shen, Zhang, Du, Jiang, 2025 https://scholar.google.com/scholar?q=LMCache%3A+An+Efficient+KV+Cache+Layer+for+Enterprise-Scale+LLM+Inference 19. Scheduling Algorithms for Multiprogramming in a Hard-Real-Time Environment (and least-laxity-first literature, e.g. Mok 1983) — Liu and Layland (1973); Mok, 1973 / 1983 https://scholar.google.com/scholar?q=Scheduling+Algorithms+for+Multiprogramming+in+a+Hard-Real-Time+Environment+%28and+least-laxity-first+literature%2C+e.g.+Mok+1983%29 20. DistServe: Disaggregating Prefill and Decoding for Goodput-Optimized LLM Serving — Zhong, Liu, Chen, Hu, Zhu, Liu, Jin, Zhang, 2024 https://scholar.google.com/scholar?q=DistServe%3A+Disaggregating+Prefill+and+Decoding+for+Goodput-Optimized+LLM+Serving Interactive Visualization: Cascade: SLO-Aware Latency Budgets for Fair, High-Goodput LLM Serving