FlatCake: Flat KVCache Storage Beats Tiered Memory Hierarchies

This episode examines FlatCake, a proposed KV cache storage architecture for LLM serving that drops the usual GPU, host DRAM, and NVMe SSD hierarchy. Instead of migrating blocks between tiers, it decides at write time whether each block lives in DRAM or on SSD, and it never moves the block afterward. The discussion explains why the authors think tiering breaks down. Reuse is bimodal: in several conversational and API traces, most blocks are used only once. In a Claude Code agentic trace, sessions are so large that a few concurrent ones overflow DRAM. Staging buffers also run out under load, and blocks written in eviction order end up scattered across the drive. The hosts walk through the proposed fix, which lays each session out contiguously on the SSD so a restore becomes a few large reads, and they note that GPUDirect Storage fixes the bounce buffer but not the scatter. They also point out the cost of the approach: placement relies on hand-tuned thresholds with no published values, and a wrong placement choice sticks. Listeners interested in serving infrastructure and storage layout get a critical look at how much the paper's experiments actually test its claims. Sources: 1. FlatCake: A Flat KVCache Storage Architecture for LLM Serving — Xin Huang, Teng Ma, Zhuoran Li, Yibo Zhao, Zhaoshi Li, Qianchao Zhu, Wei Yang, Chenxi Wang, Tao Xie, Youwei Zhuo, 2026 http://arxiv.org/abs/2610.38355 2. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — Ruoyu Qin, Zheming Li, Weiran He, Mingxing Zhang, Yongwei Wu, Weimin Zheng, Xinran Xu (Moonshot AI / Tsinghua), 2024 (FAST 2025) https://scholar.google.com/scholar?q=Mooncake%3A+A+KVCache-centric+Disaggregated+Architecture+for+LLM+Serving 3. Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention (AttentionStore) — Bin Gao, Zhuomin He, Puru Sharma, Qingxuan Kang, Djordje Jevdjic, Junbo Deng, Xingkun Yang, Zhou Yu, Pengfei Zuo, 2024 (USENIX ATC 2024) https://scholar.google.com/scholar?q=Cost-Efficient+Large+Language+Model+Serving+for+Multi-turn+Conversations+with+CachedAttention+%28AttentionStore%29 4. LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference — Yihua Cheng, Yuhan Liu, Jiayi Yao, Yuwei An, Xiaokun Chen, Shaoting Feng, Yuyang Huang, Samuel Shen, Kuntai Du, Junchen Jiang et al., 2025 https://scholar.google.com/scholar?q=LMCache%3A+An+Efficient+KV+Cache+Layer+for+Enterprise-Scale+LLM+Inference 5. FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU — Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Ré, Ion Stoica, Ce Zhang, 2023 (ICML 2023) https://scholar.google.com/scholar?q=FlexGen%3A+High-Throughput+Generative+Inference+of+Large+Language+Models+with+a+Single+GPU 6. ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning — Samyam Rajbhandari, Olatunji Ruwase, Jeff Rasley, Shaden Smith, Yuxiong He, 2021 (SC 2021) https://scholar.google.com/scholar?q=ZeRO-Infinity%3A+Breaking+the+GPU+Memory+Wall+for+Extreme+Scale+Deep+Learning 7. GPU-Initiated On-Demand High-Throughput Storage Access in the BaM System Architecture — Zaid Qureshi, Vikram Sharma Mailthody, Isaac Gelado, Seung Won Min, Amna Masood, Jeongmin Park, Jinjun Xiong, C. J. Newburn, Dmitri Vainbrand, I-Hsin Chung, Michael Garland, William Dally, Wen-mei Hwu, 2023 (ASPLOS 2023) https://scholar.google.com/scholar?q=GPU-Initiated+On-Demand+High-Throughput+Storage+Access+in+the+BaM+System+Architecture 8. GIDS: Accelerating Sampling and Aggregation Operations in GNN Frameworks with GPU Initiated Direct Storage Accesses — Jeongmin Brian Park, Vikram Sharma Mailthody, Zaid Qureshi, Wen-mei Hwu, 2024 (PVLDB 2024) https://scholar.google.com/scholar?q=GIDS%3A+Accelerating+Sampling+and+Aggregation+Operations+in+GNN+Frameworks+with+GPU+Initiated+Direct+Storage+Accesses 9. KVCache Cache in the Wild: Characterizing and Optimizing KVCache Cache at a Large Cloud Provider — Jiahao Wang, Jinbo Han, Xingda Wei, Sijie Shen, Dingyan Zhang, Chenguang Fang, Rong Chen, Wenyuan Yu, Haibo Chen, 2025 (USENIX ATC 2025) https://scholar.google.com/scholar?q=KVCache+Cache+in+the+Wild%3A+Characterizing+and+Optimizing+KVCache+Cache+at+a+Large+Cloud+Provider 10. Preble: Efficient Distributed Prompt Scheduling for LLM Serving — Vikranth Srivastava, Zijian He, Shuowei Jin (Yingxuan Jin?), Yilong Zhao, Yibo Zhu, Ion Stoica, Hao Zhang and others (UCSD / Berkeley), 2024 (ICLR 2025) https://scholar.google.com/scholar?q=Preble%3A+Efficient+Distributed+Prompt+Scheduling+for+LLM+Serving 11. FIFO Queues are All You Need for Cache Eviction — Juncheng Yang, Yazhuo Zhang, Ziyue Qiu, Yao Yue, Rashmi Vinayak, 2023 (SOSP 2023) https://scholar.google.com/scholar?q=FIFO+Queues+are+All+You+Need+for+Cache+Eviction 12. CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving — Yuhan Liu, Hanchen Li, Yihua Cheng, et al., 2024 https://scholar.google.com/scholar?q=CacheGen%3A+KV+Cache+Compression+and+Streaming+for+Fast+Large+Language+Model+Serving 13. CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion — Jiayi Yao, Hanchen Li, Yuhan Liu, et al., 2025 https://scholar.google.com/scholar?q=CacheBlend%3A+Fast+Large+Language+Model+Serving+for+RAG+with+Cached+Knowledge+Fusion 14. AttentionStore / CachedAttention: Cost-Efficient LLM Serving for Multi-turn Conversations — Bin Gao, Zhuomin He, Puru Sharma, et al., 2024 https://scholar.google.com/scholar?q=AttentionStore+%2F+CachedAttention%3A+Cost-Efficient+LLM+Serving+for+Multi-turn+Conversations 15. Managing Scalable Direct Storage Accesses for GPUs with GoFS — Shaobo Li, Yirui Eric Zhou, Yuqi Xue, Yuan Xu, Jian Huang, 2025 https://scholar.google.com/scholar?q=Managing+Scalable+Direct+Storage+Accesses+for+GPUs+with+GoFS 16. Bidaw: Enhancing Key-Value Caching for Interactive LLM Serving via Bidirectional Computation–Storage Awareness — Shipeng Hu, Guangyan Zhang, et al., 2026 https://scholar.google.com/scholar?q=Bidaw%3A+Enhancing+Key-Value+Caching+for+Interactive+LLM+Serving+via+Bidirectional+Computation%E2%80%93Storage+Awareness 17. Tutti: GPU-initiated io_uring queues for KV storage — Cited as [28] in the paper, 2025–2026 https://scholar.google.com/scholar?q=Tutti%3A+GPU-initiated+io_uring+queues+for+KV+storage Interactive Visualization: FlatCake: Flat KVCache Storage Beats Tiered Memory Hierarchies