Agentic Coding in the Wild: GitHub Copilot's Cache Behavior at Scale

This episode examines a production-scale characterization of GitHub Copilot's coding agent, built on sampled client-side telemetry from one week in June 2026 (13.5 million sessions and 760 million LLM calls, though some totals are inconsistent). It argues that agent traffic is session-shaped rather than stateless chat. About 87% of LLM calls are agent-initiated, the median call carries roughly 68,000 prompt tokens to produce about 247 output tokens, and the human appears only at turn boundaries. The discussion explains how prefix caching should give very high reuse inside a turn, and why the cache is lost after a pause, a model switch, or context compaction. It also separates "container idle" from "KV-cache idle", which suggests turn boundaries are the natural point to reclaim resources. The episode places the work alongside prior agentic KV-retention research and serverless idle-time studies. Listeners get a clear framework for why request-level scheduling and plain LRU eviction fall short for agent workloads, along with caveats about inferring server behavior from client data alone. Sources: 1. Agentic Coding in the Wild: Characterizing GitHub Copilot Traces at Production Scale — Banruo Liu, Haoran Qiu, Íñigo Goiri, Rodrigo Fonseca, Ricardo Bianchini, Esha Choukse, 2026 http://arxiv.org/abs/2608.00101v1 2. SWE-bench: Can Language Models Resolve Real-World GitHub Issues? — Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, Karthik Narasimhan, 2023 (ICLR 2024) https://scholar.google.com/scholar?q=SWE-bench%3A+Can+Language+Models+Resolve+Real-World+GitHub+Issues%3F 3. Splitwise: Efficient Generative LLM Inference Using Phase Splitting — Pratyush Patel, Esha Choukse, Chaojie Zhang, Íñigo Goiri, et al. (Microsoft), 2023 (ISCA 2024) https://scholar.google.com/scholar?q=Splitwise%3A+Efficient+Generative+LLM+Inference+Using+Phase+Splitting 4. BurstGPT: A Real-world Workload Dataset to Optimize LLM Serving Systems — Yuxin Wang, Yuhan Chen, Zeyu Li, et al., 2024 https://scholar.google.com/scholar?q=BurstGPT%3A+A+Real-world+Workload+Dataset+to+Optimize+LLM+Serving+Systems 5. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — Ruoyu Qin, Zheng Li, Weiran He, et al. (Moonshot AI), 2024 (FAST 2025 best paper) https://scholar.google.com/scholar?q=Mooncake%3A+A+KVCache-centric+Disaggregated+Architecture+for+LLM+Serving 6. MemGPT: Towards LLMs as Operating Systems — Charles Packer, Sarah Wooders, Kevin Lin, Vivian Fang, Shishir G. Patil, Ion Stoica, Joseph E. Gonzalez, 2023 https://scholar.google.com/scholar?q=MemGPT%3A+Towards+LLMs+as+Operating+Systems 7. Lost in the Middle: How Language Models Use Long Contexts — Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, Percy Liang, 2023 (TACL 2024) https://scholar.google.com/scholar?q=Lost+in+the+Middle%3A+How+Language+Models+Use+Long+Contexts 8. LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models — Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, Lili Qiu (Microsoft), 2023 (EMNLP) https://scholar.google.com/scholar?q=LLMLingua%3A+Compressing+Prompts+for+Accelerated+Inference+of+Large+Language+Models 9. Efficient Memory Management for Large Language Model Serving with PagedAttention — Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, et al., 2023 (SOSP) https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention 10. Serverless in the Wild: Characterizing and Optimizing the Serverless Workload at a Large Cloud Provider — Mohammad Shahrad, Rodrigo Fonseca, Íñigo Goiri, Gohar Chaudhry, Paul Batum, Jason Cooke, Eduardo Laureano, Colby Tresness, Mark Russinovich, Ricardo Bianchini, 2020 (USENIX ATC) https://scholar.google.com/scholar?q=Serverless+in+the+Wild%3A+Characterizing+and+Optimizing+the+Serverless+Workload+at+a+Large+Cloud+Provider 11. Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Live — Hanchen Li, Qiuyang Mang, Runyuan He, Qizheng Zhang, Huanzhi Mao, et al., 2025 https://scholar.google.com/scholar?q=Continuum%3A+Efficient+and+Robust+Multi-Turn+LLM+Agent+Scheduling+with+KV+Cache+Time-to-Live 12. FaaSCache: Keeping Serverless Computing Alive with Greedy-Dual Caching — Alexander Fuerst, Prateek Sharma, 2021 (ASPLOS) https://scholar.google.com/scholar?q=FaaSCache%3A+Keeping+Serverless+Computing+Alive+with+Greedy-Dual+Caching 13. Efficient Agentic LLM Inference with KV Cache Retention (the 'CacheTTL'/Continuum line of work, ref [19]) — Hanchen Li, Azalia Mirhoseini, Ion Stoica, 2025 https://scholar.google.com/scholar?q=Efficient+Agentic+LLM+Inference+with+KV+Cache+Retention+%28the+%27CacheTTL%27%2FContinuum+line+of+work%2C+ref+%5B19%5D%29 14. Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention — Bin Gao, Zhuomin He, Puru Sharma, et al., 2024 https://scholar.google.com/scholar?q=Cost-Efficient+Large+Language+Model+Serving+for+Multi-turn+Conversations+with+CachedAttention 15. InferCept: Efficient Intercept Support for Augmented Large Language Model Inference — Reyna Abhyankar, Zijian He, Vikranth Srivatsa, Hao Zhang, Yiying Zhang, 2024 https://scholar.google.com/scholar?q=InferCept%3A+Efficient+Intercept+Support+for+Augmented+Large+Language+Model+Inference 16. Pensieve: Retention of Past Key-Values for Stateful LLM Serving (Fast and Expressive Multi-Turn Inference with Stateful Serving) — Lingfan Yu, Jinyang Li, et al., 2023/2025 https://scholar.google.com/scholar?q=Pensieve%3A+Retention+of+Past+Key-Values+for+Stateful+LLM+Serving+%28Fast+and+Expressive+Multi-Turn+Inference+with+Stateful+Serving%29 17. Preble: Efficient Distributed Prompt Scheduling for LLM Serving — Vikranth Srivatsa, Zijian He, Reyna Abhyankar, Dongming Li, Yiying Zhang, 2024 https://scholar.google.com/scholar?q=Preble%3A+Efficient+Distributed+Prompt+Scheduling+for+LLM+Serving 18. H2O, SnapKV and StreamingLLM (KV eviction/compression family) — Zhang et al.; Li et al.; Xiao et al., 2023–2024 https://scholar.google.com/scholar?q=H2O%2C+SnapKV+and+StreamingLLM+%28KV+eviction%2Fcompression+family%29 19. Cartridges: Lightweight and general-purpose long context representations via self-study — Sabri Eyuboglu, Ryan Ehrlich, Simran Arora, et al., 2025 https://scholar.google.com/scholar?q=Cartridges%3A+Lightweight+and+general-purpose+long+context+representations+via+self-study 20. TraceLab: Characterizing Coding Agent Workloads for LLM Serving, and Agentic AI Workload Characteristics — Kan Zhu et al. (arXiv:2606.30560); Yichao Yuan et al. (arXiv:2605.26297), 2026 https://scholar.google.com/scholar?q=TraceLab%3A+Characterizing+Coding+Agent+Workloads+for+LLM+Serving%2C+and+Agentic+AI+Workload+Characteristics 21. Marconi: Prefix Caching for the Era of Hybrid LLMs — Rui Pan, Zhuang Wang, et al., 2024/2025 https://scholar.google.com/scholar?q=Marconi%3A+Prefix+Caching+for+the+Era+of+Hybrid+LLMs Interactive Visualization: Agentic Coding in the Wild: GitHub Copilot's Cache Behavior at Scale