Agent Memory Costs: Characterizing Write-Time Overhead in Long-Horizon Workloads

This episode examines a paper that profiles ten agent-memory systems as systems workloads and splits their cost into construction (the write path), retrieval and generation, rather than comparing them on accuracy alone. It covers why pasting full history into a long context fails: prefill cost grows quadratically, prefix caching breaks across sessions, and recall degrades for material in the middle of long inputs. It then explains how agent memory differs from RAG and walks through a seven-stage pipeline and four construction styles, from none to deterministic chunking, LLM-mediated fact extraction and agentic control. The ten systems, including BM25, GraphRAG, Mem0, Letta and MIRIX, are placed along a gradient of increasing LLM involvement at write time. The discussion also lays out the measurement setup (phase-tagged calls aligned with GPU telemetry on a single H100 with Qwen3 models, plus remote OpenAI models) and previews the argument that construction is the hidden bill. Listeners get a concrete view of where long-lived agents spend compute and energy, and why that matters for scheduling. Sources: 1. Agent Memory: Characterization and System Implications of Stateful Long-Horizon Workloads — Yasmine Omri, Ziyu Gan, Zachary Broveak, Robin Geens, Zexue He, Alex Pentland, Marian Verhelst, Tsachy Weissman, Thierry Tambe, 2026 http://arxiv.org/abs/2606.06448 2. MemGPT: Towards LLMs as Operating Systems — Charles Packer, Sarah Wooders, Kevin Lin, Vivian Fang, Shishir G. Patil, Ion Stoica, Joseph E. Gonzalez, 2023 https://scholar.google.com/scholar?q=MemGPT%3A+Towards+LLMs+as+Operating+Systems 3. Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory — Prateek Chhikara, Dev Khant, Saket Aryan, Taranjeet Singh, Deshraj Yadav, 2025 https://scholar.google.com/scholar?q=Mem0%3A+Building+Production-Ready+AI+Agents+with+Scalable+Long-Term+Memory 4. Evaluating Very Long-Term Conversational Memory of LLM Agents (LoCoMo) — Adyasha Maharana, Dong-Ho Lee, Sergey Tulyakov, Mohit Bansal, Francesco Barbieri, Yuwei Fang, 2024 https://scholar.google.com/scholar?q=Evaluating+Very+Long-Term+Conversational+Memory+of+LLM+Agents+%28LoCoMo%29 5. Sleep-time Compute: Beyond Inference Scaling at Test-time — Kevin Lin, Charlie Snell, Yu Wang, Charles Packer, Sarah Wooders, Ion Stoica, Joseph E. Gonzalez, 2025 https://scholar.google.com/scholar?q=Sleep-time+Compute%3A+Beyond+Inference+Scaling+at+Test-time 6. Zep: A Temporal Knowledge Graph Architecture for Agent Memory — Preston Rasmussen, Pavel Paliychuk, Travis Beauvais, Jack Ryan, Daniel Chalef, 2025 https://scholar.google.com/scholar?q=Zep%3A+A+Temporal+Knowledge+Graph+Architecture+for+Agent+Memory 7. LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference — Yihua Cheng et al., 2025 https://scholar.google.com/scholar?q=LMCache%3A+An+Efficient+KV+Cache+Layer+for+Enterprise-Scale+LLM+Inference 8. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — Ruoyu Qin et al., 2024 https://scholar.google.com/scholar?q=Mooncake%3A+A+KVCache-centric+Disaggregated+Architecture+for+LLM+Serving 9. Sarathi-Serve: Taming Throughput-Latency Tradeoff in LLM Inference with Chunked Prefills — Amey Agrawal et al., 2024 https://scholar.google.com/scholar?q=Sarathi-Serve%3A+Taming+Throughput-Latency+Tradeoff+in+LLM+Inference+with+Chunked+Prefills 10. Zep, LightMem, and other efficiency-oriented memory systems (e.g., LightMem: Lightweight and Efficient Memory-Augmented Generation) — Various, 2025 https://scholar.google.com/scholar?q=Zep%2C+LightMem%2C+and+other+efficiency-oriented+memory+systems+%28e.g.%2C+LightMem%3A+Lightweight+and+Efficient+Memory-Augmented+Generation%29 11. Cartridges: Lightweight and General-Purpose Long Context Representations via Self-Study — Sabri Eyuboglu et al., 2025 https://scholar.google.com/scholar?q=Cartridges%3A+Lightweight+and+General-Purpose+Long+Context+Representations+via+Self-Study 12. Efficient Memory Management for Large Language Model Serving with PagedAttention (vLLM) — Woosuk Kwon et al., 2023 https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention+%28vLLM%29 13. Large-scale cluster management at Google with Borg — Abhishek Verma et al., 2015 https://scholar.google.com/scholar?q=Large-scale+cluster+management+at+Google+with+Borg Interactive Visualization: Agent Memory Costs: Characterizing Write-Time Overhead in Long-Horizon Workloads