Context Language Models: Letting Models Edit Their Own Context

This episode explores Context Language Models, a proposal that lets a model write directly to its own context as an editable file instead of relying on human-designed harnesses. It first surveys the existing approaches: fixed-policy compaction, action-based tools such as Context Folding, and recursive language models. It then shows why they fall short on tasks like in-place Sudoku edits, using a diagnostic benchmark that varies context pressure. The discussion explains why edits to the middle of a context break prefix caching and force costly recomputation, and why the paper measures cost in prefix-reuse FLOPs. It also lays out the paper's five claims: zero-shot gains on deep research and coding, swarms of agents with file-based contexts, evolved natural-language skills, online reinforcement learning, and a serving change called Suffix Cache Reuse. The hosts stress the difference between percentage points and relative percent, and they note that the strongest evidence is zero-shot with a large model while the training evidence is thinner. Sources: 1. Context Language Models — Rulin Shao, Shannon Zejiang Shen, Junjie Oscar Yin, Yuetai Li, Minheng Wang, Hamish Ivison, Radha Poovendran, Nathan Lambert, Teng Xiao, Mike Lewis, Wen-tau Yih, Luke Zettlemoyer, Pang Wei Koh, 2026 http://arxiv.org/abs/2609.37725 2. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (introduces GRPO) — Zhihong Shao, Peiyi Wang, Qihao Zhu, et al. (DeepSeek-AI), 2024 https://scholar.google.com/scholar?q=DeepSeekMath%3A+Pushing+the+Limits+of+Mathematical+Reasoning+in+Open+Language+Models+%28introduces+GRPO%29 3. MEM1: Learning to Synergize Memory and Reasoning for Efficient Long-Horizon Agents — Zijian Zhou et al., 2025 https://scholar.google.com/scholar?q=MEM1%3A+Learning+to+Synergize+Memory+and+Reasoning+for+Efficient+Long-Horizon+Agents 4. Scaling Long-Horizon LLM Agent via Context-Folding — Weiwei Sun et al. (ByteDance Seed / CMU), 2025 https://scholar.google.com/scholar?q=Scaling+Long-Horizon+LLM+Agent+via+Context-Folding 5. BrowseComp-Plus: A More Fair and Transparent Evaluation Benchmark of Deep-Research Agent — Zijian Chen, Xueguang Ma, Shengyao Zhuang, et al. (Waterloo and collaborators), 2025 https://scholar.google.com/scholar?q=BrowseComp-Plus%3A+A+More+Fair+and+Transparent+Evaluation+Benchmark+of+Deep-Research+Agent 6. MemGPT: Towards LLMs as Operating Systems — Charles Packer, Sarah Wooders, Kevin Lin, Vivian Fang, Shishir G. Patil, Ion Stoica, Joseph E. Gonzalez, 2023 https://scholar.google.com/scholar?q=MemGPT%3A+Towards+LLMs+as+Operating+Systems 7. How we built our multi-agent research system (Anthropic engineering report) / Effective context engineering for AI agents — Anthropic engineering team, 2025 https://scholar.google.com/scholar?q=How+we+built+our+multi-agent+research+system+%28Anthropic+engineering+report%29+%2F+Effective+context+engineering+for+AI+agents 8. Don't Build Multi-Agents (Cognition AI blog post) — Walden Yan (Cognition AI), 2025 https://scholar.google.com/scholar?q=Don%27t+Build+Multi-Agents+%28Cognition+AI+blog+post%29 9. Agentic Context Engineering: Evolving Contexts for Self-Improving Language Models (ACE) — Qizheng Zhang et al. (Stanford, SambaNova and collaborators), 2025 https://scholar.google.com/scholar?q=Agentic+Context+Engineering%3A+Evolving+Contexts+for+Self-Improving+Language+Models+%28ACE%29 10. ReSum: Unlocking Long-Horizon Search Intelligence via Context Summarization — Wu et al., 2025 https://scholar.google.com/scholar?q=ReSum%3A+Unlocking+Long-Horizon+Search+Intelligence+via+Context+Summarization 11. Training Language Models to Reason Efficiently — Arora and Zanette, 2025 (NeurIPS) https://scholar.google.com/scholar?q=Training+Language+Models+to+Reason+Efficiently 12. Think Dense, Not Long: Dynamic Decoupled Conditional Advantage for Efficient Reasoning (DDCA) — Peng et al., 2026 https://scholar.google.com/scholar?q=Think+Dense%2C+Not+Long%3A+Dynamic+Decoupled+Conditional+Advantage+for+Efficient+Reasoning+%28DDCA%29 13. CompactionRL: Reinforcement Learning with Context Compaction for Long-Horizon Agents — Li, Hou, Jing, Tang, Dong, 2026 https://scholar.google.com/scholar?q=CompactionRL%3A+Reinforcement+Learning+with+Context+Compaction+for+Long-Horizon+Agents 14. Memento: Teaching LLMs to Manage Their Own Context — Kontonis et al., 2026 https://scholar.google.com/scholar?q=Memento%3A+Teaching+LLMs+to+Manage+Their+Own+Context 15. CacheBlend: Fast LLM Serving for RAG with Cached Knowledge Fusion; EPIC: Efficient Position-Independent Caching — Yao et al.; Hu et al., 2025 https://scholar.google.com/scholar?q=CacheBlend%3A+Fast+LLM+Serving+for+RAG+with+Cached+Knowledge+Fusion%3B+EPIC%3A+Efficient+Position-Independent+Caching 16. Recursive Language Models — Zhang, Kraska, Khattab, 2025 https://scholar.google.com/scholar?q=Recursive+Language+Models 17. Sculptor: Empowering LLMs with Cognitive Agency via Active Context Management — Li et al., 2026 https://scholar.google.com/scholar?q=Sculptor%3A+Empowering+LLMs+with+Cognitive+Agency+via+Active+Context+Management 18. Self-Generated Prompt Injections in Compaction Summaries — OpenAI, 2026 https://scholar.google.com/scholar?q=Self-Generated+Prompt+Injections+in+Compaction+Summaries 19. GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning — Agrawal et al., 2026 https://scholar.google.com/scholar?q=GEPA%3A+Reflective+Prompt+Evolution+Can+Outperform+Reinforcement+Learning 20. Meta-Harness: End-to-End Optimization of Model Harnesses — Lee, Nair, Zhang, Lee, Khattab, Finn, 2026 https://scholar.google.com/scholar?q=Meta-Harness%3A+End-to-End+Optimization+of+Model+Harnesses Interactive Visualization: Context Language Models: Letting Models Edit Their Own Context