Cartridges++: Compressed KV Memory Without Off-Context Derailment

This episode examines "Cartridges++," an Apple paper on learned, compressed KV-cache memories of a document (Cartridges) and a failure they cause on questions unrelated to that document. It explains how KV-cache compression works and compares three families: eviction methods like H2O and SnapKV, text summaries, and learned methods such as Cartridges, prefix-tuning and Attention Matching. The central argument is that a Cartridge's training objective only rewards matching the teacher on document-related queries and says nothing about unrelated ones. So the memory that best reproduces the document can derail the model elsewhere, for example by quoting a stored policy document when asked what Saturn is made of. The discussion separates context fidelity from capability preservation and flags interference, meaning fabricated quotes from the document. It argues the fair baseline is the full uncompressed document pasted as text, which already distracts models somewhat. Listeners get a clear account of why scoring a compressed memory on document questions alone hides a real cost, and why the problem looks like catastrophic forgetting applied to cache entries. Sources: 1. Cartridges++: KV Cache Compression without Off-Context Derailment — Sonia Laguna, Joao Monteiro, Marco Cuturi, Pierre Ablin, Eleonora Gualdoni, 2026 http://arxiv.org/abs/2609.35621 2. Cartridges: Lightweight and General-Purpose Long Context Representations via Self-Study — Sabri Eyuboglu, Ryan Ehrlich, Simran Arora, Neel Guha, Dylan Zinsley, Emily Liu, Will Tennien, Atri Rudra, James Zou, Azalia Mirhoseini, Christopher Ré, 2025 https://scholar.google.com/scholar?q=Cartridges%3A+Lightweight+and+General-Purpose+Long+Context+Representations+via+Self-Study 3. Overcoming catastrophic forgetting in neural networks — James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, Raia Hadsell, 2017 https://scholar.google.com/scholar?q=Overcoming+catastrophic+forgetting+in+neural+networks 4. LoRA Learns Less and Forgets Less — Dan Biderman, Jacob Portes, Jose Javier Gonzalez Ortiz, and colleagues (Databricks / Columbia), 2024 https://scholar.google.com/scholar?q=LoRA+Learns+Less+and+Forgets+Less 5. Large Language Models Can Be Easily Distracted by Irrelevant Context — Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed Chi, Nathanael Schärli, Denny Zhou, 2023 https://scholar.google.com/scholar?q=Large+Language+Models+Can+Be+Easily+Distracted+by+Irrelevant+Context 6. Fast KV Compaction via Attention Matching — Adam Zweiger et al., 2026 https://scholar.google.com/scholar?q=Fast+KV+Compaction+via+Attention+Matching 7. Cartridges at Scale — Hardalov et al., 2026 https://scholar.google.com/scholar?q=Cartridges+at+Scale 8. Prefix-Tuning: Optimizing Continuous Prompts for Generation — Xiang Lisa Li, Percy Liang, 2021 https://scholar.google.com/scholar?q=Prefix-Tuning%3A+Optimizing+Continuous+Prompts+for+Generation 9. Efficient Streaming Language Models with Attention Sinks — Guangxuan Xiao et al., 2024 https://scholar.google.com/scholar?q=Efficient+Streaming+Language+Models+with+Attention+Sinks 10. Conformal Risk Control — Anastasios Angelopoulos, Stephen Bates, Adam Fisch, Lihua Lei, Tal Schuster, 2024 https://scholar.google.com/scholar?q=Conformal+Risk+Control 11. Lost in the Middle / Large Language Models Can Be Easily Distracted by Irrelevant Context — Freda Shi et al. (distraction); Nelson Liu et al. (lost in the middle), 2023 https://scholar.google.com/scholar?q=Lost+in+the+Middle+%2F+Large+Language+Models+Can+Be+Easily+Distracted+by+Irrelevant+Context 12. Training plug-and-play knowledge modules with deep context distillation — Lucas Caccia, Alan Ansell, Edoardo Ponti, Ivan Vulić, Alessandro Sordoni, 2025 https://scholar.google.com/scholar?q=Training+plug-and-play+knowledge+modules+with+deep+context+distillation 13. Doc-to-LoRA: Learning to instantly internalize contexts — Rujikorn Charakorn, Edoardo Cetin, Shinnosuke Uesaka, Robert Tjarko Lange, 2026 https://scholar.google.com/scholar?q=Doc-to-LoRA%3A+Learning+to+instantly+internalize+contexts 14. The pitfalls of KV cache compression — Alex Chen, Renato Geh, Aditya Grover, Guy Van den Broeck, Daniel Israel, 2026 https://scholar.google.com/scholar?q=The+pitfalls+of+KV+cache+compression 15. KV-Distill: Nearly lossless learnable context compression for LLMs — Vivek Chari, Guanghui Qin, Benjamin Van Durme, 2025 https://scholar.google.com/scholar?q=KV-Distill%3A+Nearly+lossless+learnable+context+compression+for+LLMs Interactive Visualization: Cartridges++: Compressed KV Memory Without Off-Context Derailment