Latent Compaction with Test-Time Training: Learning Compact KV Caches

This episode examines "Latent Compaction with Test-Time Training," a preprint that asks whether a frozen model can shrink a document's KV cache by 16 to 32 times by running gradient descent on a small set of new key-value entries. The compaction happens before any questions are known. The discussion explains why the cache bottlenecks long-context memory and batching, and why text summaries lose details like file paths or IDs. It then contrasts the approach with eviction methods such as H2O, SnapKV, PyramidKV and KVZip, which can only select existing entries and never create blended ones. It also places the work among prefix-tuning, Cartridges, Attention Matching and other test-time training approaches, and clarifies that the "32x" figure counts cache slots, not bytes, with no measured speedup or memory savings. Listeners get a clear primer on query-aware versus query-agnostic compression, along with a skeptical look at what the headline RULER result does and doesn't show. Sources: 1. Latent Compaction with Test-Time Training — Yingfa Chen, Xingyu Shen, Chaojun Xiao, Xu Han, 2026 http://arxiv.org/abs/2610.20582v1 2. Test-Time Training with Self-Supervision for Generalization under Distribution Shifts — Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei A. Efros, Moritz Hardt, 2020 https://scholar.google.com/scholar?q=Test-Time+Training+with+Self-Supervision+for+Generalization+under+Distribution+Shifts 3. Learning to (Learn at Test Time): RNNs with Expressive Hidden States — Yu Sun, Xinhao Li, Karan Dalal, Jiarui Xu, Arjun Vikram, Genghan Zhang, Yann Dubois, Xinlei Chen, Xiaolong Wang, Sanmi Koyejo, Tatsunori Hashimoto, Carlos Guestrin, 2024 https://scholar.google.com/scholar?q=Learning+to+%28Learn+at+Test+Time%29%3A+RNNs+with+Expressive+Hidden+States 4. The Surprising Effectiveness of Test-Time Training for Few-Shot Learning — Ekin Akyürek, Mehul Damani, Linlu Qiu, Han Guo, Yoon Kim, Jacob Andreas, 2024 https://scholar.google.com/scholar?q=The+Surprising+Effectiveness+of+Test-Time+Training+for+Few-Shot+Learning 5. Titans: Learning to Memorize at Test Time — Ali Behrouz, Peilin Zhong, Vahab Mirrokni, 2024 https://scholar.google.com/scholar?q=Titans%3A+Learning+to+Memorize+at+Test+Time 6. H2O: Heavy-Hitter Oracle: Efficient Generative Inference of Large Language Models with Heavy Hitters — Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Zhangyang Wang, Beidi Chen, et al., 2023 https://scholar.google.com/scholar?q=H2O%3A+Heavy-Hitter+Oracle%3A+Efficient+Generative+Inference+of+Large+Language+Models+with+Heavy+Hitters 7. SnapKV: LLM Knows What You are Looking for Before Generation — Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, Deming Chen, 2024 https://scholar.google.com/scholar?q=SnapKV%3A+LLM+Knows+What+You+are+Looking+for+Before+Generation 8. KVzip: Query-Agnostic KV Cache Compression with Context Reconstruction — Jang-Hyun Kim, Jinuk Kim, Sangwoo Kwon, Jae W. Lee, Sangdoo Yun, Hyun Oh Song, 2025 https://scholar.google.com/scholar?q=KVzip%3A+Query-Agnostic+KV+Cache+Compression+with+Context+Reconstruction 9. Learning to Compress Prompts with Gist Tokens — Jesse Mu, Xiang Lisa Li, Noah Goodman, 2023 https://scholar.google.com/scholar?q=Learning+to+Compress+Prompts+with+Gist+Tokens 10. Cartridges: Lightweight and General-Purpose Long Context Representations via Self-Study — Sabri Eyuboglu, Ryan Ehrlich, Simran Arora, Neel Guha, Dylan Zinsley, Emily Liu, Will Tennien, Atri Rudra, James Zou, Azalia Mirhoseini, Christopher Ré, 2025 https://scholar.google.com/scholar?q=Cartridges%3A+Lightweight+and+General-Purpose+Long+Context+Representations+via+Self-Study 11. Fast KV Compaction via Attention Matching — Zweiger et al., 2026 https://scholar.google.com/scholar?q=Fast+KV+Compaction+via+Attention+Matching 12. The Power of Scale for Parameter-Efficient Prompt Tuning — Brian Lester, Rami Al-Rfou, Noah Constant, 2021 https://scholar.google.com/scholar?q=The+Power+of+Scale+for+Parameter-Efficient+Prompt+Tuning 13. Learning by Distilling Context — Charlie Snell, Dan Klein, Ruiqi Zhong, 2022 https://scholar.google.com/scholar?q=Learning+by+Distilling+Context 14. KVzap: Fast, adaptive, and faithful KV cache pruning — Jégou, Jeblick, 2026 https://scholar.google.com/scholar?q=KVzap%3A+Fast%2C+adaptive%2C+and+faithful+KV+cache+pruning 15. End-to-End Test-Time Training for Long Context — Tandon, Dalal, Li, Koceja, Rød, Buchanan, Wang, Leskovec, Koyejo, Hashimoto, Guestrin, McCaleb, Choi, Sun, 2025 https://scholar.google.com/scholar?q=End-to-End+Test-Time+Training+for+Long+Context 16. Prefix-Tuning: Optimizing Continuous Prompts for Generation — Li, Liang, 2021 https://scholar.google.com/scholar?q=Prefix-Tuning%3A+Optimizing+Continuous+Prompts+for+Generation 17. Adapting Language Models to Compress Contexts (AutoCompressors) — Chevalier, Wettig, Ajith, Chen, 2023 https://scholar.google.com/scholar?q=Adapting+Language+Models+to+Compress+Contexts+%28AutoCompressors%29 18. In-context Autoencoder for Context Compression in a Large Language Model — Ge, Hu, Wang, Wang, Chen, Wei, 2024 https://scholar.google.com/scholar?q=In-context+Autoencoder+for+Context+Compression+in+a+Large+Language+Model 19. Repeat After Me: Transformers are Better than State Space Models at Copying — Jelassi, Brandfonbrener, Kakade, Malach, 2024 https://scholar.google.com/scholar?q=Repeat+After+Me%3A+Transformers+are+Better+than+State+Space+Models+at+Copying 20. Efficient Streaming Language Models with Attention Sinks (StreamingLLM) — Xiao, Tian, Chen, Han, Lewis, 2024 https://scholar.google.com/scholar?q=Efficient+Streaming+Language+Models+with+Attention+Sinks+%28StreamingLLM%29 21. RULER: What's the Real Context Size of Your Long-Context Language Models? — Hsieh, Kriman, Acharya, Rekesh, Jia, Zhang, Ginsburg and others, 2024 https://scholar.google.com/scholar?q=RULER%3A+What%27s+the+Real+Context+Size+of+Your+Long-Context+Language+Models%3F 22. Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation — Feng, Lv, Cao, Xie, Zhou, 2025 https://scholar.google.com/scholar?q=Ada-KV%3A+Optimizing+KV+Cache+Eviction+by+Adaptive+Budget+Allocation 23. KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache — Liu, Yuan, Jin, Zhong, Xu, Braverman, Chen, Hu, 2024 https://scholar.google.com/scholar?q=KIVI%3A+A+Tuning-Free+Asymmetric+2bit+Quantization+for+KV+Cache Interactive Visualization: Latent Compaction with Test-Time Training: Learning Compact KV Caches