Test-Time Training via Context Distillation: Learning What to Remember

This episode examines "Learning What to Remember: Test-Time Training via Context Distillation" (TTCD), which asks what a model should write into a fixed-size memory when it can't keep everything. It starts with why linear-attention and recurrent models like DeltaNet and Gated DeltaNet need lossy bounded state. It then traces the progression of test-time training from the delta rule through TTT layers, Titans, LaCT, and In-Place TTT, which reuses a transformer's MLP down-projection as fast weights. The paper's method runs the same frozen backbone twice, once with a long sliding window as teacher and once with a short window as student. It writes the residual between their MLP outputs into the fast weights through an outer-product update, so the information outlives the short window. The discussion stays skeptical. It asks whether the residual really captures remote history or just a modest window extension, whether the distillation loss actually shapes the writes, and why the closest next-token competitor, TTT-E2E, is left out of the comparisons. Sources: 1. Learning What to Remember: Test-Time Training via Context Distillation — Zixuan Wang, Xingyu Dang, Rui-Jie Zhu, Zixin Wen, Hengyu Fu, Wenhao Chai, Jason D. Lee, 2026 http://arxiv.org/abs/2608.01672 2. Learning to (Learn at Test Time): RNNs with Expressive Hidden States — Yu Sun, Xinhao Li, Karan Dalal, Jiarui Xu, Arjun Vikram, Genghan Zhang, Yann Dubois, Xinlei Chen, Xiaolong Wang, Sanmi Koyejo, Tatsunori Hashimoto, Carlos Guestrin, 2024 (ICML 2025 version also circulated) https://scholar.google.com/scholar?q=Learning+to+%28Learn+at+Test+Time%29%3A+RNNs+with+Expressive+Hidden+States 3. Gated Delta Networks: Improving Mamba2 with Delta Rule — Songlin Yang, Jan Kautz, Ali Hatamizadeh, 2024 (ICLR 2025) https://scholar.google.com/scholar?q=Gated+Delta+Networks%3A+Improving+Mamba2+with+Delta+Rule 4. Titans: Learning to Memorize at Test Time — Ali Behrouz, Peilin Zhong, Vahab Mirrokni, 2024 https://scholar.google.com/scholar?q=Titans%3A+Learning+to+Memorize+at+Test+Time 5. A General Method for Test-Time Training / Learning by Distilling Context — Snell, Klein, Zhong (Learning by Distilling Context); Askell et al. (A General Language Assistant as a Laboratory for Alignment), 2022 (Snell et al.); 2021 (Askell et al.) https://scholar.google.com/scholar?q=A+General+Method+for+Test-Time+Training+%2F+Learning+by+Distilling+Context 6. The Surprising Effectiveness of Test-Time Training for Few-Shot Learning — Ekin Akyürek, Mehul Damani, Linlu Qiu, Han Guo, Yoon Kim, Jacob Andreas, 2024 https://scholar.google.com/scholar?q=The+Surprising+Effectiveness+of+Test-Time+Training+for+Few-Shot+Learning 7. End-to-End Test-Time Training for Long Context (TTT-E2E) — Arnuv Tandon, Karan Dalal, Xinhao Li, et al., 2025 https://scholar.google.com/scholar?q=End-to-End+Test-Time+Training+for+Long+Context+%28TTT-E2E%29 8. In-Place Test-Time Training — Guhao Feng, Shengjie Luo, Kai Hua, Ge Zhang, Wenhao Huang, Di He, Tianle Cai, 2026 https://scholar.google.com/scholar?q=In-Place+Test-Time+Training 9. Test-Time Training Done Right (LaCT) — Tianyuan Zhang, Sai Bi, Yicong Hong, et al., 2025 https://scholar.google.com/scholar?q=Test-Time+Training+Done+Right+%28LaCT%29 10. Learning by Distilling Context — Charlie Snell, Dan Klein, Ruiqi Zhong, 2022 https://scholar.google.com/scholar?q=Learning+by+Distilling+Context 11. Cartridges: Lightweight and general-purpose long context representations via self-study — Sabri Eyuboglu, Ryan Ehrlich, Simran Arora, et al., 2025 https://scholar.google.com/scholar?q=Cartridges%3A+Lightweight+and+general-purpose+long+context+representations+via+self-study 12. Kimi Linear: An Expressive, Efficient Attention Architecture — Kimi Team, Yu Zhang, et al., 2025 https://scholar.google.com/scholar?q=Kimi+Linear%3A+An+Expressive%2C+Efficient+Attention+Architecture 13. Linear Transformers Are Secretly Fast Weight Programmers — Imanol Schlag, Kazuki Irie, Jürgen Schmidhuber, 2021 https://scholar.google.com/scholar?q=Linear+Transformers+Are+Secretly+Fast+Weight+Programmers 14. Infini-attention: Leave No Context Behind — Tsendsuren Munkhdalai, Manaal Faruqui, Siddharth Gopal, 2024 https://scholar.google.com/scholar?q=Infini-attention%3A+Leave+No+Context+Behind 15. RULER: What's the Real Context Size of Your Long-Context Language Models? — Cheng-Ping Hsieh, et al., 2024 https://scholar.google.com/scholar?q=RULER%3A+What%27s+the+Real+Context+Size+of+Your+Long-Context+Language+Models%3F 16. Mechanistic Design and Scaling of Hybrid Architectures (MAD) — Michael Poli, et al., 2024 https://scholar.google.com/scholar?q=Mechanistic+Design+and+Scaling+of+Hybrid+Architectures+%28MAD%29 Interactive Visualization: Test-Time Training via Context Distillation: Learning What to Remember