Sliding-Window Attention Beats Linear Attention Without Any Training

This episode examines a paper claiming that a pretrained LLM can be cut down to a sliding window of the last 64 tokens plus four "attention sink" starting tokens, with no training, and still keep about 99% of its benchmark score. That result would match months of work on retrofitting models into linear attention. The discussion builds up the background: the KV cache and how eviction differs from fixed-size linear-attention state, why StreamingLLM's sinks keep windowing from collapsing, and how retrofit costs range from roughly 20 million to 100 billion training tokens. It then questions what "99% recovery" means, noting that log-likelihood multiple-choice benchmarks like MMLU and HellaSwag mostly need only nearby context, unlike generative tasks such as GSM8K and HumanEval. It also asks whether the comparison is fair when memory budgets aren't held equal. Listeners get a skeptical look at whether a simple free baseline is outperforming retrofit methods, or whether the benchmarks are just too local to tell. Sources: 1. Sliding-window beats linear attention — Alexia Jolicoeur-Martineau, Rhea Sanjay Sukthanker, Pashmina Cameron, Emy Gervais, 2026 http://arxiv.org/abs/2608.28444 2. Efficient Streaming Language Models with Attention Sinks — Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, Mike Lewis, 2023 (ICLR 2024) https://scholar.google.com/scholar?q=Efficient+Streaming+Language+Models+with+Attention+Sinks 3. Longformer: The Long-Document Transformer — Iz Beltagy, Matthew E. Peters, Arman Cohan, 2020 https://scholar.google.com/scholar?q=Longformer%3A+The+Long-Document+Transformer 4. Mistral 7B — Albert Q. Jiang et al. (Mistral AI), 2023 https://scholar.google.com/scholar?q=Mistral+7B 5. Gemma 2: Improving Open Language Models at a Practical Size (and Gemma 3 Technical Report) — Gemma Team, Google DeepMind, 2024 / 2025 https://scholar.google.com/scholar?q=Gemma+2%3A+Improving+Open+Language+Models+at+a+Practical+Size+%28and+Gemma+3+Technical+Report%29 6. Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention — Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, François Fleuret, 2020 https://scholar.google.com/scholar?q=Transformers+are+RNNs%3A+Fast+Autoregressive+Transformers+with+Linear+Attention 7. Finetuning Pretrained Transformers into RNNs — Jungo Kasai, Hao Peng, Yizhe Zhang, Dani Yogatama, Gabriel Ilharco, Nikolaos Pappas, Yi Mao, Weizhu Chen, Noah A. Smith, 2021 https://scholar.google.com/scholar?q=Finetuning+Pretrained+Transformers+into+RNNs 8. LoLCATs: On Low-Rank Linearizing of Large Language Models — Michael Zhang, Simran Arora, Rahul Chalamala, Alan Wu, Benjamin Spector, Aaryan Singhal, Krithik Ramesh, Christopher Ré, 2024 (ICLR 2025) https://scholar.google.com/scholar?q=LoLCATs%3A+On+Low-Rank+Linearizing+of+Large+Language+Models 9. Transformers to SSMs: Distilling Quadratic Knowledge to Subquadratic Models (MOHAWK) — Aviv Bick, Kevin Y. Li, Eric P. Xing, J. Zico Kolter, Albert Gu, 2024 https://scholar.google.com/scholar?q=Transformers+to+SSMs%3A+Distilling+Quadratic+Knowledge+to+Subquadratic+Models+%28MOHAWK%29 10. The Mamba in the Llama: Distilling and Accelerating Hybrid Models — Junxiong Wang, Daniele Paliotta, Avner May, Alexander M. Rush, Tri Dao, 2024 https://scholar.google.com/scholar?q=The+Mamba+in+the+Llama%3A+Distilling+and+Accelerating+Hybrid+Models 11. Efficient Streaming Language Models with Attention Sinks (StreamingLLM) — Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, Mike Lewis, 2024 https://scholar.google.com/scholar?q=Efficient+Streaming+Language+Models+with+Attention+Sinks+%28StreamingLLM%29 12. Why do LLMs attend to the first token? — Federico Barbero, Alvaro Arroyo, Xiangming Gu, Christos Perivolaropoulos, Michael Bronstein, Petar Veličković, Razvan Pascanu, 2025 https://scholar.google.com/scholar?q=Why+do+LLMs+attend+to+the+first+token%3F 13. Short window attention enables long-term memorization — Loïc Cabannes, Maximilian Beck, Maria Lomeli, Gergely Szilvasy, Matthijs Douze, Jade Copet, Pierre-Emmanuel Mazaré, Gabriel Synnaeve, Hervé Jégou, 2026 https://scholar.google.com/scholar?q=Short+window+attention+enables+long-term+memorization 14. SWAA: Sliding Window Attention Adaptation for Efficient and Quality Preserving Long Context Processing — Yijiong Yu, Jiale Liu, Qingyun Wu, Huazheng Wang, Ji Pei, 2025 https://scholar.google.com/scholar?q=SWAA%3A+Sliding+Window+Attention+Adaptation+for+Efficient+and+Quality+Preserving+Long+Context+Processing 15. STILL: Selecting Tokens for Intra-Layer Hybrid Attention to Linearize LLMs — Weikang Meng, Liangyu Huo, Yadan Luo, Jiawen Guan, Jingyi Zhang, Yingjian Li, Zheng Zhang, 2026 https://scholar.google.com/scholar?q=STILL%3A+Selecting+Tokens+for+Intra-Layer+Hybrid+Attention+to+Linearize+LLMs 16. Transformers to SSMs: Distilling Quadratic Knowledge to Subquadratic Models (MOHAWK) and Llamba: Scaling Distilled Recurrent Models — Aviv Bick, Kevin Y. Li, Eric P. Xing, J. Zico Kolter, Albert Gu; Aviv Bick, Tobias Katsch, Nimit Sohoni, Arjun Desai, Albert Gu, 2024/2025 https://scholar.google.com/scholar?q=Transformers+to+SSMs%3A+Distilling+Quadratic+Knowledge+to+Subquadratic+Models+%28MOHAWK%29+and+Llamba%3A+Scaling+Distilled+Recurrent+Models 17. Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models — Soham De, Samuel L. Smith, Anushan Fernando, et al., 2024 https://scholar.google.com/scholar?q=Griffin%3A+Mixing+Gated+Linear+Recurrences+with+Local+Attention+for+Efficient+Language+Models 18. H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models; SnapKV: LLM Knows What You Are Looking For Before Generation — Zhenyu Zhang et al.; Yuhong Li et al., 2023/2024 https://scholar.google.com/scholar?q=H2O%3A+Heavy-Hitter+Oracle+for+Efficient+Generative+Inference+of+Large+Language+Models%3B+SnapKV%3A+LLM+Knows+What+You+Are+Looking+For+Before+Generation 19. RULER: What's the Real Context Size of Your Long-Context Language Models?; BABILong — Cheng-Ping Hsieh et al.; Yuri Kuratov et al., 2024 https://scholar.google.com/scholar?q=RULER%3A+What%27s+the+Real+Context+Size+of+Your+Long-Context+Language+Models%3F%3B+BABILong 20. Gated Delta Networks: Improving Mamba2 with Delta Rule; Parallelizing Linear Transformers with the Delta Rule — Songlin Yang, Jan Kautz, Ali Hatamizadeh; Songlin Yang et al., 2025/2024 https://scholar.google.com/scholar?q=Gated+Delta+Networks%3A+Improving+Mamba2+with+Delta+Rule%3B+Parallelizing+Linear+Transformers+with+the+Delta+Rule Interactive Visualization: Sliding-Window Attention Beats Linear Attention Without Any Training