How to Loop Your Transformer: Scaling, Specialization, Parallel Inference

This episode examines how to loop a transformer's layers, meaning reuse the same weights several times per token, depending on whether model size, training compute, or decoding speed is the scarce resource. It compares Full-Stack, Middle-Stack, and Per-Layer looping. Full-Stack gives the best quality at fixed parameters, Middle-Stack gives the best return at fixed training FLOPs, and gains flatten or turn unstable past about 16 loops. The 32-loop evidence comes only from the 70M and 100M models. Small per-loop low-rank adapters add only 1 to 4 percent more parameters and help more as loop counts grow, but at comparable FLOPs, doubling true depth still reaches lower loss. The discussion then turns to the decoding-latency problem, where each loop adds a sequential pass. It covers the Parallel Loop Transformer, which uses the previous token's state to run loops in parallel, and why that shift compounds prediction errors and causes magnitude mismatches. It also covers a proposed fix that re-injects the current token's embedding through normalization and a gate. Listeners get a practical guide to matching a looping design to a real resource constraint, with caveats about how far the evidence scales. Sources: 1. How to Loop Your Transformer: Loop Scaling, Specialization, and Parallel Inference — Chanakya Ekbote, Artidoro Pagnoni, Devendra Singh Sachan, Shruti Bhosale, Mike Lewis, Maryam Fazel-Zarandi, Shashank Rajput, Sujan Kumar Gonugondla, 2026 http://arxiv.org/abs/2610.40325v1 2. Universal Transformers — Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, Łukasz Kaiser, 2018 https://scholar.google.com/scholar?q=Universal+Transformers 3. Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach — Jonas Geiping, Sean McLeish, Neel Jain, John Kirchenbauer, et al., 2025 https://scholar.google.com/scholar?q=Scaling+up+Test-Time+Compute+with+Latent+Reasoning%3A+A+Recurrent+Depth+Approach 4. Reasoning with Latent Thoughts: On the Power of Looped Transformers — Nikunj Saunshi, Nishanth Dikkala, Zhiyuan Li, Sanjiv Kumar, Sashank Reddi, 2025 https://scholar.google.com/scholar?q=Reasoning+with+Latent+Thoughts%3A+On+the+Power+of+Looped+Transformers 5. Parallel Loop Transformer (PLT) — Wu et al. (full title and author list not verified; cited in the paper as Wu et al., 2025), 2025 https://scholar.google.com/scholar?q=Parallel+Loop+Transformer+%28PLT%29 6. LoRA: Low-Rank Adaptation of Large Language Models — Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, 2021 https://scholar.google.com/scholar?q=LoRA%3A+Low-Rank+Adaptation+of+Large+Language+Models 7. Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRA — Sangmin Bae, Adam Fisch, Hrayr Harutyunyan, Ziwei Ji, Seungyeon Kim, Tal Schuster, 2024/2025 (cited as Bae et al., 2025 in the paper; verify the exact venue year) https://scholar.google.com/scholar?q=Relaxed+Recursive+Transformers%3A+Effective+Parameter+Sharing+with+Layer-wise+LoRA 8. Fast Inference from Transformers via Speculative Decoding — Yaniv Leviathan, Matan Kalman, Yossi Matias, 2023 https://scholar.google.com/scholar?q=Fast+Inference+from+Transformers+via+Speculative+Decoding 9. Efficient Memory Management for Large Language Model Serving with PagedAttention — Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, Ion Stoica, 2023 https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention 10. Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads — Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D. Lee, Tri Dao, 2024 https://scholar.google.com/scholar?q=Medusa%3A+Simple+LLM+Inference+Acceleration+Framework+with+Multiple+Decoding+Heads 11. Parallel Loop Transformer for Efficient Test-Time Computation Scaling — Bohong Wu, Mengzhao Chen, Xiang Luo, et al., 2025 https://scholar.google.com/scholar?q=Parallel+Loop+Transformer+for+Efficient+Test-Time+Computation+Scaling 12. Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computation — Sangmin Bae, Yujin Kim, Reza Bayat, et al., 2025/2026 https://scholar.google.com/scholar?q=Mixture-of-Recursions%3A+Learning+Dynamic+Recursive+Depths+for+Adaptive+Token-Level+Computation 13. Parcae: Scaling Laws for Stable Looped Language Models — Hayden Prairie, Zachary Novack, Taylor Berg-Kirkpatrick, Daniel Y. Fu, 2026 https://scholar.google.com/scholar?q=Parcae%3A+Scaling+Laws+for+Stable+Looped+Language+Models 14. How Much Is One Recurrence Worth? Iso-Depth Scaling Laws for Looped Language Models — Kristian Schwethelm, Daniel Rueckert, Georgios Kaissis, 2026 https://scholar.google.com/scholar?q=How+Much+Is+One+Recurrence+Worth%3F+Iso-Depth+Scaling+Laws+for+Looped+Language+Models 15. LoopCoder-v2: Only Loop Once for Efficient Test-Time Computation Scaling — Jian Yang, Shawn Guo, Wei Zhang, et al., 2026 https://scholar.google.com/scholar?q=LoopCoder-v2%3A+Only+Loop+Once+for+Efficient+Test-Time+Computation+Scaling 16. Hyperloop Transformers — Abbas Zeitoun, Lucas Torroba-Hennigen, Yoon Kim, 2026 https://scholar.google.com/scholar?q=Hyperloop+Transformers 17. T2MLR: Transformer with Temporal Middle-Layer Recurrence — Ziyang Cai, Xingyu Zhu, Yihe Dong, Yinghui He, Sanjeev Arora, 2026 https://scholar.google.com/scholar?q=T2MLR%3A+Transformer+with+Temporal+Middle-Layer+Recurrence 18. Scaling Latent Reasoning via Looped Language Models (Ouro) — Rui-Jie Zhu, Zixuan Wang, Kai Hua, et al., 2026 https://scholar.google.com/scholar?q=Scaling+Latent+Reasoning+via+Looped+Language+Models+%28Ouro%29 19. Staircase Attention for Recurrent Processing of Sequences — Da Ju, Stephen Roller, Sainbayar Sukhbaatar, Jason E. Weston, 2022 https://scholar.google.com/scholar?q=Staircase+Attention+for+Recurrent+Processing+of+Sequences Interactive Visualization: How to Loop Your Transformer: Scaling, Specialization, Parallel Inference