DepthBench: How Residual Connections Unlock Computational Depth in Transformers

This episode examines DepthBench, a study of why stacking more transformer layers often adds little useful computation, and which residual and normalization designs fix that when parameter count and training recipe are held constant. It reviews the Pre-LN "Curse of Depth" argument, in which growing residual-stream magnitude makes late layers act like identity. It then surveys ten architectures in three families: normalization tweaks such as LayerNorm Scaling and DeepNorm, multi-stream residuals such as Hyper-Connections and mHC, and cross-layer access such as Attention Residuals. The central claim is that the benefit of going deeper and narrower depends on the architecture: it can hurt under standard Pre-LN but keeps helping under two other designs. That would mean the common shallow-wide shapes are partly an artifact of the residual design. The discussion sets this against Kaplan-style scaling results, layer-pruning studies, and the rival explanation that deep models are simply mis-scaled in training. Listeners get a clear framework for "effective computational depth" and for judging whether the evidence supports the claim. Sources: 1. DepthBench: Measuring How Residual Connections Enable More Computational Depth — Keyu Wang, Yangyi Huang, Jiale Kang, David González-Martínez, Weiyang Liu, Shiwei Liu, 2026 http://arxiv.org/abs/2609.32534 2. Scaling Laws for Neural Language Models — Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, et al., 2020 https://scholar.google.com/scholar?q=Scaling+Laws+for+Neural+Language+Models 3. Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers — Yi Tay, Mostafa Dehghani, Jinfeng Rao, William Fedus, Samira Abnar, Hyung Won Chung, Sharan Narang, Dani Yogatama, Ashish Vaswani, Donald Metzler, 2021 https://scholar.google.com/scholar?q=Scale+Efficiently%3A+Insights+from+Pre-training+and+Fine-tuning+Transformers 4. MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases — Zechun Liu, Changsheng Zhao, Forrest Iandola, et al. (Meta), 2024 https://scholar.google.com/scholar?q=MobileLLM%3A+Optimizing+Sub-billion+Parameter+Language+Models+for+On-Device+Use+Cases 5. The Curse of Depth in Large Language Models — Wenfang Sun, Xinyuan Song, Pengxiang Li, Lu Yin, Yefeng Zheng, Shiwei Liu, 2025 https://scholar.google.com/scholar?q=The+Curse+of+Depth+in+Large+Language+Models 6. Hyper-Connections (and follow-up: mHC, Manifold-Constrained Hyper-Connections) — Defa Zhu, Hongzhi Huang, Zihao Huang, et al. (ByteDance Seed) for HC; DeepSeek-AI researchers (Xie et al.) for mHC, 2024-2025 https://scholar.google.com/scholar?q=Hyper-Connections+%28and+follow-up%3A+mHC%2C+Manifold-Constrained+Hyper-Connections%29 7. Do Language Models Use Their Depth Efficiently? — Róbert Csordás, Christopher D. Manning, Christopher Potts, 2025 https://scholar.google.com/scholar?q=Do+Language+Models+Use+Their+Depth+Efficiently%3F 8. The Unreasonable Ineffectiveness of the Deeper Layers — Andrey Gromov, Kushal Tirumala, Hassan Shapourian, Paolo Glorioso, Daniel A. Roberts, 2024 https://scholar.google.com/scholar?q=The+Unreasonable+Ineffectiveness+of+the+Deeper+Layers 9. ShortGPT: Layers in Large Language Models are More Redundant Than You Expect — Xin Men et al., 2024 https://scholar.google.com/scholar?q=ShortGPT%3A+Layers+in+Large+Language+Models+are+More+Redundant+Than+You+Expect 10. Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer (muP) — Greg Yang et al., 2022 https://scholar.google.com/scholar?q=Tensor+Programs+V%3A+Tuning+Large+Neural+Networks+via+Zero-Shot+Hyperparameter+Transfer+%28muP%29 11. Don't be lazy: CompleteP enables compute-efficient deep transformers — Nolan Dey et al., 2025 https://scholar.google.com/scholar?q=Don%27t+be+lazy%3A+CompleteP+enables+compute-efficient+deep+transformers 12. Scaling Laws with Model Shape / Gemstones: A Model Suite for Multi-Faceted Scaling Laws — Sean McLeish et al., 2025 https://scholar.google.com/scholar?q=Scaling+Laws+with+Model+Shape+%2F+Gemstones%3A+A+Model+Suite+for+Multi-Faceted+Scaling+Laws 13. The Impact of Depth on Compositional Generalization in Transformer Language Models — Jackson Petty et al., 2024 https://scholar.google.com/scholar?q=The+Impact+of+Depth+on+Compositional+Generalization+in+Transformer+Language+Models 14. Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach — Jonas Geiping et al., 2025 https://scholar.google.com/scholar?q=Scaling+up+Test-Time+Compute+with+Latent+Reasoning%3A+A+Recurrent+Depth+Approach 15. Transformers, Parallel Computation, and Logarithmic Depth — Clayton Sanford, Daniel Hsu, Matus Telgarsky, 2024 https://scholar.google.com/scholar?q=Transformers%2C+Parallel+Computation%2C+and+Logarithmic+Depth 16. The Expressive Power of Transformers with Chain of Thought / The Parallelism Tradeoff — William Merrill, Ashish Sabharwal, 2023-2024 https://scholar.google.com/scholar?q=The+Expressive+Power+of+Transformers+with+Chain+of+Thought+%2F+The+Parallelism+Tradeoff 17. Resolving Discrepancies in Compute-Optimal Scaling of Language Models — Tomer Porian et al., 2024 https://scholar.google.com/scholar?q=Resolving+Discrepancies+in+Compute-Optimal+Scaling+of+Language+Models 18. DenseFormer: Enhancing Information Flow in Transformers via Depth Weighted Averaging — Matteo Pagliardini et al., 2024 https://scholar.google.com/scholar?q=DenseFormer%3A+Enhancing+Information+Flow+in+Transformers+via+Depth+Weighted+Averaging 19. Peri-LN: Revisiting Layer Normalization in the Transformer Architecture — Jeonghoon Kim et al., 2025 https://scholar.google.com/scholar?q=Peri-LN%3A+Revisiting+Layer+Normalization+in+the+Transformer+Architecture Interactive Visualization: DepthBench: How Residual Connections Unlock Computational Depth in Transformers