Gradient Noise Scale Predicts Critical Batch Size for Large-Batch Training

This episode examines the OpenAI paper on large-batch training, which asks why ImageNet training stalls at batches of tens of thousands of images while Dota 2 agents train on millions of observations. It explains the tradeoff between serial optimizer steps and total examples processed. This tradeoff forms a Pareto frontier whose knee is the critical batch size, where compute efficiency is 50 percent. The central claim is that the gradient noise scale, a statistic computable from a single ordinary training run, predicts that critical batch size to within an order of magnitude. The discussion walks through the second-order loss derivation behind it, where progress per step is halved when batch size equals the noise scale. It also covers the theory's predictions: the noise scale grows during training, is larger for harder tasks and for reinforcement learning, and depends only weakly on model size. The authors admit the derivation rests on many unfounded assumptions. A listener would find it useful for understanding how to choose batch sizes, including why batch sizes that increase over training make sense, without brute-force sweeps. Sources: 1. An Empirical Model of Large-Batch Training — Sam McCandlish, Jared Kaplan, Dario Amodei, OpenAI Dota Team, 2018 http://arxiv.org/abs/1812.06162 2. An Empirical Model of Large-Batch Training — Sam McCandlish, Jared Kaplan, Dario Amodei, OpenAI Dota Team, 2018 https://scholar.google.com/scholar?q=An+Empirical+Model+of+Large-Batch+Training 3. Measuring the Effects of Data Parallelism on Neural Network Training — Christopher J. Shallue, Jaehoon Lee, Joseph Antognini, Jascha Sohl-Dickstein, Roy Frostig, George E. Dahl, 2018 https://scholar.google.com/scholar?q=Measuring+the+Effects+of+Data+Parallelism+on+Neural+Network+Training 4. Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour — Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, Kaiming He, 2017 https://scholar.google.com/scholar?q=Accurate%2C+Large+Minibatch+SGD%3A+Training+ImageNet+in+1+Hour 5. Large Batch Training of Convolutional Networks (LARS) — Yang You, Igor Gitman, Boris Ginsburg, 2017 https://scholar.google.com/scholar?q=Large+Batch+Training+of+Convolutional+Networks+%28LARS%29 6. On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima — Nitish Shirish Keskar, Dheevatcha Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, Ping Tak Peter Tang, 2016 https://scholar.google.com/scholar?q=On+Large-Batch+Training+for+Deep+Learning%3A+Generalization+Gap+and+Sharp+Minima 7. Don't Decay the Learning Rate, Increase the Batch Size — Samuel L. Smith, Pieter-Jan Kindermans, Chris Ying, Quoc V. Le, 2017 https://scholar.google.com/scholar?q=Don%27t+Decay+the+Learning+Rate%2C+Increase+the+Batch+Size 8. Scaling Laws for Neural Language Models — Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, Dario Amodei, 2020 https://scholar.google.com/scholar?q=Scaling+Laws+for+Neural+Language+Models 9. Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Model — Guodong Zhang, Lala Li, Zachary Nado, James Martens, Sushant Sachdeva, George E. Dahl, Christopher J. Shallue, Roger Grosse, 2019 https://scholar.google.com/scholar?q=Which+Algorithmic+Choices+Matter+at+Which+Batch+Sizes%3F+Insights+From+a+Noisy+Quadratic+Model 10. Three Factors Influencing Minima in SGD — Stanisław Jastrzębski, Zachary Kenton, Devansh Arpit, et al., 2017 https://scholar.google.com/scholar?q=Three+Factors+Influencing+Minima+in+SGD 11. Dota 2 with Large Scale Deep Reinforcement Learning — OpenAI, Christopher Berner, Greg Brockman, et al., 2019 https://scholar.google.com/scholar?q=Dota+2+with+Large+Scale+Deep+Reinforcement+Learning 12. Online Batch Selection / Adaptive sampling via gradient noise (Balles, Romero, Hennig) — Lukas Balles, Javier Romero, Philipp Hennig, 2016 https://scholar.google.com/scholar?q=Online+Batch+Selection+%2F+Adaptive+sampling+via+gradient+noise+%28Balles%2C+Romero%2C+Hennig%29 Interactive Visualization: Gradient Noise Scale Predicts Critical Batch Size for Large-Batch Training