Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficient

This episode explores the paper "Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficient," which asks whether a mixture-of-experts model can beat a dense model on loss when both train with the same compute and are capped at the same total parameter count. It covers how Switch-style top-1 routing works with experts the same size as the dense feed-forward layer. It also covers the difference between active and total parameters, using a 1.1 billion total, 426 million active, four-expert example. The discussion explains how a smaller active size lets an MoE see more tokens for the same FLOPs. That ties into deliberate overtraining under Chinchilla-style tradeoffs, and into a memory definition that includes the KV cache. The hosts also place the work alongside related studies on sparsity, inference-aware scaling and hardware efficiency. They flag the gap between a loss-based claim and real serving hardware as the question to keep in mind. Sources: 1. Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficient https://arxiv.org/pdf/2502.05172 2. Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Models — Samira Abnar, Harshay Shah, Dan Busbridge, et al., 2025 https://scholar.google.com/scholar?q=Parameters+vs+FLOPs%3A+Scaling+Laws+for+Optimal+Sparsity+for+Mixture-of-Experts+Language+Models 3. Toward Inference-Optimal Mixture-of-Expert Large Language Models — Longfei Yun, Yonghao Zhuang, Yao Fu, Eric P. Xing, Hao Zhang, 2024 https://scholar.google.com/scholar?q=Toward+Inference-Optimal+Mixture-of-Expert+Large+Language+Models 4. Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws — Nikhil Sardana, Jacob Portes, Sasha Doubov, Jonathan Frankle, 2024 https://scholar.google.com/scholar?q=Beyond+Chinchilla-Optimal%3A+Accounting+for+Inference+in+Language+Model+Scaling+Laws 5. Unified Scaling Laws for Routed Language Models — Aidan Clark et al., 2022 https://scholar.google.com/scholar?q=Unified+Scaling+Laws+for+Routed+Language+Models 6. Scaling Laws for Fine-Grained Mixture of Experts — Jan Ludziejewski et al., 2024 https://scholar.google.com/scholar?q=Scaling+Laws+for+Fine-Grained+Mixture+of+Experts 7. Reconciling Kaplan and Chinchilla Scaling Laws / Resolving Discrepancies in Compute-Optimal Scaling of Language Models — Tomer Porian et al.; Tim Pearce & Jinyeop Song, 2024-2025 https://scholar.google.com/scholar?q=Reconciling+Kaplan+and+Chinchilla+Scaling+Laws+%2F+Resolving+Discrepancies+in+Compute-Optimal+Scaling+of+Language+Models 8. Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations — Alexander Hägele et al., 2024 https://scholar.google.com/scholar?q=Scaling+Laws+and+Compute-Optimal+Training+Beyond+Fixed+Training+Durations 9. OLMoE: Open Mixture-of-Experts Language Models — Niklas Muennighoff et al., 2024 https://scholar.google.com/scholar?q=OLMoE%3A+Open+Mixture-of-Experts+Language+Models 10. Fast Inference of Mixture-of-Experts Language Models with Offloading / expert-offloading systems work (e.g., Mixtral offloading, MoE-Infinity) — Eliseev & Mazur; Xue et al., 2023-2024 https://scholar.google.com/scholar?q=Fast+Inference+of+Mixture-of-Experts+Language+Models+with+Offloading+%2F+expert-offloading+systems+work+%28e.g.%2C+Mixtral+offloading%2C+MoE-Infinity%29 11. GQA: Training Generalized Multi-Query Transformer Models / DeepSeek-V2 (MLA) — Ainslie et al.; DeepSeek-AI, 2023-2024 https://scholar.google.com/scholar?q=GQA%3A+Training+Generalized+Multi-Query+Transformer+Models+%2F+DeepSeek-V2+%28MLA%29 Interactive Visualization: Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficient