Speculative Decoding Surprisingly Excels on Sparse MoE Models

This episode examines why speculative decoding, long thought to give little benefit for sparse mixture-of-experts models at batch size one, actually helps most in that regime for many newer models. It covers ten MoE models from 30 billion to a trillion parameters, and explains how a drafter proposes tokens that the target model verifies in a single pass. It then contrasts coarse-grained experts (Mixtral-style) with fine-grained ones (Qwen3, Kimi-K2.5), where each token touches only a tiny slice of the layer. The discussion challenges the bandwidth-bound cost model from prior work, which predicts a speedup curve that rises, peaks at mid-sized batches, then falls. The authors point to two missing factors: adjacent tokens routing to overlapping experts, and real GPU execution dynamics. The episode also explains target efficiency, a metric that isolates verification cost from draft acceptance, and outlines the vLLM benchmarking setup on H100 and B200 GPUs. Listeners who care about low-latency serving for agents, test-time scaling, or RL rollouts will find it useful for deciding when speculative decoding is worth enabling. Sources: 1. Why Speculative Decoding Works Better Than Predicted on Sparse MoE Models — Ekagra Ranjan, Komal K. Teru, Bharat Venkitesh, Acyr Locatelli, 2026 http://arxiv.org/abs/2610.38981 2. Fast Inference from Transformers via Speculative Decoding — Yaniv Leviathan, Matan Kalman, Yossi Matias, 2022 (arXiv); ICML 2023 https://scholar.google.com/scholar?q=Fast+Inference+from+Transformers+via+Speculative+Decoding 3. Accelerating Large Language Model Decoding with Speculative Sampling — Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, John Jumper, 2023 https://scholar.google.com/scholar?q=Accelerating+Large+Language+Model+Decoding+with+Speculative+Sampling 4. MoESD: Unveil Speculative Decoding's Potential for Accelerating Sparse MoE — Zongle Huang, Lei Zhu, Zongyuan Zhan, Ting Hu, Weikai Mao, Xianzhi Yu, Yongpan Liu, Tianyu Zhang (verify author list), 2025 https://scholar.google.com/scholar?q=MoESD%3A+Unveil+Speculative+Decoding%27s+Potential+for+Accelerating+Sparse+MoE 5. EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test — Yuhui Li, Fangyun Wei, Chao Zhang, Hongyang Zhang, 2025 https://scholar.google.com/scholar?q=EAGLE-3%3A+Scaling+up+Inference+Acceleration+of+Large+Language+Models+via+Training-Time+Test 6. Exploiting Inter-Layer Expert Affinity for Accelerating Mixture-of-Experts Model Inference — Jinghan Yao, Quentin Anthony, Aamir Shafi, Hari Subramoni, Dhabaleswar K. Panda, 2024 https://scholar.google.com/scholar?q=Exploiting+Inter-Layer+Expert+Affinity+for+Accelerating+Mixture-of-Experts+Model+Inference 7. Fast Inference of Mixture-of-Experts Language Models with Offloading — Artyom Eliseev, Denis Mazur, 2023 https://scholar.google.com/scholar?q=Fast+Inference+of+Mixture-of-Experts+Language+Models+with+Offloading 8. DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models — Damai Dai, Chengqi Deng, Chenggang Zhao, and others (DeepSeek-AI), 2024 https://scholar.google.com/scholar?q=DeepSeekMoE%3A+Towards+Ultimate+Expert+Specialization+in+Mixture-of-Experts+Language+Models 9. Mixtral of Experts — Albert Q. Jiang and others (Mistral AI), 2024 https://scholar.google.com/scholar?q=Mixtral+of+Experts 10. On a Least Squares Adjustment of a Sampled Frequency Table When the Expected Marginal Totals are Known — W. Edwards Deming, Frederick F. Stephan, 1940 https://scholar.google.com/scholar?q=On+a+Least+Squares+Adjustment+of+a+Sampled+Frequency+Table+When+the+Expected+Marginal+Totals+are+Known 11. I-Divergence Geometry of Probability Distributions and Minimization Problems — Imre Csiszár, 1975 https://scholar.google.com/scholar?q=I-Divergence+Geometry+of+Probability+Distributions+and+Minimization+Problems 12. Information Theory and Statistical Mechanics — E. T. Jaynes, 1957 https://scholar.google.com/scholar?q=Information+Theory+and+Statistical+Mechanics 13. Sinkhorn Distances: Lightspeed Computation of Optimal Transport — Marco Cuturi, 2013 https://scholar.google.com/scholar?q=Sinkhorn+Distances%3A+Lightspeed+Computation+of+Optimal+Transport 14. Roofline: An Insightful Visual Performance Model for Multicore Architectures — Samuel Williams, Andrew Waterman, David Patterson, 2009 https://scholar.google.com/scholar?q=Roofline%3A+An+Insightful+Visual+Performance+Model+for+Multicore+Architectures 15. Efficiently Scaling Transformer Inference — Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Anselm Levskaya, Jonathan Heek, Kefan Xiao, Shivani Agrawal, Jeff Dean, 2022 https://scholar.google.com/scholar?q=Efficiently+Scaling+Transformer+Inference 16. MegaBlocks: Efficient Sparse Training with Mixture-of-Experts — Trevor Gale, Deepak Narayanan, Cliff Young, Matei Zaharia, 2023 https://scholar.google.com/scholar?q=MegaBlocks%3A+Efficient+Sparse+Training+with+Mixture-of-Experts 17. Dissecting the NVIDIA Hopper Architecture through Microbenchmarking and Multiple Level Analysis — Weile Luo, Ruibo Fan, Zeyu Li, Dayou Du, Qiang Wang, Xiaowen Chu, 2025 https://scholar.google.com/scholar?q=Dissecting+the+NVIDIA+Hopper+Architecture+through+Microbenchmarking+and+Multiple+Level+Analysis 18. Utility-driven Speculative Decoding for Mixture-of-Experts — Saxena, Tsai, Taneja, Jaleel, Qureshi, 2025 https://scholar.google.com/scholar?q=Utility-driven+Speculative+Decoding+for+Mixture-of-Experts 19. MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decoding — Sadhukhan et al., 2025 https://scholar.google.com/scholar?q=MagicDec%3A+Breaking+the+Latency-Throughput+Tradeoff+for+Long+Context+Generation+with+Speculative+Decoding 20. ExpertFlow: Efficient MoE Inference via Predictive Expert Caching and Token Scheduling — He et al., 2026 https://scholar.google.com/scholar?q=ExpertFlow%3A+Efficient+MoE+Inference+via+Predictive+Expert+Caching+and+Token+Scheduling 21. Probing Semantic Routing in Large Mixture-of-Expert Models — Olson et al., 2025 https://scholar.google.com/scholar?q=Probing+Semantic+Routing+in+Large+Mixture-of-Expert+Models 22. On the Spatial Structure of Mixture-of-Experts in Transformers — Bershatsky, Oseledets, 2025 https://scholar.google.com/scholar?q=On+the+Spatial+Structure+of+Mixture-of-Experts+in+Transformers 23. Scaling Laws for Fine-Grained Mixture of Experts — Ludziejewski et al., 2024 https://scholar.google.com/scholar?q=Scaling+Laws+for+Fine-Grained+Mixture+of+Experts 24. On a Least Squares Adjustment of a Sampled Frequency Table When the Expected Marginal Totals are Known / I-divergence Geometry of Probability Distributions and Minimization Problems — Deming & Stephan (1940); Csiszár (1975), 1940 / 1975 https://scholar.google.com/scholar?q=On+a+Least+Squares+Adjustment+of+a+Sampled+Frequency+Table+When+the+Expected+Marginal+Totals+are+Known+%2F+I-divergence+Geometry+of+Probability+Distributions+and+Minimization+Problems Interactive Visualization: Speculative Decoding Surprisingly Excels on Sparse MoE Models