Nemotron-Labs-Diffusion: One Model, Three Decoding Modes

This episode examines "Nemotron-Labs-Diffusion," an 8B model that runs in three modes from one set of weights: ordinary autoregressive decoding, block-diffusion parallel decoding, and self-speculation. In self-speculation, the diffusion pathway drafts tokens and the causal pathway verifies them, so no separate drafter is needed. It first explains why single-user decoding is memory-bound, then covers the prior approaches: masked and block diffusion, speculative decoding, and multi-token-prediction and Eagle3 drafters. A central theme is that the headline claims mix different metrics. Tokens per forward pass, acceptance rate and real hardware throughput are not interchangeable. A self-speculation cycle costs two forward passes, so true tokens per pass is roughly half the raw acceptance count. The episode also covers the training recipe, which uses a noisy block-bidirectional stream alongside a strictly causal clean stream. It notes that the "speed-of-light" analysis is an oracle upper bound, not a deployable sampler. Listeners get a clear framework for reading speedup claims in LLM inference papers critically. Sources: 1. Nemotron-Labs-Diffusion: A Tri-Mode Language Model Unifying Autoregressive, Diffusion, and Self-Speculation Decoding — Yonggan Fu, Lexington Whalen, Abhinav Garg, Chengyue Wu, Maksim Khadkevich, Nicolai Oswald, Enze Xie, Daniel Egert, Sharath Turuvekere Sreenivas, Shizhe Diao, Chenhan Yu, Ye Yu, Weijia Chen, Sajad Norouzi, Jingyu Liu, Shiyi Lan, Ligeng Zhu, Jin Wang, Jindong Jiang, Morteza Mardani, Mehran Maghoumi, Song Han, Ante Jukić, Nima Tajbakhsh, Jan Kautz, Pavlo Molchanov, 2026 http://arxiv.org/abs/2607.05722 2. Fast Inference from Transformers via Speculative Decoding — Yaniv Leviathan, Matan Kalman, Yossi Matias, 2022 (ICML 2023) https://scholar.google.com/scholar?q=Fast+Inference+from+Transformers+via+Speculative+Decoding 3. EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test (and the earlier EAGLE / EAGLE-2) — Yuhui Li, Fangyun Wei, Chao Zhang, Hongyang Zhang, 2024-2025 https://scholar.google.com/scholar?q=EAGLE-3%3A+Scaling+up+Inference+Acceleration+of+Large+Language+Models+via+Training-Time+Test+%28and+the+earlier+EAGLE+%2F+EAGLE-2%29 4. Large Language Diffusion Models (LLaDA) — Shen Nie, Fengqi Zhu, et al., 2025 https://scholar.google.com/scholar?q=Large+Language+Diffusion+Models+%28LLaDA%29 5. Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models (BD3-LMs) — Marianne Arriola, Subham Sekhar Sahoo, et al., 2025 (ICLR 2025) https://scholar.google.com/scholar?q=Block+Diffusion%3A+Interpolating+Between+Autoregressive+and+Diffusion+Language+Models+%28BD3-LMs%29 6. Better & Faster Large Language Models via Multi-token Prediction — Fabian Gloeckle, Badr Youbi Idrissi, et al. (Meta FAIR), 2024 https://scholar.google.com/scholar?q=Better+%26+Faster+Large+Language+Models+via+Multi-token+Prediction 7. EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test — Yuhui Li et al., 2025 https://scholar.google.com/scholar?q=EAGLE-3%3A+Scaling+up+Inference+Acceleration+of+Large+Language+Models+via+Training-Time+Test 8. Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models (BD3-LM) — Marianne Arriola et al., 2025 https://scholar.google.com/scholar?q=Block+Diffusion%3A+Interpolating+Between+Autoregressive+and+Diffusion+Language+Models+%28BD3-LM%29 9. SDAR: A Synergistic Diffusion-AutoRegression Paradigm for Scalable Sequence Generation — Shuang Cheng et al., 2025 https://scholar.google.com/scholar?q=SDAR%3A+A+Synergistic+Diffusion-AutoRegression+Paradigm+for+Scalable+Sequence+Generation 10. Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding — Chengyue Wu et al., 2025 https://scholar.google.com/scholar?q=Fast-dLLM%3A+Training-free+Acceleration+of+Diffusion+LLM+by+Enabling+KV+Cache+and+Parallel+Decoding 11. Dream 7B: Diffusion Large Language Models — Jiacheng Ye et al., 2025 https://scholar.google.com/scholar?q=Dream+7B%3A+Diffusion+Large+Language+Models 12. Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads — Tianle Cai et al., 2024 https://scholar.google.com/scholar?q=Medusa%3A+Simple+LLM+Inference+Acceleration+Framework+with+Multiple+Decoding+Heads 13. DeepSeek-V3 Technical Report — DeepSeek-AI, 2024 https://scholar.google.com/scholar?q=DeepSeek-V3+Technical+Report 14. Self Speculative Decoding / Draft & Verify (layer skipping) — Jun Zhang et al., 2023 https://scholar.google.com/scholar?q=Self+Speculative+Decoding+%2F+Draft+%26+Verify+%28layer+skipping%29 15. The Pitfalls of Next-Token Prediction — Gregor Bachmann, Vaishnavh Nagarajan, 2024 https://scholar.google.com/scholar?q=The+Pitfalls+of+Next-Token+Prediction 16. SPEED-Bench — Cited as [1] in the paper, 2026 https://scholar.google.com/scholar?q=SPEED-Bench 17. Diffusion Language Models are Super Data Learners — Jinjie Ni et al., 2025 https://scholar.google.com/scholar?q=Diffusion+Language+Models+are+Super+Data+Learners Interactive Visualization: Nemotron-Labs-Diffusion: One Model, Three Decoding Modes