
Does MoE Really Speed Up Inference on Edge Hardware?
Show notes
This episode examines whether Mixture-of-Experts models deliver real speed and energy savings on consumer and edge hardware, using a 64-expert OLMoE-1B-7B model, about 1.3B of whose 6.9B parameters are active per token, against three small dense models on an Apple M2 Pro laptop and an 8GB Jetson Orin Nano. It explains why batch-size-one decoding is limited by memory bandwidth rather than arithmetic, so that sparse compute does not automatically translate into faster tokens. The discussion tests the paper's headline claim that cost tracks total parameters, not active ones. It also raises the question of whether the relevant quantity is instead the bytes actually read per token, and points to on-device MoE work such as EdgeMoE, SwapMoE and HOBBIT that the paper overlooks. It also walks through the benchmarking setup: llama.cpp with 4-bit quantization, 45 runs per model-and-device cell, a thread-count sweep, and tegrastats-based energy measurement on the Jetson only. Listeners get a clear explanation of the MoE memory-versus-compute tradeoff and a critical look at how much of a measured result comes from the architecture and how much from a vanilla runtime. Sources: 1. Does Mixture-of-Experts Actually Help Inference on Consumer and Edge Hardware? An Empirical Study — Alfarizy Alfarizy, Hung Truong Thanh Nguyen, René Richard, Roozbeh Razavi-Far, Hung Cao, 2026 http://arxiv.org/abs/2606.21428 2. Mixtral offloading / Fast inference of MoE with offloading (already cited) and HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inference — Tang et al., 2024 https://scholar.google.com/scholar?q=Mixtral+offloading+%2F+Fast+inference+of+MoE+with+offloading+%28already+cited%29+and+HOBBIT%3A+A+Mixed+Precision+Expert+Offloading+System+for+Fast+MoE+Inference 3. EdgeMoE: Empowering Sparse Large Language Models on Mobile Devices — Yi et al., 2023 https://scholar.google.com/scholar?q=EdgeMoE%3A+Empowering+Sparse+Large+Language+Models+on+Mobile+Devices 4. SwapMoE: Serving Off-the-shelf MoE-based Large Language Models with Tunable Memory Budget — Kong et al., 2023 https://scholar.google.com/scholar?q=SwapMoE%3A+Serving+Off-the-shelf+MoE-based+Large+Language+Models+with+Tunable+Memory+Budget 5. Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference — Hwang et al., 2024 https://scholar.google.com/scholar?q=Pre-gated+MoE%3A+An+Algorithm-System+Co-Design+for+Fast+and+Scalable+Mixture-of-Expert+Inference 6. Mixtral-8x7B / DeepSeek-V2-Lite / Qwen1.5-MoE-A2.7B technical reports and Qwen3-30B-A3B — Jiang et al.; DeepSeek-AI; Qwen Team, 2024-2025 https://scholar.google.com/scholar?q=Mixtral-8x7B+%2F+DeepSeek-V2-Lite+%2F+Qwen1.5-MoE-A2.7B+technical+reports+and+Qwen3-30B-A3B 7. Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts LLMs — Lu et al., 2024 https://scholar.google.com/scholar?q=Not+All+Experts+are+Equal%3A+Efficient+Expert+Pruning+and+Skipping+for+Mixture-of-Experts+LLMs 8. Roofline: an insightful visual performance model for multicore architectures — Williams, Waterman, Patterson, 2009 https://scholar.google.com/scholar?q=Roofline%3A+an+insightful+visual+performance+model+for+multicore+architectures 9. Fiddler / MoE-Lightning / Klotski-style MoE offloading systems for memory-constrained devices — Various, 2024-2025 https://scholar.google.com/scholar?q=Fiddler+%2F+MoE-Lightning+%2F+Klotski-style+MoE+offloading+systems+for+memory-constrained+devices Interactive Visualization: Does MoE Really Speed Up Inference on Edge Hardware?