Skip to content

Does MoE Really Speed Up Inference on Edge Hardware?

November 20, 2026

Show notes

This episode examines whether Mixture-of-Experts models deliver real speed and energy savings on consumer and edge hardware, using a 64-expert OLMoE-1B-7B model, about 1.3B of whose 6.9B parameters are active per token, against three small dense models on an Apple M2 Pro laptop and an 8GB Jetson Orin Nano. It explains why batch-size-one decoding is limited by memory bandwidth rather than arithmetic, so that sparse compute does not automatically translate into faster tokens. The discussion tests the paper's headline claim that cost tracks total parameters, not active ones. It also raises the question of whether the relevant quantity is instead the bytes actually read per token, and points to on-device MoE work such as EdgeMoE, SwapMoE and HOBBIT that the paper overlooks. It also walks through the benchmarking setup: llama.cpp with 4-bit quantization, 45 runs per model-and-device cell, a thread-count sweep, and tegrastats-based energy measurement on the Jetson only. Listeners get a clear explanation of the MoE memory-versus-compute tradeoff and a critical look at how much of a measured result comes from the architecture and how much from a vanilla runtime. Sources: 1. Does Mixture-of-Experts Actually Help Inference on Consumer and Edge Hardware? An Empirical Study — Alfarizy Alfarizy, Hung Truong Thanh Nguyen, René Richard, Roozbeh Razavi-Far, Hung Cao, 2026 http://arxiv.org/abs/2606.21428 2. Mixtral offloading / Fast inference of MoE with offloading (already cited) and HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inference — Tang et al., 2024 https://scholar.google.com/scholar?q=Mixtral+offloading+%2F+Fast+inference+of+MoE+with+offloading+%28already+cited%29+and+HOBBIT%3A+A+Mixed+Precision+Expert+Offloading+System+for+Fast+MoE+Inference 3. EdgeMoE: Empowering Sparse Large Language Models on Mobile Devices — Yi et al., 2023 https://scholar.google.com/scholar?q=EdgeMoE%3A+Empowering+Sparse+Large+Language+Models+on+Mobile+Devices 4. SwapMoE: Serving Off-the-shelf MoE-based Large Language Models with Tunable Memory Budget — Kong et al., 2023 https://scholar.google.com/scholar?q=SwapMoE%3A+Serving+Off-the-shelf+MoE-based+Large+Language+Models+with+Tunable+Memory+Budget 5. Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference — Hwang et al., 2024 https://scholar.google.com/scholar?q=Pre-gated+MoE%3A+An+Algorithm-System+Co-Design+for+Fast+and+Scalable+Mixture-of-Expert+Inference 6. Mixtral-8x7B / DeepSeek-V2-Lite / Qwen1.5-MoE-A2.7B technical reports and Qwen3-30B-A3B — Jiang et al.; DeepSeek-AI; Qwen Team, 2024-2025 https://scholar.google.com/scholar?q=Mixtral-8x7B+%2F+DeepSeek-V2-Lite+%2F+Qwen1.5-MoE-A2.7B+technical+reports+and+Qwen3-30B-A3B 7. Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts LLMs — Lu et al., 2024 https://scholar.google.com/scholar?q=Not+All+Experts+are+Equal%3A+Efficient+Expert+Pruning+and+Skipping+for+Mixture-of-Experts+LLMs 8. Roofline: an insightful visual performance model for multicore architectures — Williams, Waterman, Patterson, 2009 https://scholar.google.com/scholar?q=Roofline%3A+an+insightful+visual+performance+model+for+multicore+architectures 9. Fiddler / MoE-Lightning / Klotski-style MoE offloading systems for memory-constrained devices — Various, 2024-2025 https://scholar.google.com/scholar?q=Fiddler+%2F+MoE-Lightning+%2F+Klotski-style+MoE+offloading+systems+for+memory-constrained+devices Interactive Visualization: Does MoE Really Speed Up Inference on Edge Hardware?

All episodes of AI Post Transformers

Settings

Display

Language

Decides which stations you see first on the home page.

History

Recently played stations and episodes and your latest searches are only stored in this browser.

Notifications

A short notice when a podcast you follow has a new episode. Turn off single podcasts with the bell on their page.

Account

Save favorites – in the app and on the website.

Keyboard shortcuts

/
Open search
Space
Play or pause
← →
In the player: back 15 s, forward 30 s (episodes) · in a row: next tile
↑ ↓
In a row: target within the tile
Home / End
In a row: first or last tile
Esc
Close window or menu
?
This overview
Now playing

Go to station page

Play next

    Add episodes with “Play next” – they play after the current episode.