Zum Inhalt springen

Does MoE Really Speed Up Inference on Edge Hardware?

20.11.2026

Shownotes

This episode examines whether Mixture-of-Experts models deliver real speed and energy savings on consumer and edge hardware, using a 64-expert OLMoE-1B-7B model, about 1.3B of whose 6.9B parameters are active per token, against three small dense models on an Apple M2 Pro laptop and an 8GB Jetson Orin Nano. It explains why batch-size-one decoding is limited by memory bandwidth rather than arithmetic, so that sparse compute does not automatically translate into faster tokens. The discussion tests the paper's headline claim that cost tracks total parameters, not active ones. It also raises the question of whether the relevant quantity is instead the bytes actually read per token, and points to on-device MoE work such as EdgeMoE, SwapMoE and HOBBIT that the paper overlooks. It also walks through the benchmarking setup: llama.cpp with 4-bit quantization, 45 runs per model-and-device cell, a thread-count sweep, and tegrastats-based energy measurement on the Jetson only. Listeners get a clear explanation of the MoE memory-versus-compute tradeoff and a critical look at how much of a measured result comes from the architecture and how much from a vanilla runtime. Sources: 1. Does Mixture-of-Experts Actually Help Inference on Consumer and Edge Hardware? An Empirical Study — Alfarizy Alfarizy, Hung Truong Thanh Nguyen, René Richard, Roozbeh Razavi-Far, Hung Cao, 2026 http://arxiv.org/abs/2606.21428 2. Mixtral offloading / Fast inference of MoE with offloading (already cited) and HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inference — Tang et al., 2024 https://scholar.google.com/scholar?q=Mixtral+offloading+%2F+Fast+inference+of+MoE+with+offloading+%28already+cited%29+and+HOBBIT%3A+A+Mixed+Precision+Expert+Offloading+System+for+Fast+MoE+Inference 3. EdgeMoE: Empowering Sparse Large Language Models on Mobile Devices — Yi et al., 2023 https://scholar.google.com/scholar?q=EdgeMoE%3A+Empowering+Sparse+Large+Language+Models+on+Mobile+Devices 4. SwapMoE: Serving Off-the-shelf MoE-based Large Language Models with Tunable Memory Budget — Kong et al., 2023 https://scholar.google.com/scholar?q=SwapMoE%3A+Serving+Off-the-shelf+MoE-based+Large+Language+Models+with+Tunable+Memory+Budget 5. Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference — Hwang et al., 2024 https://scholar.google.com/scholar?q=Pre-gated+MoE%3A+An+Algorithm-System+Co-Design+for+Fast+and+Scalable+Mixture-of-Expert+Inference 6. Mixtral-8x7B / DeepSeek-V2-Lite / Qwen1.5-MoE-A2.7B technical reports and Qwen3-30B-A3B — Jiang et al.; DeepSeek-AI; Qwen Team, 2024-2025 https://scholar.google.com/scholar?q=Mixtral-8x7B+%2F+DeepSeek-V2-Lite+%2F+Qwen1.5-MoE-A2.7B+technical+reports+and+Qwen3-30B-A3B 7. Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts LLMs — Lu et al., 2024 https://scholar.google.com/scholar?q=Not+All+Experts+are+Equal%3A+Efficient+Expert+Pruning+and+Skipping+for+Mixture-of-Experts+LLMs 8. Roofline: an insightful visual performance model for multicore architectures — Williams, Waterman, Patterson, 2009 https://scholar.google.com/scholar?q=Roofline%3A+an+insightful+visual+performance+model+for+multicore+architectures 9. Fiddler / MoE-Lightning / Klotski-style MoE offloading systems for memory-constrained devices — Various, 2024-2025 https://scholar.google.com/scholar?q=Fiddler+%2F+MoE-Lightning+%2F+Klotski-style+MoE+offloading+systems+for+memory-constrained+devices Interactive Visualization: Does MoE Really Speed Up Inference on Edge Hardware?

Alle Folgen von AI Post Transformers

Einstellungen

Darstellung

Sprache

Bestimmt, welche Sender dir auf der Startseite zuerst gezeigt werden.

Verlauf

Zuletzt gehörte Sender und Folgen und deine letzten Suchen liegen nur in diesem Browser.

Hinweise

Ein kurzer Hinweis, wenn ein Podcast, dem du folgst, eine neue Folge hat. Einzelne Podcasts stellst du über die Glocke auf ihrer Seite ab.

Konto

Favoriten speichern – in der App und auf der Website.

Tastenkürzel

/
Suche öffnen
Leertaste
Abspielen oder anhalten
← →
Im Player: 15 s zurück, 30 s vor (Folgen) · im Regal: nächste Kachel
↑ ↓
Im Regal: Ziel innerhalb der Kachel
Pos1 / Ende
Im Regal: erste bzw. letzte Kachel
Esc
Fenster oder Menü schließen
?
Diese Übersicht
Läuft gerade

Zur Senderseite

Als Nächstes

    Folgen mit „Als Nächstes“ vormerken – sie laufen nach der aktuellen Folge.