Saltar al contenido

Does MoE Really Speed Up Inference on Edge Hardware?

20 de noviembre de 2026

Notas del episodio

This episode examines whether Mixture-of-Experts models deliver real speed and energy savings on consumer and edge hardware, using a 64-expert OLMoE-1B-7B model, about 1.3B of whose 6.9B parameters are active per token, against three small dense models on an Apple M2 Pro laptop and an 8GB Jetson Orin Nano. It explains why batch-size-one decoding is limited by memory bandwidth rather than arithmetic, so that sparse compute does not automatically translate into faster tokens. The discussion tests the paper's headline claim that cost tracks total parameters, not active ones. It also raises the question of whether the relevant quantity is instead the bytes actually read per token, and points to on-device MoE work such as EdgeMoE, SwapMoE and HOBBIT that the paper overlooks. It also walks through the benchmarking setup: llama.cpp with 4-bit quantization, 45 runs per model-and-device cell, a thread-count sweep, and tegrastats-based energy measurement on the Jetson only. Listeners get a clear explanation of the MoE memory-versus-compute tradeoff and a critical look at how much of a measured result comes from the architecture and how much from a vanilla runtime. Sources: 1. Does Mixture-of-Experts Actually Help Inference on Consumer and Edge Hardware? An Empirical Study — Alfarizy Alfarizy, Hung Truong Thanh Nguyen, René Richard, Roozbeh Razavi-Far, Hung Cao, 2026 http://arxiv.org/abs/2606.21428 2. Mixtral offloading / Fast inference of MoE with offloading (already cited) and HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inference — Tang et al., 2024 https://scholar.google.com/scholar?q=Mixtral+offloading+%2F+Fast+inference+of+MoE+with+offloading+%28already+cited%29+and+HOBBIT%3A+A+Mixed+Precision+Expert+Offloading+System+for+Fast+MoE+Inference 3. EdgeMoE: Empowering Sparse Large Language Models on Mobile Devices — Yi et al., 2023 https://scholar.google.com/scholar?q=EdgeMoE%3A+Empowering+Sparse+Large+Language+Models+on+Mobile+Devices 4. SwapMoE: Serving Off-the-shelf MoE-based Large Language Models with Tunable Memory Budget — Kong et al., 2023 https://scholar.google.com/scholar?q=SwapMoE%3A+Serving+Off-the-shelf+MoE-based+Large+Language+Models+with+Tunable+Memory+Budget 5. Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference — Hwang et al., 2024 https://scholar.google.com/scholar?q=Pre-gated+MoE%3A+An+Algorithm-System+Co-Design+for+Fast+and+Scalable+Mixture-of-Expert+Inference 6. Mixtral-8x7B / DeepSeek-V2-Lite / Qwen1.5-MoE-A2.7B technical reports and Qwen3-30B-A3B — Jiang et al.; DeepSeek-AI; Qwen Team, 2024-2025 https://scholar.google.com/scholar?q=Mixtral-8x7B+%2F+DeepSeek-V2-Lite+%2F+Qwen1.5-MoE-A2.7B+technical+reports+and+Qwen3-30B-A3B 7. Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts LLMs — Lu et al., 2024 https://scholar.google.com/scholar?q=Not+All+Experts+are+Equal%3A+Efficient+Expert+Pruning+and+Skipping+for+Mixture-of-Experts+LLMs 8. Roofline: an insightful visual performance model for multicore architectures — Williams, Waterman, Patterson, 2009 https://scholar.google.com/scholar?q=Roofline%3A+an+insightful+visual+performance+model+for+multicore+architectures 9. Fiddler / MoE-Lightning / Klotski-style MoE offloading systems for memory-constrained devices — Various, 2024-2025 https://scholar.google.com/scholar?q=Fiddler+%2F+MoE-Lightning+%2F+Klotski-style+MoE+offloading+systems+for+memory-constrained+devices Interactive Visualization: Does MoE Really Speed Up Inference on Edge Hardware?

Todos los episodios de AI Post Transformers

Ajustes

Visualización

Idioma

Determina qué emisoras ves primero en la página de inicio.

Historial

Las emisoras y episodios escuchados recientemente y tus últimas búsquedas solo se guardan en este navegador.

Avisos

Un breve aviso cuando un pódcast que sigues tiene un episodio nuevo. Puedes desactivar pódcast concretos con la campana de su página.

Cuenta

Guarda tus favoritos: en la app y en la web.

Atajos de teclado

/
Abrir búsqueda
Espacio
Reproducir o pausar
← →
En el reproductor: 15 s atrás, 30 s adelante (episodios) · en una fila: siguiente tarjeta
↑ ↓
En una fila: destino dentro de la tarjeta
Inicio / Fin
En una fila: primera o última tarjeta
Esc
Cerrar ventana o menú
?
Este resumen
Reproduciendo ahora

Ir a la emisora

A continuación

    Añade episodios con «A continuación»: sonarán después del episodio actual.