Pular para o conteúdo

Does MoE Really Speed Up Inference on Edge Hardware?

20 de novembro de 2026

Notas do episódio

This episode examines whether Mixture-of-Experts models deliver real speed and energy savings on consumer and edge hardware, using a 64-expert OLMoE-1B-7B model, about 1.3B of whose 6.9B parameters are active per token, against three small dense models on an Apple M2 Pro laptop and an 8GB Jetson Orin Nano. It explains why batch-size-one decoding is limited by memory bandwidth rather than arithmetic, so that sparse compute does not automatically translate into faster tokens. The discussion tests the paper's headline claim that cost tracks total parameters, not active ones. It also raises the question of whether the relevant quantity is instead the bytes actually read per token, and points to on-device MoE work such as EdgeMoE, SwapMoE and HOBBIT that the paper overlooks. It also walks through the benchmarking setup: llama.cpp with 4-bit quantization, 45 runs per model-and-device cell, a thread-count sweep, and tegrastats-based energy measurement on the Jetson only. Listeners get a clear explanation of the MoE memory-versus-compute tradeoff and a critical look at how much of a measured result comes from the architecture and how much from a vanilla runtime. Sources: 1. Does Mixture-of-Experts Actually Help Inference on Consumer and Edge Hardware? An Empirical Study — Alfarizy Alfarizy, Hung Truong Thanh Nguyen, René Richard, Roozbeh Razavi-Far, Hung Cao, 2026 http://arxiv.org/abs/2606.21428 2. Mixtral offloading / Fast inference of MoE with offloading (already cited) and HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inference — Tang et al., 2024 https://scholar.google.com/scholar?q=Mixtral+offloading+%2F+Fast+inference+of+MoE+with+offloading+%28already+cited%29+and+HOBBIT%3A+A+Mixed+Precision+Expert+Offloading+System+for+Fast+MoE+Inference 3. EdgeMoE: Empowering Sparse Large Language Models on Mobile Devices — Yi et al., 2023 https://scholar.google.com/scholar?q=EdgeMoE%3A+Empowering+Sparse+Large+Language+Models+on+Mobile+Devices 4. SwapMoE: Serving Off-the-shelf MoE-based Large Language Models with Tunable Memory Budget — Kong et al., 2023 https://scholar.google.com/scholar?q=SwapMoE%3A+Serving+Off-the-shelf+MoE-based+Large+Language+Models+with+Tunable+Memory+Budget 5. Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference — Hwang et al., 2024 https://scholar.google.com/scholar?q=Pre-gated+MoE%3A+An+Algorithm-System+Co-Design+for+Fast+and+Scalable+Mixture-of-Expert+Inference 6. Mixtral-8x7B / DeepSeek-V2-Lite / Qwen1.5-MoE-A2.7B technical reports and Qwen3-30B-A3B — Jiang et al.; DeepSeek-AI; Qwen Team, 2024-2025 https://scholar.google.com/scholar?q=Mixtral-8x7B+%2F+DeepSeek-V2-Lite+%2F+Qwen1.5-MoE-A2.7B+technical+reports+and+Qwen3-30B-A3B 7. Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts LLMs — Lu et al., 2024 https://scholar.google.com/scholar?q=Not+All+Experts+are+Equal%3A+Efficient+Expert+Pruning+and+Skipping+for+Mixture-of-Experts+LLMs 8. Roofline: an insightful visual performance model for multicore architectures — Williams, Waterman, Patterson, 2009 https://scholar.google.com/scholar?q=Roofline%3A+an+insightful+visual+performance+model+for+multicore+architectures 9. Fiddler / MoE-Lightning / Klotski-style MoE offloading systems for memory-constrained devices — Various, 2024-2025 https://scholar.google.com/scholar?q=Fiddler+%2F+MoE-Lightning+%2F+Klotski-style+MoE+offloading+systems+for+memory-constrained+devices Interactive Visualization: Does MoE Really Speed Up Inference on Edge Hardware?

Todos os episódios de AI Post Transformers

Configurações

Exibição

Idioma

Define quais estações você vê primeiro na página inicial.

Histórico

As rádios e episódios ouvidos recentemente e suas últimas buscas ficam só neste navegador.

Avisos

Um aviso rápido quando um podcast que você segue tem um episódio novo. Desative podcasts individuais pelo sino na página deles.

Conta

Salve favoritos – no app e no site.

Atalhos de teclado

/
Abrir busca
Espaço
Reproduzir ou pausar
← →
No player: voltar 15 s, avançar 30 s (episódios) · na fileira: próximo bloco
↑ ↓
Na fileira: alvo dentro do bloco
Home / End
Na fileira: primeiro ou último bloco
Esc
Fechar janela ou menu
?
Esta visão geral
Tocando agora

Ir para a estação

A seguir

    Adicione episódios com “A seguir” – eles tocam depois do episódio atual.