Intelligence per Watt: Measuring Local AI Efficiency Against the Cloud

This episode examines "Intelligence per Watt," a Stanford and Together AI paper that asks whether small local models on laptop-class hardware can take real query load off the cloud. It walks through the paper's definitions. "Local" means at most 20 billion active parameters per token, which includes mixture-of-experts models like gpt-oss-120B. The proposed metric is average task accuracy divided by average power draw, and the paper also reports an energy-per-query (joules) variant. The discussion points out that watts alone ignore latency, so a low-power device that runs slowly can still cost more energy per query. It also flags the measurement caveats: power is read from the accelerator only, not the wall; queries run at batch size 1; chat accuracy is judged against a single reference model's answers; and "best-of-local" coverage is an oracle router that couldn't be deployed. The study spans about a million single-turn queries, more than twenty models and eight accelerators, and the episode places it against earlier LLM energy work such as "From Words to Watts" and MLPerf Power. Listeners get a careful guide to what the metric does and doesn't capture before the headline efficiency numbers are discussed. Sources: 1. Intelligence per Watt: Measuring Intelligence Efficiency of Local AI — Jon Saad-Falcon, Avanika Narayan, Hakki Orhun Akengin, J. Wes Griffin, Herumb Shandilya, Adrian Gamarra Lafuente, Medhya Goel, Rebecca Joseph, Shlok Natarajan, Etash Kumar Guha, Shang Zhu, Ben Athiwaratkun, John Hennessy, Azalia Mirhoseini, Christopher Ré, 2025 http://arxiv.org/abs/2511.07885 2. Intelligence per Watt: Measuring Intelligence Efficiency of Local AI — Jon Saad-Falcon, Avanika Narayan, Hakki Orhun Akengin, et al. (Stanford, Together AI; senior authors incl. John Hennessy, Azalia Mirhoseini, Christopher Ré), 2025 (v7 revised Oct 2026) https://scholar.google.com/scholar?q=Intelligence+per+Watt%3A+Measuring+Intelligence+Efficiency+of+Local+AI 3. MLPerf Power: Benchmarking the Energy Efficiency of Machine Learning Systems from Microwatts to Megawatts for Sustainable AI — Arya Tschand, Vijay Janapa Reddi, et al. (MLCommons), 2024 https://scholar.google.com/scholar?q=MLPerf+Power%3A+Benchmarking+the+Energy+Efficiency+of+Machine+Learning+Systems+from+Microwatts+to+Megawatts+for+Sustainable+AI 4. Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Model — Alexandra Sasha Luccioni, Sylvain Viguier, Anne-Laure Ligozat, 2022 https://scholar.google.com/scholar?q=Estimating+the+Carbon+Footprint+of+BLOOM%2C+a+176B+Parameter+Language+Model 5. From Words to Watts: Benchmarking the Energy Costs of Large Language Model Inference — Siddharth Samsi, Dan Zhao, Joseph McDonald, et al. (MIT Lincoln Laboratory), 2023 https://scholar.google.com/scholar?q=From+Words+to+Watts%3A+Benchmarking+the+Energy+Costs+of+Large+Language+Model+Inference 6. Small Language Models are the Future of Agentic AI — Peter Belcak, Greg Heinrich, Shizhe Diao, et al. (NVIDIA Research), 2025 https://scholar.google.com/scholar?q=Small+Language+Models+are+the+Future+of+Agentic+AI 7. Efficient Memory Management for Large Language Model Serving with PagedAttention — Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, et al., 2023 https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention 8. AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and Acceleration — Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Song Han, et al., 2023 (MLSys 2024) https://scholar.google.com/scholar?q=AWQ%3A+Activation-aware+Weight+Quantization+for+On-Device+LLM+Compression+and+Acceleration 9. Energy Considerations of Large Language Model Inference and Efficiency Optimizations — Jared Fernandez, Clara Na, Vashisth Tiwari, Yonatan Bisk, Sasha Luccioni, Emma Strubell, 2025 https://scholar.google.com/scholar?q=Energy+Considerations+of+Large+Language+Model+Inference+and+Efficiency+Optimizations 10. Part-time Power Measurements: nvidia-smi's Lack of Attention — Zeyu Yang, Karel Adamek, Wesley Armour, 2023 https://scholar.google.com/scholar?q=Part-time+Power+Measurements%3A+nvidia-smi%27s+Lack+of+Attention 11. Energy Use of AI Inference: Efficiency Pathways and Test-Time Compute — Felipe Oviedo et al., 2025 https://scholar.google.com/scholar?q=Energy+Use+of+AI+Inference%3A+Efficiency+Pathways+and+Test-Time+Compute 12. Minions: Cost-Efficient Collaboration Between On-Device and Cloud Language Models — Avanika Narayan, Dan Biderman, Sabri Eyuboglu, Avner May, Scott Linderman, James Zou, Christopher Ré, 2025 https://scholar.google.com/scholar?q=Minions%3A+Cost-Efficient+Collaboration+Between+On-Device+and+Cloud+Language+Models 13. RouteLLM: Learning to Route LLMs with Preference Data — Isaac Ong et al., 2025 https://scholar.google.com/scholar?q=RouteLLM%3A+Learning+to+Route+LLMs+with+Preference+Data 14. PowerInfer: Fast Large Language Model Serving with a Consumer-Grade GPU — Yixin Song, Zeyu Mi, Haotong Xie, Haibo Chen, 2024 https://scholar.google.com/scholar?q=PowerInfer%3A+Fast+Large+Language+Model+Serving+with+a+Consumer-Grade+GPU 15. Green AI — Roy Schwartz, Jesse Dodge, Noah A. Smith, Oren Etzioni, 2020 https://scholar.google.com/scholar?q=Green+AI 16. Implications of Historical Trends in the Electrical Efficiency of Computing — Jonathan Koomey, Stephen Berard, Marla Sanchez, Henry Wong, 2010 https://scholar.google.com/scholar?q=Implications+of+Historical+Trends+in+the+Electrical+Efficiency+of+Computing Interactive Visualization: Intelligence per Watt: Measuring Local AI Efficiency Against the Cloud