E22 deploying models on small microcontrollers

We talk with Sebastian Boblest and Marco Mader Mendes from Bosch about the surprisingly challenging art of deploying AI models on resource-constrained embedded devices. They share insights from their work on ETAS Embedded AI Coder, a tool that generates optimized C code from neural network models for microcontrollers—sometimes with as little as 100 bytes of RAM available.

The conversation covers practical strategies for model compression (often by factors of 100x or more), the counterintuitive benefits of float models over quantized ones on tiny devices, and why feature engineering still matters. Sebastian and Marco explain how they navigate the trade-offs between RAM, compute time, and accuracy, and why each project presents unique constraints—from confidential hardware specs to compiler quirks. They also discuss real-world applications, including Bosch's AI-powered wall scanner that uses radar and neural networks to detect cables in walls.

Key Topics:

• [03:30] Why Bosch started exploring embedded AI on tiny hardware in 2020
• [06:45] Real-world application: AI-powered wall scanner using radar to detect cables
• [09:20] How ETAS Embedded AI Coder generates C code from neural network models
• [12:00] Deploying models with as few as 300 parameters and 100 bytes of RAM
• [16:30] Model compression strategies: squeezing networks down by 100x or more
• [21:15] When float models outperform quantized ones on tiny devices
• [28:00] Trading off RAM vs. compute time through code generation techniques
• [33:45] Benchmarking challenges with confidential hardware and compilers
• [40:20] AutoML and architecture search for constrained embedded targets
• [46:00] Free tool access for universities and opportunities for students

Notable Quotes:

"We go down to applications where we use 100 bytes of RAM and you can still do something useful with this on a Cortex-M0. It's very surprising how small you can get with neural networks." — Sebastian Boblest

"Sometimes we have to squeeze it down not by one or two X, sometimes it's up to 100 X and more. This is really a regular task for us." — Marco Mader Mendes

"One thing that's super counterintuitive for many people on these smaller devices is to go from a quantized model to a float model. It can actually help you. It's exactly the opposite of what people do on these larger systems." — Sebastian Boblest

Resources Mentioned:

• ETAS Embedded AI Coder - Code generation tool for deploying neural networks on embedded devices; free for universities
• ARM CMSIS-NN - Library containing functions for neural network layers on Cortex-M devices
• MLPerf Tiny Benchmarks - Benchmarking suite for tiny ML systems that Sebastian's team participated in
• Bosch AI-powered Wall Scanner - Radar-based power tool using neural networks to detect cables in walls
• Agile Embedded Podcast Slack - Community discussion channel where you can reach Ryan and Luca, with a dedicated sub-channel for Embedded AI topics