
Ep#118: Devol Robotics
Show notes
A robot could line up an ethernet cord almost perfectly and still fail to insert it if it’s off by a millimeter. If robot foundation models had a better understanding of force and contact, this would not be an issue. This week on RoboPapers, we talk to Elijah Ong of Devol Robotics about his team’s approach for vision-language-action foundation models, which allows for precise, accurate world modeling and action generation that also understands contact. They don’t just predict position and velocity, but also the stiffness and damping of an impedance controller, which allows them to complete precise, highly compliant tasks like assembly much more reliably than other methods.
Watch Episode 118 of RoboPapers with Chris Paxton and Ruijie He now!
Abstract
Devol-ONE: One Autoregressive Mixture of Transformers to Unify Vision-Language-Action and Latent World Modeling:
Vision Language Action (VLA) models condition actions directly on current visual and language context, without an explicit account of how the scene evolves under candidate actions. World Action Models (WAM) attempt to address this limitation by predicting future states, but existing designs keep prediction and policy learning architecturally separate, connecting them only through the predicted output, whether through pixel space video generation or a latent forecasting module trained independently of the policy. We present Devol-ONE, a Mixture of Transformers architecture that unifies vision language understanding, latent world dynamics prediction, and action generation within a single autoregressive framework. Instead of encoding vision language tokens once and feeding them to the action expert, Devol-ONE runs autoregressive prediction jointly across a vision language stream and a V-JEPA pretrained dynamics stream, attending to the vision language key-value cache at every layer to forecast future latent states under language guidance. The action expert is in turn shaped continuously by semantic reasoning and predicted physical dynamics rather than by a fixed representation computed in advance. Extensive experiments are conducted on LIBERO, LIBERO-PLUS, RoboTwin2.0 along with real-world evaluation on Flexiv single-arm and dual-arm setups. Ablation studies show the effectiveness of dynamic stream prediction and layer-wise unified attention to validate our model architectural coherency.
Interaction World Model:
The Interaction World Model is a family of robot foundation models built on Interaction Geometry. IWM 1.0 learns a visuo-impedance map that connects visual context, geometric robot state, and the directional structure of motion. Geometric anchoring keeps physical information represented in the latent used by prediction, planning, and action generation.
The model predicts reference pose, velocity, stiffness, and damping for an impedance controller. It learns this behavior from approximately 1,000 hours of heterogeneous pretraining data, without force, torque, or explicit stiffness supervision. Its non-backbone action pathway contains approximately 42 million trainable parameters.
Learn More
Devol-ONE on ArXiV: https://arxiv.org/abs/2609.32193
Transcript
Edited for length and clarity. Timestamps mark the start of each section.
00:05 · Learning the physics of contact
Chris Paxton: Hey everyone, and welcome to another episode of RoboPapers. Today we have Elijah Ong from Devol Robots here to talk about representations and architectures for robot foundation models. Elijah, can you tell us a little about yourself?
Elijah Ong: Hi everyone. I’m Elijah, CTO and co-founder of Devol Robots. I’ve been working in robotics for around ten years. What I’m really curious about is how we can use different learning representations and architectures to improve robotic manipulation, especially the foundational skills.
At Devol, we build physics-grounded foundation models that learn the geometry and physics of contact directly from trajectory data. We build on what we call interaction geometry, our Riemannian framework for physical intelligence. The idea is that a model can transfer across embodiments, from a single arm to dual arms and even humanoids.
My co-founder, Sze, has about a decade of experience in manufacturing and industrial engineering. He leads production and industrial deployments, and I lead technology development. We founded the company in 2023. We have around 20 people and we’re headquartered in San Francisco. Our website is devolrobots.ai.
Chris: One reason I really wanted to talk about this is that I saw your demos in person. They were incredibly impressive. We’ll get to those later, but I figured I might as well say that right away.
Elijah: Let me start with two problems. In factories, a lot of processes are already automated. Many of the ones that are still hard involve interactions between objects. You might start with a highly variable pickup and end with a very precise jig or fixture. Take optical lenses. You have to pick up a lens that could be sitting in any of a number of poses, then put it into a coating jig. That’s challenging in production.
The second problem is that many vision-language-action models, or VLAs, and world-action models give little attention to physics and control. They don’t have a physics prior built in. They plan, and the motion coming out of the model is often executed open loop. They also need significant amounts of data for pretraining and fine-tuning.
Our approach has three parts. First, our Interaction World Model, or IWM, uses impedance as a physics prior. We represent the trajectory data in the appropriate geometric spaces and use a JEPA model to learn in latent space. We want the model to understand stiffness and damping.
Second, Devol-ONE brings semantic understanding, world prediction, and action planning into one architecture. The model predicts actions conditioned on how it expects the world to change.
Finally, we close the loop with an execution layer. If the model predicts a good action chunk but the robot executes it badly, it can still knock the object over. We want each chunk to execute precisely and smoothly across different robot embodiments and controllers.
Chris: Before we move on, why is it so important to capture physics? It’s something I’ve always wanted to ask about these models. What does it buy you?
Elijah: Imagine asking a robot to pick up a cable and plug it into your phone. Understanding the instruction gets you part of the way. But just before insertion, the robot has to regulate force very precisely. At what moment does force matter? How much force? In which direction does the robot need to be stiff? A lot of contact-rich tasks depend on getting those things right.
RJ He: In the phone example, you also implicitly understand the shape of the connector and the port it goes into. Tactile sensing alone doesn’t give you all of that. Are you talking about an implicit world model, the physics of how those two objects interact, or both? Is there a distinction?
Elijah: We started with impedance because we wanted an abstraction that could capture physical interaction across different hardware and sensing modalities. Tactile sensing is important. The question is how to represent that signal so the model can use it.
We put the robot’s proprioceptive data into a Riemannian framework. Once the data lives in the appropriate curved space, we can extract how compliance changes along the trajectory and use that schedule as a supervision signal. IWM 1.0 did this without force/torque sensing. With 2.0, we added force/torque feedback and saw a significant improvement. I’d expect tactile sensing to help too. We still need to work out the right prior for learning from it.
Chris: So part of your thesis is that action supervision alone isn’t enough. If the model understands something about the physics, it’ll be more robust. Is that the right way to think about it?
Elijah: Yes.
12:13 · What can you infer from a trajectory?
RJ: Can you give us an introduction to the Riem…
This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit robopapers.substack.com