RoboPapers
- Episódios
- 115
- Publicação
- várias vezes por semana
- Último episódio
- 10 nov 2026
Sobre o podcast
Chris Paxton & Michael Cho geek out over robotic papers with paper authors.
Episódios 115
Ep#116: SimEX: Simulation-Integrated Robotics AutoResearch
Coding agents powered by large language models are increasingly able to complete complex, long-horizon goals, but this capability is still largely relegated to computer tasks. Bringing agents to real-world …
Notas do episódio
Coding agents powered by large language models are increasingly able to complete complex, long-horizon goals, but this capability is still largely relegated to computer tasks. Bringing agents to real-world robotics is much harder.
Enter SimEX, by Jiaheng Hu, Zhenyu Jiang, and company. It is a simulation-based auto-research framework, wherein agents can develop skills which can be used to control real robots, developing a robust toolbox of skills which can then be used for real-world tasks like towel folding and plate manipulation.
To learn more, watch Episode 116 of RoboPapers with Michael Cho and Jiafei Duan!
Abstract
Coding agents powered by large language models (LLMs) have shown remarkable abilities to autonomously reason about and achieve goals in the digital world. However, bringing this success to the physical world remains challenging. On the one hand, direct generation methods (e.g., Code as Policies) often suffer from the LLMs' insufficient understanding of robots and physical environments. On the other hand, iterative trial-and-error tuning in the physical world (e.g., physical autoresearch) induces significant experimental cost and safety concerns. We introduce SimEX: Simulation-Integrated Robotics AutoResearch, an autoresearch framework that tightly integrates simulated experimentation, enabling coding agents to efficiently acquire physical capabilities for controlling real robots. SimEX operates in two stages. First, the agent conducts open-ended probe-and-optimize iterations in simulation, developing a robot toolbox with robust and generalizable capabilities. Second, the agent adapts the toolbox and the simulator together through only a few physical trials: each trial corrects the simulator, and the corrected simulator is used to diagnose failures and screen candidate repairs. We evaluate SimEX extensively in sim-to-sim settings and on physical robots. On challenging real-world manipulation tasks including towel folding, barcode scanning, and plate manipulation, SimEX enables coding agents to efficiently acquire robot skills without any demonstration and with only 10 minutes of real-robot interaction. These results suggest that simulation can be a critical component in achieving physical intelligence, not only as a source of training data that must closely replicate the real world, but also as a roughly correct laboratory where a coding agent develops the knowledge and procedures needed to act on the robot. More details and robot videos at this https URL
Learn More
Project Page: https://robo-simex.github.io/
ArXiV: https://arxiv.org/abs/2609.38982
Transcript
Edited for clarity and readability.
Introducing SimX
Michael Cho: Welcome to another episode of RoboPapers. Today we have Jiaheng Hu and Zhenyu Jiang, two of the coauthors of SimX: Simulation-Integrated Robotics Auto-Research. Thank you for making the time. Could you each share a little about yourselves?
Jiaheng Hu: Thank you for inviting us. I’m a final-year PhD student at UT Austin, co-advised by Professor Peter Stone and Professor Roberto Martín-Martín. My research has always focused on building self-improving robots that can continually learn and adapt in the real world.
Recently, I’ve become more interested in how foundation models, such as VLAs and coding agents, can facilitate that process. During my PhD, I’ve interned at the Allen Institute for AI, Google DeepMind, and Amazon FAR. The project we’re discussing today was my internship project at Amazon.
Zhenyu Jiang: I’m a research scientist at Amazon FAR. I graduated from UT Austin last year, where I worked with Yuke Zhu. I now work on manipulation. My focus is straightforward: how do we make robotics work in the real world? Whatever makes it work, I’ll try.
Bringing trial and error to robotics
Jiaheng Hu: One of the biggest motivations for SimX is the recent success of agentic AI. Coding agents are a primary example, but there are also applications such as AlphaEvolve, where agents discover new algorithms.
These agents are powerful partly because the LLMs behind them are smart, but also because they can perform trial and error in the digital world. They implement something, see whether it works, and improve it based on feedback from the environment.
As roboticists, we want to know whether we can bring this kind of agentic AI to robots. Previous work falls into two main categories.
The first is direct generation. A representative approach is Code as Policies: you tell a coding agent that you want a robot to perform a particular task, and it writes a program that will hopefully solve it. The problem is that direct generation loses one of agentic AI’s key strengths: the ability to learn from mistakes through trial and error. You’re relying entirely on the LLM’s intrinsic knowledge, and its physical knowledge is often insufficient to solve a task zero-shot.
The second category does use trial and error. A representative example is Embodied-π, where a coding agent controls a robot, collects real-world rollouts, and updates its program based on the results. In principle, that process can continue indefinitely.
But the physical world introduces issues that don’t exist in the digital world. One is time: you can’t accelerate time on a real robot. You have to wait for it to complete each trial. Another is safety: a robot can break itself. You often need human knowledge or hard constraints to keep exploration safe.
The key question behind SimX is whether we can bring simulation into this physical auto-research process.
Why use an imperfect simulator?
Jiaheng Hu: Simulation offers several benefits. First, it’s an encoding of human physical knowledge. A robot’s URDF tells you how it moves, and a simulator can tell you how objects respond to collisions according to physical laws.
Second, a simulator is written entirely in code. That makes it easy for a coding agent to inspect, observe its outcomes, and modify it.
Finally, a simulator is essentially a world model. It can answer counterfactual questions: what would happen if I took this action instead of that one, or executed this skill instead of another?
Of course, simulation doesn’t exactly match the real world, and making it match perfectly is very difficult. The idea behind SimX is to embrace simulation’s strengths while tolerating its imperfections.
We do that through a two-stage process. Throughout both stages, we optimize a robot toolbox with two components: code and usage knowledge.
The code consists of skills or APIs that the robot develops through simulation experiments. The usage knowledge describes how to call particular skills and how to approach particular tasks.
In the first stage, SimX operates entirely in simulation. You can think of it as analogous to pretraining. We use the coding agent’s ability to discover problems in the toolbox and create new simulations, tasks, and reward functions. It alternates between identifying weaknesses, generating training tasks, and hill climbing on those tasks.
The result is a toolbox containing many skills. We want those skills to be robust across different situations so that they have a chance of transferring to a real robot.
In the second stage, we take the toolbox into the real world and adapt it to a particular downstream task. Simulation helps us answer counterfactual questions about real-world failures and decide how to repair the toolbox.
Stage one: Open-ended discovery
Jiaheng Hu: At the start of each iteration, the coding agent generates a broad range of probing tasks. Their purpose is to expose weaknesses in the current toolbox.
We run the toolbox in those simulated environments and observe where it succeeds or fails. Based on those results, the coding agent identifies a particular weakness. For example, the toolbox might be bad at grasping objects near the edge of a tote. The agent can then generate a set of tasks focused on that problem.
Once we have those tasks, we run a standard auto-research loop. The coding agent proposes a fix, tests it, examines the res…
This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit robopapers.substack.comEp#115: Delta-0 (Δ₀): A New Chapter in Humanoid Intelligence
Delta Intelligence’s Δ₀ is a humanoid full-body foundation model, which can perform long-horizon, whole body loco-manipulation. They show a single, generalist policy which can control a 69-DoF humanoid to …
Notas do episódio
Delta Intelligence’s Δ₀ is a humanoid full-body foundation model, which can perform long-horizon, whole body loco-manipulation. They show a single, generalist policy which can control a 69-DoF humanoid to perform everyday tasks at nearly human speed — using whole-body human data for large-scale pretraining.
Xiaojian Ma and Siyuan Huang tell us more about how they achieved this, and about the importance of jointly designing policy and controller. Delta uses a delta action space, for example, which makes it easy to provide human interventions without interrupting the whole body controller: a human operator can correct the robot seemlessly while it works.
They show a wide range of impressive humanoid loco manipulation tasks using their generalist model.
Learn more on Episode 115 of RoboPapers, with Michael Cho and Chris Paxton!
Learn More
Blog post: https://deltai.com/en/blog/delta-0
Transcript
This transcript has been edited for clarity and readability. Repetitions, false starts, and incidental screen-sharing chatter have been removed.
Introducing Delta Intelligence
Michael Cho: Welcome to another episode of RoboPapers. Today we’re glad to have Xiaojian and Siyuan. Xiaojian is the CEO and Siyuan is the chief scientist of a new startup called Delta Intelligence. They’ve just released their model, Delta-0, along with a detailed technical blog post.
We’re very glad to have both of you here to share your work. To start, could you each tell us a little about yourselves? Xiaojian, perhaps you can go first.
Xiaojian Ma: Thanks, Michael and Chris, for the invitation. I’m Xiaojian, the CEO and founder of Delta Intelligence. Delta is dedicated to solving general intelligence for humanoid robots. We’re focusing on bipedal robots because we believe humanoids are the future and a general form of physical intelligence.
We’re building foundation models that include both the brain—the large models—and the controller. We also work on hybrid force-position control for humanoids. Delta-0 is our first foundation model, showcasing the capabilities we currently have on Unitree G1 robots. We’re expanding beyond Unitree to other robot vendors in China and elsewhere, because we want our model to have cross-embodiment capabilities.
Before founding Delta, I was a researcher in machine learning and robotics. I returned to China to join BIGAI as a research scientist in embodied AI. Before that, I was a PhD student at UCLA. During my PhD, I did research internships with teams at Google Robotics, NVIDIA Research, and Google DeepMind.
My projects mostly involved building large multimodal models for reasoning and embodied decision-making. That’s the background I’m coming from.
Siyuan Huang: Thank you for the invitation, Chris and Michael—two old friends. We’ve shared papers with you and my students several times, but this is the first time we’re discussing a blog post about Delta-0, our new foundation model.
I’m the chief scientist of Delta. We released this blog at the end of last month, and it has prompted discussion about how to approach humanoid intelligence and build foundation models for whole-body loco-manipulation.
We’re excited about the development of humanoid foundation models. This blog summarizes how we think we can build better models and address the coupled problems of the controller, hardware, and model.
Watching the Demo
Siyuan Huang: Let’s go over the demo first. We have two robots in an indoor environment. The first is doing what we call whole-body dexterous manipulation. The hardware is a Unitree G1 with two WUJI hands. The first task is putting a record onto a turntable.
Michael Cho: And doing a dance, too. Nice.
Siyuan Huang: Our expectation is that, in the future, we can have robots in our homes doing the same kinds of things people do: putting on a record, listening to music, doing a little dance, and taking a rest.
They can also do everyday housework, such as making the bed, picking up a game controller, and arranging the bedding. These are long-horizon loco-manipulation tasks. They require whole-body coordination to finish everyday activities, such as picking up a toy from the ground and putting it on the bed.
The robot can also interact with articulated objects, applying force to open and close doors. Here it’s opening the dishwasher and using its left hand for support so it can apply more force.
Chris Paxton: That’s really cool. It was bracing itself against the fridge and pulling with the other hand.
Xiaojian Ma: That’s right. It’s like using your core to adjust your body and pull with more force.
Chris Paxton: It’s very human, very fluent.
Siyuan Huang: Here it checks the food in the refrigerator—and takes a rest as well.
Michael Cho: This is so cool. It’s really long horizon.
Chris Paxton: It’s a fantastic video. I also want to note how few cuts there were. Was there just one, when you switched from the first robot to the second?
Siyuan Huang: There are actually two cuts.
Michael Cho: The second part is continuous. All this opening of doors is nontrivial. Very impressive. Please take us through it.
Siyuan Huang: The video has two parts. The first shows very dexterous manipulation. The second shows very long-horizon loco-manipulation: about three minutes of uncut video, with nine atomic tasks and more than thirty steps. It’s challenging for the robot to finish all of those.
We’ve seen companies such as Figure show videos like this. As Michael said, opening and closing objects requires a lot of interaction with articulated objects. Connecting those tasks is particularly hard because each task starts from wherever the previous one ended.
We don’t use any SLAM-based navigation skill in this video. The model drives navigation, upper-body interaction, and lower-body interaction. All of that comes from one model.
Why Loco-Manipulation Is Hard
Siyuan Huang: First, we want to explain how we think about humanoid foundation models and why humanoid intelligence matters.
The humanoid form offers the possibility of working wherever people do. We think it’s a general form that can work in more places than a wheeled robot. Humanoid intelligence turns that possibility into useful, reliable actions.
Right now, we’ve seen a lot of demonstrations. In the future, we want to turn those capabilities into deployments. That’s the vision of Delta Intelligence.
One challenge is long-horizon loco-manipulation. We expect the robot to perform extended activities, rather than just individual pick-up actions in front of a table. Every action changes the conditions for what comes next: the surroundings change, the robot’s location changes, and the mode of interaction changes. It’s difficult to ensure stability throughout a long-horizon task.
We see loco-manipulation as one of the most challenging unsolved problems in robotics, and a core problem for humanoid intelligence. There are several reasons.
First, the robot must control a moving body while perceiving and manipulating a changing world. That’s different from manipulation alone. Previously, we often tried to separate locomotion and manipulation.
For locomotion, an RL-based controller can make the robot dance or perform extreme movements. We’ve discussed impressive examples on RoboPapers, such as OmniXtreme, where a single policy produces a broad range of movements. That’s locomotion and whole-body control.
For manipulation, you can have a fixed base and use a gripper or dexterous hand. But fixing the base limits the robot’s action space. We want a larger action space, more like a human’s, so we combine locomotion and manipulation. The robot needs to move while perceiving and interacting with a changing world. That creates a large action space and a difficult coupled problem.
Second, each action determines the starting conditions for the next. That’s one reason long-horizon tasks remain a holy grail of robotics. A large language model can decompose a task into atomic subtasks, but physically connecting them requires precise positioning and adjustment. The robot needs to tr…
This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit robopapers.substack.comEp#114: Oopsie Data
Robotics has a data problem, but not the one you’re thinking of: almost all robotics data is of successful instances of robot behaviors. But many of the most exciting research areas for robotics today …
Notas do episódio
Robotics has a data problem, but not the one you’re thinking of: almost all robotics data is of successful instances of robot behaviors. But many of the most exciting research areas for robotics today, including world models and reward model learning, require failures: interesting examples of manipulation trajectories that did not succeed.
Claas Voelcker, Arpit Bahety, and Paul Zhou join us to talk about their data collection effort and their upcoming workshop on “oopsie data”: all these valuable failures that are currently being thrown out. Their goal is to collect the first large community-driven open-source dataset of failure examples.
Watch Episode 114 of RoboPapers, with Chris Paxton and Jiafei Duan, to learn more!
Learn More
Project site: https://oopsie-data.com/
This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit robopapers.substack.comEp#113: MolmoBOT: Large-Scale Simulation Enables Zero-Shot Manipulation
Using simulation to teach robots would unblock the massive scale of robotics data that seems to be missing right now. The issue is usually sim-to-real transfer; MolmoBOT aims to solve this with sufficient …
Notas do episódio
Using simulation to teach robots would unblock the massive scale of robotics data that seems to be missing right now. The issue is usually sim-to-real transfer; MolmoBOT aims to solve this with sufficient diversity of simulation data, so that no real data is necessary for robot deployment. It has a few parts: MolmoBot-Engine produces diverse pick-and-place demonstrations across many different simulated environments; MolmoBot-Data is a dataset of 1.7 million expert trajectories for item manipulation and pick-and-place tasks; and they train a number of VLA policies which can achieve strong performance on real-world pick-and-place tasks with zero real robot data. Rose Hendrix, Snehal Jauhri, Abhay Deshpande, and Maya Guru join us to give us the details.
To learn more, watch RoboPapers episode 113 with Chris Paxton and Ruijie He!
Abstract
A prevailing view in robot learning is that simulation alone is not enough; effective sim-to-real transfer is widely believed to require at least some real-world data collection or task-specific fine-tuning to bridge the gap between simulated and physical environments. We challenge that assumption. With sufficiently large-scale and diverse simulated synthetic training data, we show that zero-shot transfer to the real world is not only possible, but effective for both static and mobile manipulation.
We introduce MolmoBot-Engine, a fully open-source pipeline for procedural data generation across robots, tasks, and diverse simulated environments in MolmoSpaces. With it, we release MolmoBot-Data, a dataset of 1.7 million expert trajectories for articulated object manipulation and pick-and-place tasks. We train three policy classes: MolmoBot (a Molmo2-based VLM with a flow-matching action head), MolmoBot-Pi0 (replicating the π₀ architecture for controlled comparison), and MolmoBot-SPOC (a lightweight policy for edge deployment). Without any real-world fine-tuning, our policies achieve zero-shot transfer to unseen objects and environments, reaching 79.2% success rate on real-world tabletop pick-and-place, outperforming π₀.₅ at 39.2%.
Learn More
Project Page: https://allenai.github.io/MolmoBot/
ArXiV PDF: https://arxiv.org/pdf/2603.16861
Transcript
Produced by ChatGPT and Riverside.
Introducing MolmoBot
Chris Paxton: Hey everybody, and welcome to another episode of RoboPapers. Today we’re excited to have the team behind the MolmoBot paper here to talk about their work. Would you like to introduce yourselves?
Rose Hendrix: I’m Rose Hendrix. Until recently, I was a research scientist at AI2, and now I’m at a startup.
Abhay Deshpande: I’m Abhay Deshpande. I conducted this work while I was a predoctoral researcher at AI2, and now I’m a PhD student at Princeton University.
Maya Guru: I’m Maya Guru. I’m currently a predoctoral researcher at AI2, and I’m looking for PhD positions for this fall.
Chris Paxton: Awesome. Would you like to get started and tell us a little about what you’ve been working on? Rose, maybe you can share your screen?
Abhay Deshpande: I’ll pull it up.
Rose Hendrix: Thanks, Abhay. On a previous episode of RoboPapers, the MolmoSpaces team came on to talk about their work, which is closely related to this. MolmoSpaces is about scaling synthetic environments and evaluation. MolmoBot is about cashing out that potential: scaling synthetic data generation, then training policies on that data to achieve zero-shot sim-to-real transfer for manipulation.
MolmoBot is a data engine and a family of policies. We show that the policies transfer in large-scale simulation evaluations and in the real world, both for tabletop pick-and-place and for mobile manipulation on the RB-Y1 platform. That includes opening doors, drawers, and cabinets, as well as object manipulation and pick-and-place.
Generating synthetic robot data at scale
Abhay Deshpande: As Rose mentioned, we have this large framework, MolmoSpaces, for synthetic data and large-scale simulation. Within that, we built MolmoBot-Engine, a data engine for generating extremely diverse, large-scale datasets.
The basic idea is to roll out heuristic demonstrators in these environments, then apply very aggressive randomization: camera extrinsics and intrinsics, textures, lighting, colors, and all sorts of other things. You keep rolling that out and scaling with compute, and you get high-throughput, large-scale data.
For a concrete example, you sample an arbitrary scene from MolmoSpaces. There are more than 100,000 scenes available. Then you sample a workspace, choose an object for a pick-and-place operation, place a receptacle for the object, and put the robot in the workspace.
From there, we randomize lighting, the robot’s initial configuration, inertial and frictional parameters, and other aspects of the scene. Then we roll out a scripted demonstrator. Since we have access to privileged information in simulation, a fairly simple heuristic policy can perform well under a wide variety of conditions.
Of course, there are so many things we could randomize: textures, lighting, robot configurations, camera poses, task instructions, templates, and much more. How do we decide which ones matter? What we found is that a good philosophy is basically: yes. If you can randomize it, you generally should. That extreme diversity goes a long way toward making policies robust in the real world.
We generate several different modalities. By default, we have two external cameras and a wrist camera. Since we’re using the DROID setup, we replicate those cameras in simulation: ZED 2 cameras for the external views and a ZED Mini for the wrist camera. We also have a randomized GoPro view.
The GoPro has a very wide field of view and significant fisheye distortion. We apply post-processing to warp the rectified image so it looks like what the real GoPro would produce.
In addition to RGB and depth, we save the camera intrinsics, since we’re randomizing them. We also save task-relevant object keypoints, which lets us compute visibility heuristics or do image-space reasoning downstream.
For actions, we save all of the action representations rather than prescribing just one. That way, people using MolmoBot-Data can support a wide variety of downstream applications. We also save proprioception, joint positions and velocities, end-effector poses and twists, and other task-relevant privileged state: the demonstrator’s current phase, contact forces and points, grasps, door states, and so on.
Finally, we label everything with rewards, success indicators, scene metadata, task prompts, and language referring expressions.
Chris Paxton: One question: all these grasps and door states you’re talking about—where do those come from? Are those heuristics?
Abhay Deshpande: Generally, they’re derived from the scripted heuristics or read directly from simulation. A door state might just be a joint angle, which we can access directly. The same goes for contact points and forces. Others, like rewards and success, are based on predicates: conditions that need to evaluate to true for us to say the task succeeded.
Rose Hendrix: This was also motivated by the fact that knowing what the demonstrator is trying to do can be a useful training signal. That’s a strength of synthetic data. If you’re collecting human teleoperation demonstrations, it isn’t necessarily possible to extract that kind of nuanced information about what the operator intended at any given moment.
RJ He: Was all the data scripted, or were there other sources?
Abhay Deshpande: It was all scripted. We found that fairly simple heuristic policies go a long way. We have around 1.7 million demonstrations, and they’re all generated with scripted policies. By randomizing enough dimensions, you get surprisingly far.
You can see in the data distribution that the scripted motions sometimes look a bit robotic—which, of course, is fitting for a robot. But we didn’t encounter major downstream problems, and the policies performed well in the real world. It might become more of an issue when co-training with dat…
This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit robopapers.substack.comEp#112: SlipSense and Touch2Trace
Touch is crucial for humans when they want to perform precise tasks to a high degree of reliability, and yet robots almost universally use vision — they’re often completely blind to this whole modality. Two …
Notas do episódio
Touch is crucial for humans when they want to perform precise tasks to a high degree of reliability, and yet robots almost universally use vision — they’re often completely blind to this whole modality. Two works from Tong Jian, Tao Yu, Saleh Nabi, and Matteo Grimaldi show us how important this sense is. In SlipSense, they look at how to detect objects slipping in grippers using a compact multimodal tactile sensor, which generalizes across unseen objects and to different robots. They then talk about how tactile sensing can be used to enable high real-world performance with a challenging cable-tracing task, where a cable must be fed through the hand with a repeated pinch-and-curl motion.
To learn more, watch Episode 112 of RoboPapers today, with Michael Cho and Ruijie He!
SlipSense Abstract
Slip detection is fundamental to dexterous manipulation, yet existing systems often lack precise characterization of detection latency and cross-platform generalization. We present SlipSense, a multimodal tactile slip-detection framework built on TacV5, a compact sensor integrating a 32×32 piezoresistive array operating at 240 Hz and a 3-axis MEMS accelerometer operating at 8 kHz. The piezoresistive array captures spatial pressure distributions, while the accelerometer captures friction-induced vibrations, providing complementary slip cues. The framework performs modality-specific encoding, intra-sensor fusion, and cross-modal attention with causal temporal prediction at 240 Hz. Experiments on a dataset of 1.4 million frames spanning 37 objects demonstrate the complementarity of the two modalities. SlipSense achieves 96.7% Macro F1 with a false-positive rate below 1.6%, detecting 76% of slip events within 23.1 ms. When trained solely on UMI data, SlipSense generalizes zero-shot to a Tesollo dexterous hand, transferring across unseen objects, distinct sensor units, and robotic platforms without retraining.
Touch2Trace Abstract
Dexterous manipulation of deformable objects demands continuous fingertip-level regulation of pressure, friction, and incipient slip. We study one of the most challenging cases: dexterous cable tracing, feeding a cable through the hand with repeated pinch-and-curl motions of the thumb and index finger. We introduce Touch2Trace, a tactile-driven imitation-learning system for this task, and provide, to our knowledge, the first systematic real-world characterization of how encoder pretraining, control rate, temporal context, and spatial resolution each shape policy performance. The winning learning recipe combines a tactile encoder pretrained for a custom 32 x 32 piezoresistive sensor (TacV5) via self-supervised learning with a lightweight transformer policy trained on teleoperated demonstrations via behavior cloning, deployed at 60 Hz on a Tesollo DG-5F hand. Tactile feedback without vision or explicit cable-state estimation significantly improves tracing performance versus a proprioception-only baseline: from 0.2 cm to 20.1 cm mean distance and 0% to 93% success rate, with zero-shot transfer to unseen cables and routing conditions. The results quantify the influence of key parameters in tactile-driven systems for reliable dexterous deformable object manipulation.
Learn More
SlipSense Project Page: https://tjian-ai.github.io/corl26-slipsense/
SlipSense on ArXiV: https://arxiv.org/abs/2609.15910
Touch2Trace on Project Page: https://grmmtt.github.io/corl26-touch2trace/
Touch2Trace on ArXiV: https://arxiv.org/abs/2609.15921
Transcript
Introduction
Michael Cho: Hello, everyone. Welcome to another episode of RoboPapers. Today we’re really happy to have the crew from Analog Devices, the Dexterous AI Group: Tao Yu, Tong Jian, Saleh, and Matteo. They’re coauthors of two papers that they’ll walk us through. One is Touch2Trace: Tactile-Driven Imitation Learning for Dexterous Cable Tracing, and the other is SlipSense: Multimodal Tactile Learning for Low-Latency and Generalized Slip Detection.
Thank you so much for coming, guys. Maybe we can start with everyone introducing themselves, and then go through the papers. Tao, do you want to start?
Tao Yu: Yeah, great. Thanks. My name is Tao Yu. I lead the DexAI team, short for Dexterous AI Group, at Analog Devices. The team was set up about two years ago, and our mission is to unlock humanoid dexterity. That’s how we started looking into tactile sensing, which we believe is one of the missing pieces in dexterous manipulation.
With me are the other team members. Maybe Tom can go next.
Tong Jian: Yeah, I’m Tom. I’ve been with the DexAI team for two years, working on applied machine learning and, more recently, robot learning.
Saleh: Matteo, do you want to go next?
Matteo Grimaldi: Hello, everyone. I’m Matteo. I’ve been working with the DexAI team for two years now, mostly on robot learning.
Saleh: And I’m Saleh. I also joined Dexterous AI about two years ago. When I met Tao, we were still prototyping a lot of things on the sensor side. The two works you’ll see today are the result of a lot of interesting internal research, both on the hardware side and on the algorithm side.
Tao Yu: I also want to add that I personally believe this isn’t just a hardware problem or an algorithm problem. It’s a combination of both. One question we’ll touch on in these papers is what hardware capabilities can unlock learning, and vice versa: what can learning tell us about the hardware requirements?
We believe the path forward is a loop: create new hardware, integrate it with a robot, train a model, demonstrate whether the new hardware capability improves performance, and then use what we’ve learned to design the next generation of hardware.
Michael Cho: Very cool. I’m really looking forward to these two papers. Which one do you want to start with?
Tong Jian: Maybe we can start with SlipSense. Let me share my screen.
Michael Cho: Okay, cool.
SlipSense: Detecting Slip Quickly and Reliably
Tong Jian: Can you see the screen? Great. We’ll be talking about two works accepted at CoRL 2026, both built on our TacV5 tactile sensor. They demonstrate tasks where vision alone isn’t enough.
I’ll begin with SlipSense, and Matteo will follow with Touch2Trace. As the title suggests, SlipSense studies slip detection. When an object starts to slip from a robot gripper or fingers, we want a model that can detect it quickly and reliably.
Slip detection has a long history. Researchers have been working on it for decades, and the core problem is well understood. But two things remain relatively underexplored. The first is detection latency. A lot of prior work reports accuracy without really characterizing latency.
To emphasize, detection latency is not the same as inference speed. It’s the time between when slip actually begins—the slip onset—and when the model first detects it. That’s the interval between these two lines on the plot. A model might run in two milliseconds, but if it needs 100 milliseconds of evidence to make a correct prediction, that’s still 100 milliseconds of detection latency. By the time the model detects slip, the object may already be gone.
That’s why we care so much about detection latency. It depends both on the sensor’s capabilities and on the model’s capabilities.
RJ He: Can I jump in here? This might be a naive question, but it sounds like you’re saying that different materials and different objects can actually produce different slip-detection latencies. Is that correct?
Tong Jian: Yes. It depends on many things: the material, the object, the actual slip speed, how quickly the sensor can capture the relevant slip features, and how quickly the model can interpret those features and make the correct detection. All of those things matter. There’s not just one dominant factor.
RJ He: I see. And my assumption is that this is a learned detector, not just a property of the sensor?
Tong Jian: Yes, exactly. It’s learned. And that brings us to another important aspect: detection latency naturally trades off against false alarms. It’s a sensitivity-versus-specificity…
This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit robopapers.substack.comEp#111: SOLE-R1: Video-Language Reasoning as the Sole Reward for On-Robot Reinforcement Learning
Reinforcement learning is often seen as key to achieving scalable, highly reliable real-world robot performance, yet it requires good reward functions that can supervise the wide range of tasks that need to be …
Notas do episódio
Reinforcement learning is often seen as key to achieving scalable, highly reliable real-world robot performance, yet it requires good reward functions that can supervise the wide range of tasks that need to be performed. The obvious thing to try would be a large vision-language model (VLM), but these models often fail under common real-world conditions like partial observability and occlusion.
As a solution Philip Schroeder and Thomas Weng propose SOLE-R1, a video-language reasoning model trained to serve as a reward signal for online RL. Given real video, SOLE-R1 will reason over input video frames, generates reasoning traces, and predicts a reward signal. This allows online RL from random initialization on new manipulation tasks.
To learn more, watch Episode 111 of RoboPapers, with Michael Cho and Jiafei Duan!
Abstract
Vision-language models (VLMs) have shown impressive capabilities across diverse tasks, motivating efforts to leverage these models to supervise robot learning. However, when used as evaluators in reinforcement learning (RL), today's strongest models often fail under partial observability and distribution shift, enabling policies to exploit perceptual errors rather than solve the task. We introduce SOLE-R1 (Self-Observing LEarner), a video-language reasoning model explicitly designed to serve as the sole reward signal for online RL. Given only raw video observations and a natural-language goal, SOLE-R1 performs per-timestep spatiotemporal chain-of-thought (CoT) reasoning and produces dense estimates of task progress that can be used directly as rewards. To train SOLE-R1, we develop a large-scale video trajectory and reasoning synthesis pipeline that generates temporally grounded CoT traces aligned with continuous progress supervision. This data is combined with foundational spatial and multi-frame temporal reasoning, and used to train the model with a hybrid framework that couples supervised fine-tuning with RL from verifiable rewards. Across four different simulation environments and a real-robot setting, SOLE-R1 enables zero-shot online RL from random initialization: robots learn previously unseen manipulation tasks without ground-truth rewards, success indicators, demonstrations, or task-specific tuning. SOLE-R1 succeeds on 24 unseen tasks and substantially outperforms strong vision-language rewarders, including Robometer, RoboReward, ReWiND, GPT-5, and Gemini-3-Pro, while exhibiting markedly greater robustness to reward hacking. We release all models, data, code, and demos at the anonymous page: this https URL
Learn More
Project page: https://philip-mit.github.io/sole-r1/
ArXiV: https://arxiv.org/abs/2603.28730
Transcript
Transcripts are automatically generated by Riverside and are lightly edited by ChatGPT.
Michael Cho [00:06]
Hello guys, welcome to another episode of RoboPapers. Today we are really happy to have Philip Schroeder and Thomas Weng, two of the co-authors of this paper, SOLE-R1: Video-Language Reasoning as the Sole Reward for On-Robot Reinforcement Learning. Thank you so much for making the time, guys. Maybe as a start, Philip, you can share a bit about yourself and then Thomas.
Philip Schroeder [00:27]
Yeah, so my name is Philip. I’m a final-year PhD student at MIT in computer science. My work has mainly focused on improving the reasoning capabilities of language and vision-language models in challenging embodied settings. And over the last couple years, my work has mainly focused on reward reasoning over video, which is what we do for this paper.
Thomas Weng [00:55]
Cool. I’m Thomas. I’m a research scientist at RAInstitute. And more recently my work has been focusing on the pieces required to achieve like post-training and deployment for robotics. So reward modeling is one piece of that. On-robot reinforcement learning where you have data from deployment is also a piece of that. But that’s what I do.
Michael Cho [01:22]
All right, great to have both of you on. Yeah, maybe Philip you can take us through.
Philip Schroeder [01:27]
Yeah, yeah. Okay. So the goal of this paper was to develop a vision-language model that has strong enough reward reasoning that we can train policies from scratch through online reinforcement learning. So the model we developed is called SOLE-R1. And we show that it can serve as the sole reward signal for learning over 20 different unseen manipulation tasks through online RL. And I guess the motivating problem for this paper was that if you use just general vision-language models or VLM-based reward models that have recently come out, you get this reward hacking behavior when you try to train a policy from scratch using the predicted rewards.
So here, the task is to open the drawer, and you can see that these VLM-based reward models give very high rewards just for the policy fiddling with the drawer and just slightly opening it. And then here’s another example with the Pick Cube task in RoboSuite. Again, you see that the policy is getting these very high rewards by just finding these odd camerangles and positions that trick the VLM into thinking the task is being accomplished without actually doing the task. And you can see here the SOLE-R1 predicted rewards are plotted here in blue, and the ground-truth rewards are plotted in red. And you can see that solar SOLE-R1. Much more robust to this type of reward hacking.
And you kind of think of this paper as a recipe for how to train a reward model that is robust to reward hacking. And there’s kind of two major components to this that differentiate our work from prior work and really help with this reward hacking robustness. And I can quickly mention those and then go over them in more detail later. So the first is our training data mixture. We do I think three unique things with our training data mixture. One is that we include a lot of general spatial reasoning data. And a lot of this data is new. It came out with NeurIPS 2025 and ICLR 2026 papers. So at the time of this work it was data sets that were just released.
So it includes just general spatial re spatial reasoning data SSR chain-of-thought And some robotics data sets, some recent one called From Seeing to Doing, and some very famous ones like Embody chain-of-thought and Robo2VLM. And we combine this with data that forces the model to reason over multiple images within the same space, capturing different moments in time. So this includes general spatial reasoning data or spatiotemporal reasoning data spot the diff and robot-specific video data that we generate and then. Also curate from Open X-Embodiment. And our video data includes failure datand I think much more failure data than prior work because we generate a lot of it ourselves in RoboCasa. So that’s one big thing.
And then another big thing is that we annotate the time each time step of the videos with dense progress values and dense chain-of-thought reasoning traces to help the model explicitly reason about progress with these failure trajectories or suboptimal trajectories. And then we mix this with these artificial failure data from Open X So that’s the training data mixture. And then another novel thing we do is have the model do this chain-of-thought reasoning at each time step of the video. And this has been shown to significantly improve language model performance and visual language model performance. So and we show ablations where it helps significantly.
So the models con it at each time step is reasoning about what’s changing over time between the current and previous time step and predicting a dense progress value. Relative to the previous progress value that it predicted at the last time step. And then we do a lot of experiments that I can that I can show. But let me know if you have any questions so far. I can go over in more detail the both how we generate our training datand then how we train this chain-of-thought reasoning over video.
Jiafei Duan [05:56]
Can you let me yeah. Yeah, I’m gonna go ahead. Yeah, no, you go ahead.
Michael Cho…
This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit robopapers.substack.comEp#110: ENPIRE: Agentic Robot PolicySelf-Improvement in the Real World
In many fields, automated research has become increasingly plausible with the improving capabilities of frontier language models, and the field robotics may be no exception. In ENPIRE, Wenli Xiao, Jia Xie, and …
Notas do episódio
In many fields, automated research has become increasingly plausible with the improving capabilities of frontier language models, and the field robotics may be no exception. In ENPIRE, Wenli Xiao, Jia Xie, and Tonghe Zhang build a harness framework that captures physical feedback and allows for automated robotics research, minimizing human effort while training policies that achieve a 99% success rate on challenging, dexterous tasks like organizing a pin box or fastening a zip tie.
Learn more in Episode 110 of RoboPapers, with Michael Cho, Chris Paxton, and Ruijie He!
Abstract
Achieving dexterous robotic manipulation in the real world heavily relies on human supervision and algorithm engineering, which becomes a central bottleneck in the pursuit of general physical intelligence. Although emerging coding agents can generate code to automate algorithm search, their successes remain largely confined in digital environments. We conjecture that the missing abstraction to automate robotics research is a repeatable feedback loop for real-world policy improvement: reset the scene, execute a policy, verify the outcome, and refine the next iteration. To bridge this gap, we introduce ENPIRE, a harness framework for coding agents that instantiates this physical feedback routine with four core modules: an Environment module (EN) for automatic reset and verification, a Policy Improvement module (PI) that launches policy refinement, a Rollout module (R) to evaluate policies with one or multiple physical robots operating in parallel, and an Evolution module (E) in which coding agents analyze logs, consult literature, improve training infrastructure and algorithm code to address failure modes. This closed-loop system transforms real-world manipulation learning into a controllable optimization procedure, minimizing human effort while allowing fair ablations across training recipe and agent variants. Powered by ENPIRE, frontier coding agents can autonomously train a policy to achieve a 99% success rate on challenging, dexterous manipulation tasks, such as organizing a pin box, fastening a zip tie, and tool use, a process that further accelerates when we dispatch an agent team on a robot fleet. Our results suggest a practical and scalable path toward deploying coding agents to autonomously advancing robotics in the physical world.
Learn More
Project Page:
ArXiV: https://arxiv.org/abs/2606.19980
Transcript
Michael Cho (00:06.088)Hello everyone, welcome to another episode of RoboPapers. Today we’re super happy to have Wenli, Jia Xie and Tonghe. They are three of the co-authors of this paper called ENPIRE: Agentic Robot Policy Self-Improvement in the Real World. Thank you so much for coming to the pod. Maybe as a start three of you can share a little bit about yourself first and then you guys can go through the paper.
Wenli Xiao (00:30.821)Okay, I’ll get started. Sorry. So my name is Wenli. I’m a final-year PhD at CMU, advised by Professor Guanya Shi. Previously I did two-year internship at NVIDIA GEAR Lab with Jim Fan and Yuke Zhu. And later I joined Physical Intelligence as intern.
Yeah, I did a bunch of VLA training, post-training, real-world RL and agentic stuff and now I’m mainly doing pretraining, studying the science of training and data. Yeah.
Michael Cho (01:14.488)Yeah, I think just yeah.
Jia Xie (01:16.463)Okay, sure. Yeah. Hi everyone. My name is Jia and I’m a master’s student at CMU, advised by Guanya. Of course we’re labmates. And I’ve interned at xDof as a research intern over the summer. And I’ve been working on some agent stuff and the extent stuff lately. So yeah, thanks for having us.
Tonghe Zhang (01:43.739)Okay. Hi, my name is Tonghe. I’m also a master’s student at Professor Guanya Shi’s lab. I was doing internship at Amazon this summer and previously I’ve been working on reinforcement learning. And thank you guys for inviting us.
Michael Cho (02:01.15)Alright, perfect. Great to have you guys. Yeah, Wenli, I know you have a slide, maybe you can go through with us.
Wenli Xiao (02:08.228)Okay, yeah. So yeah, today the main topic is to introduce our paper, but as we all know, recently there’s a major breakthrough in AGI and that affects robotics, that is GPT-6 Astra. There are a lot of demos from Twitter that shows people apply this model to do different tasks.
That related to robotics, like real-to-sim, sim-to-real, in-context learning, and code-as-policy. And we are very excited and we are kind of a little bit frustrating and also wondering how well robotics been solved. But we will leave the discussion to later section. And because even before the GPT-6 Astra
Already witnessed a huge improvement of coding agent this year because we felt like coding agent is a biggest thing in AI this year because starting from February or last year’s December we see like a lot of a lot very good coding agent like Opus and Codex. They can do a lot of stuff like starting from front-end to
Back-end and build a lot of great applications. And actually, people have been trying to use these models in robotics every time there’s a breakthrough in AI. And for this year there’s a CaP-X project, which we apply the and benchmark the state-of-the-art LLM model and VLM model to
Try to solve and hill-climb on the robotic simulation benchmarks and it did pretty well in simulation. And we also tried to use coding agent to just solve the very challenging manipulation benchmark. This is a very classical task which would you need to manipulate a dot to push a T-shaped object until you perfectly matches the target.
Wenli Xiao (04:31.492)And this was really challenging and people find a solution or people didn’t find a solution until diffusion policy. But now we just prompt the coding agent out of the box. This is actually Codex. We prompted to write a heuristic policy with no neural network training to achieve a hundred percent success rate. It just try and try and eventually it can like
Write very good heuristic b policy. And this applies to different coding agents like Codex Claude Code and Kimi from Moonshot. This was very exciting to me. And actually during the past two years there are many impressive works about applying coding agent for robotics.
So here I just like briefly divide them into three categories, like VLA plus harness, so you can achieve great performance that can be better than either VLA or code-as-policy itself. And there’s of course Code-as-Policy, which people apply coding agent to try and practice in simulation so they can learn some skills and later apply to the real-world.
Real world robotics. And there’s also physical autoresearch, which is the ENPIRE work that we are gonna discuss today. So the major difference between the first two categories and the physical autoresearch is even all of them applied the method in the real-world. For the first two part of first two categories, they need tons of human effort.
So if we summarize the procedure of the researchers do these projects, it’s like they apply coding agent to write code and do some practice in simulation and also with some guidance of human, it develops some impressive skills. And then the researchers, PhD students, deploy the policy in the real-world and watch the results.
Jia Xie (06:54.974)Okay.
Wenli Xiao (06:55.682)And maybe for the first trial it doesn’t work, or even the robot just do nonsense. And then the student just describes what happened and type in the terminal, and then the coding agent fix it and do another trial error and apply and suddenly it works, and then can submit paper. So in this process we observe that humans are a major bottleneck.
Because we become the agent’s actuators and sensors. Yeah, because the observation was from like six months ago, now with GPT 6 extra, maybe we don’t need to be the sensors of the robot. GPT can like if you set up a webcam in a third-person view, maybe GPT can observe that the policy doesn’t work.
But six months ago we didn’t have such good GPT and we have GPT 5.5 and we still…
This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit robopapers.substack.comEp#109: Enigma - Connecting 100 Robots To The Internet
What happens if you connect a hundred AI-powered robots to the internet and just let people give them any instructions they want? Enigma founder Jonathan Jacobi joins us to talk about his experience deploying a …
Notas do episódio
What happens if you connect a hundred AI-powered robots to the internet and just let people give them any instructions they want? Enigma founder Jonathan Jacobi joins us to talk about his experience deploying a fleet of robots on the internet, keeping them operational, and watching people truly test the generality and robustness of frontier robotics models.
Watch Episode 109 of RoboPapers with Chris Paxton and Ruijie He today!
Learn More
Robots.online website (no longer active)
Transcript
Transcript is automatically generated by Riverside and edited by ChatGPT.
Chris Paxton (00:05)
Hey everyone, and welcome to another episode of RoboPapers. Today we’re super excited to have Jonathan Jacobi from Enigma here to tell us about their work in foundation models and about putting a hundred robots online.
So, could you introduce yourself?
Jonathan Jacobi (00:19)
Yeah. Thank you, Chris. Thank you, RJ. It’s a pleasure to be on RoboPapers. Big fans.
A bit about Enigma: we’re a company working on foundation models for robotics, as Chris said, as well as building innovative new interfaces for robotics. How should we interact with robots? How should the interface feel to users?
We’re also interested in the back-and-forth between interface work and research on the models.
About a month and a half ago, we put a hundred real robots online. You could actually interact with them over the internet. They were running our models through a variety of interfaces, and you could simply ask them to do different things.
So we’re here to talk about the models, the interfaces, and the experience we’ve built.
Chris Paxton (01:21)
Awesome. So what did people do with all these robots that you put online?
Jonathan Jacobi (01:25)
The basic concept was that we wanted people to feel robotics today.
We all see a lot of demos. We see a lot of cherry-picked demos—some of them are cherry-picked, some of them aren’t—and we wanted people to feel that something is here right now.
One of the problems is that not enough people have robots in their houses. So how do we make enough people feel that robotics is happening now, while we still have this challenge of not having enough robots deployed in the world for the average person to use?
We thought quite hard about how to make that possible. One of the best ideas we came up with was: maybe we can set up a huge warehouse with a lot of robots, put our models on those robots, and let anyone online interact with them.
We also got a very cool domain: robots.online.
We tried to create enough different setups that one robot could be in one scenario, another robot could be in another scenario, and people would have enough flexibility to do random things that they could really feel the AI work we’ve done.
They could ask the robot to do things themselves, without the logistical challenge of putting a hundred humanoids all over the world.
That’s why we used tabletop robots—arms that we customized and designed ourselves, building on public open-source designs—and let people interact with them freely.
Some robots were artists. People could paint. People could do pantomime. People did very unexpected things.
Someone asked one of the robots holding a paintbrush to pretend that the brush was a broom.
Jonathan Jacobi (03:50)
So it actually swept the floor with the paintbrush.
We have some very cool examples of that that we can walk through later.
The concept was that, at Enigma, we focus on generalization—text, video, audio—and we have very different interfaces and input methods for our models. We wanted people to be able to experience that generalization directly and see that it’s working today.
Some of the robots were artists. Some were doing chemistry experiments. We had one that could defuse a bomb. We had one that could do lightsaber-esque fighting.
Not actual lightsabers. Don’t sue me.
But basically, we let people do different things with different robots online.
RJ He (05:01)
Okay, cool. Can you describe a little more of the interface that was set up?
It sounds like each one is a single manipulator fixed to a tabletop, and then the interface is a regular chat-prompt type of thing, as if you were speaking to an AI agent. Is that right?
Jonathan Jacobi (05:21)
Yeah. That’s a good question.
We had a static manipulator. One of them was based on the SO-101 design. We took the SO-101 and said, “We want this to look really polished.”
So we brought in incredibly talented industrial designers and worked with them for about a month or two. They were working full-time on designing this new version.
The robot you’re seeing on the screen is an SO-101 that we customized. The colors, the enclosure, everything was custom-designed. You can see that even the cables are hidden within the robot rather than hanging around it.
These were static manipulators, and each table had a different type of interface.
The artist robot had a chat interface similar to an AI agent. But we wanted to make it less boring than simply asking the robot to paint or perform pantomime, so we turned it into a game.
You could give it an idea and it would paint something associated with that idea, and then you had to guess what it painted.
For pantomime, you could tell it to pretend that an object was something else, and the robot would mime with it.
So that was one chat-based interface.
Then we had other interfaces. Let me walk through some of the things we built.
This is our company page. You can see more than a hundred robots here. This is part of the actual warehouse. Those are spare robots and leftover components that we built.
Here’s one of the chat interfaces where people were doing pantomime. You could do drawing, pantomime, and a few other things.
And here’s another interesting interface.
You could tap objects directly on the screen and then ask the robot to do something with them. You could tap the green bottle and say, “Pick this up,” then tap the orange flask and say, “Pour this into this.”
Jonathan Jacobi (07:45)
You could say, “Throw this here,” or whatever you wanted.
We wanted to introduce a new kind of interactive interface—not just text, not just teleoperation, not just audio, but something new.
Our vision is that if you extrapolate this ten or a hundred steps forward, the interface almost feels like a game. It feels like you’re playing the physical world.
Think about something like StarCraft. Imagine having an interface where you can control a hundred robots. You can tap, type, drag and drop, and build agents on top of that to automate entire processes.
That’s where we imagine the world going. Controlling robots becomes incredibly intuitive: you can type, tap, drag, drop, maybe talk to them.
Another interface was the bomb-defusing robot. You were on a mission to defuse a bomb, and while you’re trying to do it, an operator is shouting at you: “You have to defuse it faster!” So you’re under pressure while talking to and controlling the robot.
People did very unexpected things with this.
Someone said, “Knock on the bomb twice at the bottom.”
Someone else said, “Pour the orange flask over the pink button.”
The pink button wasn’t standing upright—it was lying on the table.
Chris Paxton (09:45)
Can I ask—what simulation is that? That actually looks pretty good. The water physics there.
Jonathan Jacobi (09:50)
This is the physical world. This is not simulation.
Chris Paxton (09:52)
That’s not simulation? Okay. Well, that makes sense.
RJ He (09:54)
No way.
Jonathan Jacobi (09:56)
Yes. Every one of these is the physical world. None of this is simulation.
Chris Paxton (10:01)
Right. I knew the others were real. I don’t know why that one looked like a simulation to me. Sorry for doubting you there.
RJ He (10:05)
Yeah, agreed.
Actually, maybe that’s a good question.
First, what was the main motivation for doing this whole thing? Was it about the human-robot interface, or was it about seeing what kinds of prompts and questions people would ask?
And then the second question is about this simulation-versus-real-world issue. Why did you need to do i…
This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit robopapers.substack.comEp#108: VLS: Steering Pretrained Robot Policies via Vision–Language Models
Pretrained vision-language-action models for robots often fail when a task is performed in a slightly different environment. With Vision-Language Steering, you can get better generalization performance out of a …
Notas do episódio
Pretrained vision-language-action models for robots often fail when a task is performed in a slightly different environment. With Vision-Language Steering, you can get better generalization performance out of a frozen robot policy, by steering the sampling process of your pretrained diffusion or flow-matching policy to handle out-of-distribution changes like obstacle position or changing visual appearance. Shuo Lin and Ishneet Sukhviner Singh join us to talk about how to better generalize robot policies with no new training data.
Watch Episode 108 of RoboPapers today, with Jiafei Duan and Ruijie He!
Abstract
Why do pretrained diffusion or flow-matching policies fail when the same task is performed near an obstacle, on a shifted support surface, or amid mild clutter? Such failures rarely reflect missing motor skills; instead, they expose a limitation of imitation learning under train–test shifts, where action generation is tightly coupled to training-specific spatial configurations and task specifications. Retraining or fine-tuning to address these failures is costly and conceptually misaligned, as the required behaviors already exist but cannot be selectively adapted at test time.
We propose Vision–Language Steering (VLS), a training-free framework for inference-time adaptation of frozen generative robot policies. VLS treats adaptation as an inference-time control problem, steering the sampling process of a pretrained diffusion or flow-matching policy in response to out-of-distribution observation–language inputs without modifying policy parameters. By leveraging vision–language models to synthesize trajectory-differentiable reward functions, VLS guides denoising toward action trajectories that satisfy test-time spatial and task requirements.
Learn More
Project Page: https://vision-language-steering.github.io/webpage/
ArXiV: https://arxiv.org/pdf/2602.03973
Transcript
Transcripts are lightly edited by ChatGPT.
RJ He (00:01)
Okay. Hi everyone. Welcome to another episode of Robot Papers. Today we are super excited to have with us Shuo Liu and Ishneet, who will be sharing their paper on Vision-Language Steering: Steering Pre-Trained Robot Policies via Vision-Language Models. Shuo, Ishneet, do you all want to introduce yourselves?
Shuo Liu (00:29)
Yes. Hi everyone. My name is Shuo Liu. For now I’m a research engineer at Cortex Robot AI and previously I graduated from University of Washington, where I was also a student researcher at Allen Institute for AI, advised by Professor Ranji Krishna and Professor Jiafei Duan, focus on robot learning from develop foundation models to deploying them into real world.
RJ He (01:01)
Isn’t it?
Ishneet (01:02)
Yeah. Hi everyone, my name is Ishneet. I work with Professor Jiafei on this paper, VLS, with Shuo. I’m currently at Menlo Research, where I also work on Asimov, and I’ll be going to Oxford for my undergrad later this year.
RJ He (01:17)
Super exciting. Amazing that you’re already having publications at this young age. It’s awesome. Okay, without further ado, Shuo, why don’t you take us through the paper?
Shuo Liu (01:26)
Okay, sure, yeah. Let me share the screen. Okay, so just make sure it’s it’s good to say right.
RJ He (01:38)
Yeah, it’s good.
Shuo Liu (01:39)
Yeah. Hi everyone. Today I want to share our work vision language theory. This work is finished when I was a master student at the University of Washington and advised by Professor Ranjik Krishna and Professor Jiafei Duan. And so today I want to introduce motivation of this work and the method we solve this problem and the experiment results and also some future works maybe could be done from this direction. So the problem is from the observation of some small front twining of the Like models like π0 or π0 Po, we found that current generative robot policies are can’t are kind of tightly coupled of the training specific configurations, so which cause some failing to decouple the reusable motor skills from the normal task constraints. Like for example, here we have some cubes on the table and if you want to stack them up to a tower with specific numbers you may call the π0.5 pre-trained value a policy to stack some cubes like if you want to stack two cubes and using the termination signal like the pushing the red button on the table it can do very well because you Because for this specific task you can fine twin the base policy a bit, like stack two cubes and push the button or stack four cubes to push the button. And but what if at test time you ask the base policy to stack three cubes? It’s never it’s never contained in the fine-tuning data set, so the base policy
Shuo Liu (03:33)
Can learn how to stack two cubes or stack four cubes and push the button, but it will never know how to st how to stack three cubes and push the button. This is because the training configuration we combine the language instruction and the image observation with the task two cubes and four cubes together and the polic the base policy is never contained with task specific instruction stack three cubes. So the motor skills like moving the gripper to the cube and stack it on another one and going push the red button is already included in the base policy. But at test time if you given another out of distribution scenario or out of distribution language instruction, the base policy can always fail. So we think this is from the standard imitation learning reset. So for now if we have a base policy like take π0.5 or Momo Act 2 for example, the base policy can be trained on the behavior cloning target, which means given an expert data set which contains observation, action, and the language for each index and the optimization target is to maximize the likelihood of the base policy to be to predict the next action chunk based on current observation and the absurd language instruction. So the objective is stand static and also is very distribution dependent, which means after fun training your base policy can only capture the special and semantic correlations on the training manifold if you change the observation or language at test time the base policy may confuse. This observation is clear at our daily work because if you take one pre-trained VLA policy you just don’t want to like use it in the
Shuo Liu (05:53)
In distribution tasks, you may change the scenario, you may change the language to the VLA. So for example, sometimes we may face observation out of distribution signal which means we may made some visual or special shift and some semantic shift which means the change of language distribution. So at that time the world may drift away from the training distribution and how we can solve it. We for now we have some two solutions. The first one is we recollect a small dataset out of distribution scenario and run another training cycle. But we need to repeat this circle for every new deployment and make the base policy useful for this specific scenario. But this is kind of misaligned with the target of we train very large VLA policy. That’s because the target is to make the base policy general generalizable but and we can using the pre-train and post-train recipe to make the skills like pick and place or open close this kind of low level skills already inside the VLA policy but we are paying the cost to relear the existing skills at test time s simply because we are not selectively adapt to the base policy at test time. So in this work we want to train we want to try another way that is steering the base policy at inference time, which means we’re taking the we’re taking the ability of generative models like photomatching or diffusion models which are
Shuo Liu (07:46)
Broadly used in current VLAs like the action part action expert part. We can take the advan advantag advancement of their ability to denoise iteratively. So for example the diffusion model is denoising to the target distribution from Gaussian noise and using many diff denoising steps and the flow matching are learning denoising paths to from the Ga…
This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit robopapers.substack.comEp#107: Do As I Do: Dexterous Manipulation Data from Everyday Human Videos
Achieving human-like dexterity is the next frontier for robotics, and yet dexterity data is often subtly hard to scale. Real-world dexterity data, including things like finger-pose estimates, is often slightly …
Notas do episódio
Achieving human-like dexterity is the next frontier for robotics, and yet dexterity data is often subtly hard to scale. Real-world dexterity data, including things like finger-pose estimates, is often slightly off, making it physically invalid and hard to execute on real hardware and hard to learn from.
DO AS I DO is an algorithm for reconstructing and retargeting monocular RGB videos to robot hands, outperforming the state of the art and even working from generated videos. Bhawna Paliwal, Haritheja Etukuru, Will Liang, and Mahi Shafiullah join us to tell us more.
Watch Episode 107 of RoboPapers, with Chris Paxton and Jiafei Duan, now!
Abstract
How can we scalably generate data for robotic manipulation, especially on human-like platforms such as dexterous multi-fingered hands? Learning from human videos has recently emerged as a likely answer to this question. However, difficulties in estimating hand-object interaction and crossing the human-to-robot embodiment gap have hindered the adoption of abundant monocular RGB-only human videos as the primary source of robot manipulation data. In this work, we present DO AS I DO, an algorithm to reconstruct and retarget monocular RGB human videos to multi-fingered dexterous robotic hands. DO AS I DO reconstructs hand-object interactions from various egocentric and exocentric in-the-wild video sources. The algorithm then retargets these hand-object interaction estimates into a sequence of actions executable in the real world, yielding robot-complete manipulation data from disparate human videos. Overall, DO AS I DO outperforms previous state of the art in estimating hand-object interactions and extracting dexterous manipulation trajectories from RGB videos, as we show in experiments on datasets with ground truths and on a dataset of video clips collected online. Our experiments enable us to propose an efficacy playbook for practitioners collecting human data for manipulation.
Learn More
Project Page: https://do-as-i-do.com/
ArXiV: https://arxiv.org/abs/2606.19333
Github: https://github.com/malik-group/do-as-i-do
Transcript
Lightly edited for transcription errors, names, technical terminology, punctuation, and readability. Conversational phrasing is otherwise preserved. Editing done automatically with ChatGPT.
Chris Paxton (00:05.479)
Hey everyone, welcome to another episode of RoboPapers. Today we have some great work from Bhawna, Haritheja, Will, and Mahi: “Do as I Do: Dexterous Manipulation Data from Everyday Human Videos.” Welcome to the show, guys. Could you introduce yourselves quickly before we get started?
Bhawna Paliwal (00:24.977)
Yeah, hi, I’m Bhawna. I’m a PhD student here at UC Berkeley, advised by Jitendra Malik.
Haritheja Etukuru (00:32.692)
I’m Haritheja. I’m a second-year PhD student at Berkeley, also advised by Jitendra.
William Liang (00:37.929)
I’m Will. I’m also a second-year PhD student, advised by Jitendra and Pieter.
Mahi Shafiullah (00:42.606)
Hey everybody, I’m Mahi. I’m a postdoc currently at Berkeley, and I’m also spending some time at Amazon FAR right now.
Chris Paxton (00:50.983)
Awesome. Well, yeah, would you guys like to tell us a little bit about this? What have you guys been working on?
Bhawna Paliwal (01:01.499)
So I’ll just start with a little introduction to the paper. So mainly the motivation behind this work was how can we utilize the massive video data available on internet for training robots. And most of the prior work has been which which has involved learning from human videos, has involved co-training with robot data, which which eventually involves having like hundreds or thousands of hours of teleop data combined with, let’s say, twenty thousand hours of video. So our objective in this paper is to understand. Whether there is some intermediate representation, let’s say in in 3D, where we can extract the right kind of information from human videos and make learning from human videos much more efficient. So essentially every human video where hand-object interaction is going on turns into a separate trajectory on which you can train like a robot model. So prior work has looked into this problem mostly through by collecting their own data and maybe having like specific settings where they’re They have like specific angles and specific robot specific hands hand-object tasks available. But here we look at internet videos in general where there might be a lot of occlusion, motion blur and so on, which are not necessarily recorded for robot learning. Yeah.
Chris Paxton (02:21.331)
Right. Some of the videos here in the highlights look incredibly good, particularly with the dexterous hand. So how does this work?
Bhawna Paliwal (02:35.113)
So I think As like broadly there are like two steps. First we have reconstruction where we reconstruct the object in 3D, as shown here. And here and then we retarget it onto a robot hand, in our case the Sharpa hand. And for reconstruction, like the hardest part was object tracking. That is most of the times when we are looking into hand-object videos, the objects are mostly occluded, especially the tiny objects such as this one. There are like just fewer than a hundred pixels where the object is actually visible. And reconstructing the object becomes hard first. And secondly, tracking the object becomes even harder, even if you know the shape. So, in this work, we first we utilize like all the computer vision progress that has been made recently, especially models like SAM 3D, which can just like take a single image and give you the shape and pose of the object in that particular frame. And then we come up with our own method of how do you utilize the SAM 3D kind of methods to also track the object over time. And the motivation behind was just behind that was just this that SAM 3D has been trained on lots of images and artificially constructed images where like some particular object is blurred out or it has been hidden by some other object. And then so it knows when the how the object looks when it’s like heavily occluded. And then what we do is like we predict the shape and pose at one particular frame and then propagate it to next into like further frames. while keeping the constraint that the shape of the object should not change too much and things like that. So I let Will explain the retargeting part. Yeah. So
Jiafei Duan (04:18.402)
Yeah, I think I have a sorry. I have a question. Yeah, so I actually I read I read the paper quite a few times. and I find it very interesting. So what we are seeing here is an open-loop replay, is that my understanding? Right. So would that be like any limitation that you guys foresee? I mean, maybe I’m jumping ahead of time, but I’m just curious, right? Like for example, certain tasks when you are, you know, doing requires some level of precision or perception, then how will you address that? Or is it just more of the motion that matters at this point?
Bhawna Paliwal (04:51.655)
Yeah, so these trajectories are right now open loop and the how we foresee this should be changed into a perceptive task, which is definitely necessary, is by training taking this reference, domain randomizing the object position, and then training an RL policy, let’s say using point cloud as input. So this will involve just like since you already have the reference trajectory in simulation, you also know the object shape, and then you can like train an RL policy. The object pose is changing and let’s say like even the your environment is changing, lighting and so on, and then you can do all these kind of domain randomization tricks to kind of make it work for one object. And then once you have lots of such policies, you can probably distill it into one big model with let’s say behavior cloning. yeah. Mahi, maybe you would like to add more things.
Mahi Shafiullah (05:46.155)
Yeah, I mean I think one of the things that I think about when it comes to this project is really that we’re exploring how to get robot data from kind of arbitrary i…
This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit robopapers.substack.comEp#106: Flex-π: A Multi-Stream World-Action Model with Compute Flexibility
World Action Models are becoming more popular in robotics, as they learn to predict the world jointly with learning how to act on it. However, these world predictions are usually purely based on reconstructing …
Notas do episódio
World Action Models are becoming more popular in robotics, as they learn to predict the world jointly with learning how to act on it. However, these world predictions are usually purely based on reconstructing color images from video. This is limiting, because color is far from the most important quality for a robot moving around in the world — more important are qualities like 3D geometry and object semantics. In Flex-π, Ge Yan, Jesse Zhang, and team train a 6 billion parameter world action model to do exactly this, predicting 3D pointmaps and DINO features along with color. This results in a policy which is much more demonstration-efficient, generalizes well, and can perform complex long-horizon tasks.
Learn more in Episode 106 of RoboPapers, hosted by Jiafei Duan and Ruijie He!
Abstract
World-action models (WAMs) predict the future to act better, but nearly all of them predict only RGB latents, trained purely for pixel reconstruction, with no explicit signal for the 3D geometry or object semantics manipulation needs. We find a surprising free lunch: the same frozen video-generation VAE that encodes RGB also encodes 3D pointmaps almost losslessly, with no pointmap-specific training at all. This lets us supervise Flex-π, a 6B-parameter WAM, on 3D geometry and object-centric DINO semantics alongside RGB, at no cost in new sensors, new pre-training, or inference latency. Every visual signal is projected into this shared latent space and denoised jointly with actions inside a Mixture-of-Transformers backbone; per-stream dropout with cross-modality forcing then lets a single trained checkpoint run on any subset of these streams, from a fast action-only mode to full joint generation. The result is a policy that is exceptionally demonstration-efficient and generalizes well, beating the strongest baselines by up to 2-7× on dexterous, precise, real-world bimanual manipulation tasks both in and out of distribution, all while running faster than π0.5. Our project website: this https URL
Learn More
Project Page: https://flex-pi.github.io/
ArXiV: https://flex-pi.github.io/
Transcript
Transcript edited and prepared with automated transcription and checked by GPT 6. There may still be errors.
Jiafei Duan (00:04.231)
Okay, sure, sure, sure. Yeah, hello everyone. Welcome to a new episode of RoboPapers. We passed our hundredth paper mark and this is, I think, 102 or 103. And and today we have like two of the returning speakers from the previous episodes, Jesse and Ge Yan, to come on and really share about their new work, Flex-π. So maybe do you guys want to Reintroduce yourself a bit more and then I think for some of the audience who never heard the previous podcast before. Yeah.
Ge Yan (00:38.151)
Yeah, yeah, hi Jiafei, and thanks for having us. So I’m right now like a second year PhD student who is going to enter my third year at the University of Washington. And my PhD advisor is Dieter Fox. And now I’m working also closely with TRI and Jesse. Yeah, Jesse, it’s yours.
Jesse Zhang (01:01.599)
Yeah, I’m a postdoc at UW and I’m advised by Dieter Fox, and Abhishek. And so I started this Ge and I worked on this project for the past how long has it been? Maybe almost a year, yeah, it’s been a while. And it’s changed quite a bit since the initial version was definitely not a WAM. so it’s been very interesting how this has turned out.
Ge Yan (01:18.833)
Yeah, it’s been a long time.
Ge Yan (01:26.567)
Yeah, great.
Jiafei Duan (01:27.145)
Yeah, it’s it’s been it’s been a quite a while. I mean, I remember I was still there when all the people collecting data.
Ge Yan (01:35.464)
Yeah.
Jesse Zhang (01:35.623)
Yeah, yeah. I mean, even like Phil presented on on ideas that led to this and they were very different than what it is now.
Jiafei Duan (01:42.677)
Yeah, yeah. Yeah. I mean I mean for audience like just context, we are from the same group, so I I know exactly what this is about. But yeah, it definitely changed. But yeah, we hope to hear from Ge exactly what Flex-π is all about. So yeah, Ge, you want to take over.
Ge Yan (01:49.629)
Yeah.
Ge Yan (02:00.219)
Okay, thank you. Okay, hello everyone. Today we’re going to introduce Flex-π, a multi-stream world-action model with compute flexibility. I think we framed it as a policy that can do one checkpoint, any observed inputs, any generated features, and the speed accuracy operating point you pick and deployment. And this work is done by me, Jinghao, Yuzhi, Lori, Minwen, Jesse, and Dieter at the University of Washington. So I think recently we’ve seen a lot of work on on WAMs.
I think it starts from the trend of very good video generation model, and people start to leverage those video priors from those very large video models and use it for action prediction. And we’ve seen a lot of progress, including like Fast-WAM, LingBot-VA, DINO most recent DINOTube paper work and and a lot of others. And I think we know that the WAMs are great, but they usually only predict RGB latent and that is specifically used to for reconstruction. And the previous work like DINO or DreamZero, so they really shows us that the world modeling objective itself enables better action prediction.
And I think it is quite intuitive right now, like with with so many results show its benefits. So I think in our work we kind of want to ask a question like why not predict additional and that is especially manipulation-centric visual signals. So we really want to leverage those other signals because we want to further push the world modeling objective strengths, and we want to do some very high precision and dexterous tasks, and we want to achieve better demo efficiency and generalization. So the right is a a multi-view.
Ge Yan (04:02.367)
RGB video shown in a YAM setup that a model do this Soft-Bag Zipping task. And so right now you can see it’s purely RGB right? And so for Flex-π we focus on like two additional signals. So for better Demo efficiency and the generalization. So one is DINO. So we know the DINO had provides very good object-centric semantics that can be very good for generalization, especially in those cloud cloud environments. And also the 3D point. I think this 3D information really provides a lot of demo efficiency, lots of model to achieve sometimes even have high performance with much less data.
As as we know, there’s no like free lunch, right? as we introduce like additional signals, it can like possibly like always introduce like additional cost. So that that really comes to one problem, because the most WAMs the the inference mode is decided at training time. So like what kind of inputs the model takes and what kind of features the model needs to generate. And we know that predicting the future makes us better, but actually w when you generate those at the inference time make it pretty slow. And it’s usually we see a lot of model that joint and generate those features, it’s very hard to do real time.
And especially with our additional visual signals, that makes even makes it even harder. So we we kind of like want to do like one model that can do both. they can do like action-only prediction, and then you can also predict the futures as other WAMs. And it can be decided at the deployment time.
Ge Yan (06:00.741)
And before introducing our exact architecture, we want to share that because we build on top of the Wan video model backbone. So this the bottom video shows after training on internet-scale videos, the Wan model, this backbone has a very good physical prior. You can see it can generate this looks reasonable like videos. Although there you can see there are still some artifacts, but it really sh it shows the model can learn like the r what waterwheel looks like a reasonable behavior, even though it’s not exactly the the right. And and one thing about the one backbone is like the VAE part. So VAE can be regarded as an encoder.
So for a VLA you can have like any those image language encoder, and for these one models, we use something called a Wan 2.2 VAE. …
This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit robopapers.substack.comEp#105: Beyond Imitation: Self-Improving Robot Policies via Off-Policy Q-Planning
Imitation learning, especially with interventions, has driven so much recent robotics progress. However, improving a policy via targeted interventions until it reaches a useful and deployable success rate is a …
Notas do episódio
Imitation learning, especially with interventions, has driven so much recent robotics progress. However, improving a policy via targeted interventions until it reaches a useful and deployable success rate is a time and labor intensive process. Instead, wouldn’t it be great if policies could improve on their own?
That’s what Varun Giridhar and Animesh Garg join us to talk about. In Q-Planning, they start with a large policy like pi-0.5, and add a Q-function estimator to predict value instead of just actions, then use both successful and failed rollouts to update this Q-function online, then use it to guide sampling and trajectory selection. With just a few rollouts they can dramatically improve policy performance online. This provides a way to do really difficult tasks like inserting a credit card into a wallet, increasing success rate from 25% to 80% in just a few iterations.
Learn more in Episode 105 of RoboPapers, hosted by Michael Cho, Chris Paxton, and Ruijie He!
Abstract
Behaviour Cloning (BC) has driven remarkable progress in robot manipulation, yet it is fundamentally limited by its inability to self-improve: a policy that fails cannot learn from that failure without additional human demonstrations. Reinforcement Learning fine-tuning offers a path to self-improvement but has proven difficult to scale to the multi-billion-parameter models underpinning modern robot policies. We propose Q-Planning, which equips a large visuomotor BC policy with a small off-policy Q-function. Because a Q-function estimates value rather than imitates actions, it can be trained on the same successful demonstrations as the BC policy and later absorb both successful and failed deployment rollouts, an asymmetry BC does not have. We exploit this asymmetry to enable value-guided action selection at inference (a single-step QQ-weighted average over BC draws) and online self-improvement that fine-tunes only the Q-function, leaving the BC weights untouched. On LIBERO and bimanual RoboTwin, ten iterations of self-improvement lift every benchmark score we tested (LIBERO-10 93 → 99%, RoboTwin 83.8 → 91.4%) and shorten successful episodes on the near-ceiling suites (LIBERO-Object, LIBERO-Goal). On two contact-rich bimanual real-robot tasks, the same loop (BC frozen, no human intervention) improves purely from its own deployment rollouts: stack-cups 40 → 90% and insert-wallet 25 → 80% in five iterations, whereas SFT on successful rollouts alone stalls at 55% and 30%. Under an identical online budget Q-Planning is the only method, among Best-of-N, filtered SFT, IBRL, DSRL, and DAWR, that improves stably from failures without training an auxiliary actor.
Learn More
Project page: https://q-planning.github.io/
ArXiV: https://arxiv.org/abs/2608.21204
This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit robopapers.substack.comEp#104: Light-Loco-Parkour: Versatile Perceptive Whole-Body Locomotion via Multi-Skill Distillation
One of the key advantages of legged robots like humanoids should be how effectively they can move across a wide variety of terrain types to accomplish their task. But Light-Loco-Parkour from the team at Light …
Notas do episódio
One of the key advantages of legged robots like humanoids should be how effectively they can move across a wide variety of terrain types to accomplish their task. But Light-Loco-Parkour from the team at Light Origins aims to change that: using only onboard sensing, they show a policy which can decide when to walk, vault, climb, or otherwise traverse as it moves through a complex environment. Unlike many others, it uses sparse seeds instead of relying on a large motion corpus, learning when to use its skills to move around without specific sub-task labels. Xiaodao Chen and Yuntao Ma join us to go into the details.
Watch Episode 104 of RoboPapers now, with Michael Cho and Chris Paxton, to learn more!
Abstract
Existing humanoid whole-body control systems still fall short of the way humans move through cluttered terrain: they either track expressive whole-body references without terrain generalization, or react to terrain online while leaving the arms, torso, and knees largely unused. We present Light-Loco-Parkour (LightLP), an end-to-end perceptive whole-body locomotion system that closes this gap with a single deployable policy. Conditioned only on onboard depth and a velocity command, the policy decides when to walk, balance, climb, step down, or vault, with no reference input, skill label, hand-coded gate, or runtime motion graph. Compared with prior humanoid systems, LightLP makes three contributions. First, it introduces a whole-body perceptive-control pipeline that extends an RL-trained, velocity-tracking locomotion policy with parkour skills learned from object-interacting motions, so the same policy tracks velocity in open terrain, executes whole-body traversal at obstacles, and resumes locomotion afterward. Second, it acquires terrain-conditioned skills from sparse seeds by expanding a single motion into dynamically feasible, terrain-paired references across obstacle geometry, rather than relying on a large motion corpus. Third, it learns autonomous skill transitions from reward, letting the policy decide when and which whole-body skill to invoke from depth and command alone, with no one-hot skill label, hand-coded state machine, or runtime motion generator.
Simulation and real-world experiments show high success across both benchmarked terrains and unseen obstacle variations, and the same policy transfers zero-shot to indoor and outdoor hardware experiments. These results demonstrate autonomous perceptive whole-body locomotion on a humanoid in outdoor settings, using only onboard sensing and a single deployable policy.
Learn More
Project page: https://light-loco-parkour.github.io/
Paper: https://light-loco-parkour.github.io/paper.pdf
This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit robopapers.substack.comEp#103: Freeform Preference Learning for Robotic Manipulation
Specifying reward functions for robots is one of the hardest things about reinforcement learning. Robot rewards often need to be very detailed; metrics like progress can be ill-defined and hard to estimate …
Notas do episódio
Specifying reward functions for robots is one of the hardest things about reinforcement learning. Robot rewards often need to be very detailed; metrics like progress can be ill-defined and hard to estimate. This leads to most robot learning defaulting to sparse rewards or simple preference learning. But naive preference learning (having human annotators choose one trajectory over another) is an easy solution, but obscures a lot of the signal in complex tasks and can make learning a lot less efficient.
Marcel Torné, Anubha Mahajan, and Abhijnya Bhat join us to talk about their solution: freeform preference learning, which lets annotators define natural-language axes to compare trajectories over. This improves real-world performance on long-horizon manipulation tasks over sparse rewards and simple binary preference learning.
Watch Epsiode 103 of RoboPapers, with Michael Cho and Jiafei Duan, today to learn more!
Abstract
Reward design remains a central bottleneck for autonomous robot policy improvement, especially in long-horizon manipulation tasks where sparse success labels provide too little signal and binary preferences collapse many competing notions of quality into one ambiguous signal. We introduce Freeform Preference Learning (FPL), a method for learning robot policies from freeform human preferences. Rather than asking annotators which of two trajectories is better overall, FPL lets them define natural-language preference axes, such as speed, safety, quality of placement, or carefulness, and provide pairwise preferences along each axis. These annotations are used to learn a language-conditioned reward model that maps a trajectory and preference label to an axis-specific reward. We use this model to train a reward-conditioned policy that optimizes across the multiple human-specified dimensions. Across four real-world and two simulated long-horizon manipulation tasks, FPL improves over sparse-reward and binary-preference methods by 38 percentage points. Beyond improved performance, FPL learns dense progress signals without explicit subtask segmentation, shows compositionality of behavior not present in the data, and allows users to steer the policy towards different behaviors at test time without retraining. Blog post with videos available at this https URL
Learn More
Project page: https://freeform-pl.github.io/fpl.website/
ArXiV: https://arxiv.org/abs/2606.32027
This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit robopapers.substack.comEp#102: EgoSuite-Open100K from Lightwheel
Robotics has a data problem, and egocentric data is a compelling way to solve it because egocentric data is (1) scalable — it’s cheap to collect — and (2) easily captures the breadth and diversity of real human …
Notas do episódio
Robotics has a data problem, and egocentric data is a compelling way to solve it because egocentric data is (1) scalable — it’s cheap to collect — and (2) easily captures the breadth and diversity of real human tasks, while (3) being true to real physics. Lightwheel recently released a hundred thousand hours of high-quality egocentric video data, spread across a wide range of human tasks from cleaning and tidying to assembly and construction to retail tasks.
Here to tell us about this tremendous effort is Steve Xie, CEO and co-founder of Lightwheel. The data is also available on HuggingFace right now.
Watch Episode 102 of RoboPapers, with Michael Cho and Jiafei Duan, to learn more!
Learn More
Project page: https://egosuite100k.lightwheel.ai/
Blog post: https://huggingface.co/blog/LightwheelAI/egosuite-open100k
Data on HuggingFace: https://huggingface.co/collections/LightwheelAI/egosuite-open100k
This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit robopapers.substack.comEp#101: Coupled Local and Global World Models for Efficient First Order RL
Reinforcement learning has unlocked tremendous gains in broader AI and humanoid robot whole body control, but still has yet to deliver strong results for robot manipulation due to the difficulty of capturing …
Notas do episódio
Reinforcement learning has unlocked tremendous gains in broader AI and humanoid robot whole body control, but still has yet to deliver strong results for robot manipulation due to the difficulty of capturing tasks in an accurate simulation; one way to fix this is to create complex environments with learned world models. But learning fully within world models is often intractable as powerful world models are computationally too expensive.
Instead, Joseph Amigo and Rooholla Khorrambahkt propose to break the problem down: into a large-scale world model that can generate forward trajectories, and a lightweight, low-dimensional latent-space model which can approximate local dynamics of a problem, without needing to back-propagate through a heavy global model. What this means: you can do reinforcement learning in this coupled local-global world model, and learn contact rich skills for a robot.
To learn more, watch Episode #101 of RoboPapers, with Michael Cho and Jiafei Duan!
Abstract
World models offer a promising avenue for capturing complex environment dynamics where simulators face challenges. However, large-scale world models required for complex real-world settings are computationally expensive to adopt in popular RL approaches. We introduce a novel first-order RL method that enables policy training via a decoupled first-order gradient (FoG): a large-scale world model generates accurate forward trajectories while a lightweight latent-space surrogate approximates its local dynamics for efficient gradient computation. This coupled local-and-global formulation allows high-fidelity forward dynamics alongside the computationally efficient differentiation needed for model-based RL. Across a range of real-world robotic tasks we demonstrate tractable RL and zero-shot deployment, with significantly better sample efficiency than PPO on a canonical real-world Push-T benchmark and similar gains on more complex ego-centric manipulation and grasping.
Learn More
Project page: https://coupled-global-local-wm-rl.pages.dev/
ArXiV: https://arxiv.org/abs/2602.06219
This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit robopapers.substack.comEp#100: 100 Episodes of Embodied AI
Embodied AI is exploding, and it’s been amazing to learn about it with you all.We have now recorded over a hundred discussions with the best researchers in the world on embodied artificial intelligence and …
Notas do episódio
Embodied AI is exploding, and it’s been amazing to learn about it with you all.
We have now recorded over a hundred discussions with the best researchers in the world on embodied artificial intelligence and robotics, covering topics like scaling robot learning, humanoid whole body control, tactile sensors, sim-to-real, and so much more.
For our hundredth episode, we brought on our newest co-host, Ruijie He, to discuss what we’ve seen and learned in the first hundred episodes. Which were our most-watched episodes? Which institution shows up the most? Who was our most frequent guest? What did we learn?
Find out on Episode 100 of RoboPapers, with Michael Cho, Chris Paxton, Jiafei Duan, and Ruijie He!
Learn More
This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit robopapers.substack.comEp#99: DYNA-2: A 1 Million Hour Scaling Law for World-Action Models
Excitement about world-action models and robot learning has never been higher — they promise a way to use human egocentric data to train massive robotics models which can provide the “GPT” moment for robotics …
Notas do episódio
Excitement about world-action models and robot learning has never been higher — they promise a way to use human egocentric data to train massive robotics models which can provide the “GPT” moment for robotics and unlock general-purpose embodied intelligence. And yet there’s been little concrete demonstration of scaling in robot learning. Dyna Robotics aims to change that, with an in-depth look at how scaling works as they approach 1 million hours of training data.
Jason Ma, Tianyu Li, Anh-Quan Pham, and Chet Bhateja joined us to tell us more. They show that as the amount of data they use in pretraining increased, they saw predictable, statistically significant gains on accuracy metrics on held-out data (data not seen during training). They go on to talk about what they learned, and show how this can be applied to many different problems.
Watch Episode #99 of RoboPapers, with Michael Cho, Chris Paxton, and Jiafei Duan today to learn more!
Learn More
This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit robopapers.substack.comEp#97: Cross-Embodiment Robot Manipulation via a Unified Hand Action Space
Representing actions for robot hands is actually very challenging. Robot hands have different sizes and shapes, and different degrees of freedom. Representing their actions purely as joint states — as we …
Notas do episódio
Representing actions for robot hands is actually very challenging. Robot hands have different sizes and shapes, and different degrees of freedom. Representing their actions purely as joint states — as we naively would — leads to less transfer across the wide variety of available hands. Instead, could we design a universal hand action space to allow robots to learn cross-embodiment dexterous manipulation more easily?
Luis Felipe Casas and Yu Xiang come on to tell us about UHAS: the Unified Hand Action Space, a sphere-based action representation which enables cross-ebmodiment learning of dexterous manipulation skills.
Watch Episode #97 of RoboPapers, with Chris Paxton and Jiafei Duan, today to learn more!
Abstract
Robot manipulation policies are typically tied to specific robotic hand embodiments, limiting the transfer of learned behaviors across platforms with different kinematic structures. In this work, we propose the Unified Hand Action Space (UHAS), a sphere-based unified action representation for cross-embodiment dexterous manipulation. UHAS represents robotic hand actions as geometric deformations of a canonical sphere and uses a Cascade Inverse Kinematics (CIK) algorithm to map the shared representation to embodiment-specific joint configurations. Using reinforcement learning, we train dexterous manipulation policies directly in the proposed action space for in-hand cube reorientation tasks. We evaluate our method in both simulation and real-world experiments across multiple robotic hands, including the Allegro Hand, LEAP Hand, Shadow Hand, and MANO Human Hand. Experimental results demonstrate effective dexterous manipulation, zero-shot transfer to unseen hands, rapid finetuning across embodiments, and successful real-world deployment. Our experiments show that the proposed UHAS representation enables stable dexterous control and cross-embodiment policy transfer across robotic hands.
Learn More
ArXiV: https://arxiv.org/abs/2607.03570
This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit robopapers.substack.comEp#96: EgoHumanoid: Unlocking In-the-Wild Loco-Manipulation with Robot-Free Egocentric Demonstration
How can roboticists scale human manipulation data in the wild? EgoHumanoid builds a comprehensive pipeline which unlocks mobile manipulation training for robots — meaning that you can walk around and collect …
Notas do episódio
How can roboticists scale human manipulation data in the wild? EgoHumanoid builds a comprehensive pipeline which unlocks mobile manipulation training for robots — meaning that you can walk around and collect data for your robot anywhere you go. There are two key components to their recipe: view alignment, to reduce mismatch between human and robot camera views, and action alignment, to make sure human actions can be converted into robot actions. This makes it easier than ever to train robot policies and deploy them in previously-unseen environments. Modi Shi joins us to tell us the details.
Watch Episode #96 of RoboPapers with Michael Cho and Jiafei Duan today!
Abstract
Human demonstrations offer rich environmental diversity and scale naturally, making them an appealing alternative to robot teleoperation. While this paradigm has advanced robot-arm manipulation, its potential for the more challenging, data-hungry problem of humanoid loco-manipulation remains largely unexplored. We present EgoHumanoid, the first framework to co-train a vision-language-action policy using abundant egocentric human demonstrations together with a limited amount of robot data, enabling humanoids to perform loco-manipulation across diverse real-world environments. To bridge the embodiment gap between humans and robots, including discrepancies in physical morphology and viewpoint, we introduce a systematic alignment pipeline spanning from hardware design to data processing. A portable system for scalable human data collection is developed, and we establish practical collection protocols to improve transferability. At the core of our human-to-humanoid alignment pipeline lies two key components. The view alignment reduces visual domain discrepancies caused by camera height and perspective variation. The action alignment maps human motions into a unified, kinematically feasible action space for humanoid control. Extensive real-world experiments demonstrate that incorporating robot-free egocentric data significantly outperforms robot-only baselines by 51\%, particularly in unseen environments. Our analysis further reveals which behaviors transfer effectively and the potential for scaling human data.
Learn More
Project page: https://opendrivelab.com/EgoHumanoid/
ArXiV: https://arxiv.org/abs/2602.10106
Github: https://github.com/OpenDriveLab/EgoHumanoid
This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit robopapers.substack.comEp#95: Action-to-Action Flow Matching
Diffusion Policy was one of the big breakthroughs that has enabled an explosion in real-world robot learning. However, it’s always had a weakness, which is that it works by computing a final action trajectory …
Notas do episódio
Diffusion Policy was one of the big breakthroughs that has enabled an explosion in real-world robot learning. However, it’s always had a weakness, which is that it works by computing a final action trajectory from random noise, which leads to high latency when predicting a final action sequence.
Instead, why not initialize the search based on previous actions? This allows for incredibly fast policy inference and in many cases improved generalization, generating high-quality predictions with sub-ms latency. Jindou Jia and Jianfei Yang join us to explain.
Learn more on Episode 95 of RoboPapers, with Michael Cho and Chris Paxton!
Abstract
Diffusion-based policies have recently achieved remarkable success in robotics by formulating action prediction as a conditional denoising process. However, the standard practice of sampling from random Gaussian noise often requires multiple iterative steps to produce clean actions, leading to high inference latency that incurs a major bottleneck for real-time control. In this paper, we challenge the necessity of uninformed noise sampling and propose Action-to-Action flow matching (A2A), a novel policy paradigm that shifts from random sampling to initialization informed by the previous action. Unlike existing methods that treat proprioceptive action feedback as static conditions, A2A leverages historical proprioceptive sequences, embedding them into a high-dimensional latent space as the starting point for action generation. This design bypasses costly iterative denoising while effectively capturing the robot's physical dynamics and temporal continuity. Extensive experiments demonstrate that A2A exhibits high training efficiency, fast inference speed, and improved generalization. Notably, A2A enables high-quality action generation in as few as a single inference step (0.56 ms latency), and exhibits superior robustness to visual perturbations and enhanced generalization to unseen configurations. Lastly, we also extend A2A to video generation, demonstrating its broader versatility in temporal modeling.
Learn More
Project Page: https://jingliangli.com/A2A_Flow_Matching/
ArXiV: https://arxiv.org/abs/2602.07322
Github: https://github.com/JIAjindou/A2A_Flow_Matching
This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit robopapers.substack.comEp#94: Learning a Thousand Tasks in a Day
For general-purpose robots to be useful, we must be able to teach them new tasks quickly and repurpose them to new roles. But most techniques for teaching robots — even from human demonstrations — take a great …
Notas do episódio
For general-purpose robots to be useful, we must be able to teach them new tasks quickly and repurpose them to new roles. But most techniques for teaching robots — even from human demonstrations — take a great deal of new data. But it turns out that it is possible, in part through decomposing tasks into component subtasks, that it’s possible to learn a robot skills quickly.
Kamil Dreczkowski, Pietro Vitiello and Edward Johns join us to talk about their extensive study on how to teach robots new skills efficiently, while also generalizing to novel object instances, using a combination of task decomposition and retrieval, work that was published in Science Robotics.
Watch episode #94 of RoboPapers now, with Michael Cho and Jiafei Duan, to learn more!
Abstract
Humans are remarkably efficient at learning tasks from demonstrations, but today's imitation learning methods for robot manipulation often require hundreds or thousands of demonstrations per task. We investigate two fundamental priors for improving learning efficiency: decomposing manipulation trajectories into sequential alignment and interaction phases, and retrieval-based generalisation. Through 3,450 real-world rollouts, we systematically study this decomposition. We compare different design choices for the alignment and interaction phases, and examine generalisation and scaling trends relative to today's dominant paradigm of behavioural cloning with a single-phase monolithic policy. In the few-demonstrations-per-task regime (<10 demonstrations), decomposition achieves an order of magnitude improvement in data efficiency over single-phase learning, with retrieval consistently outperforming behavioural cloning for both alignment and interaction. Building on these insights, we develop Multi-Task Trajectory Transfer (MT3), an imitation learning method based on decomposition and retrieval. MT3 learns everyday manipulation tasks from as little as a single demonstration each, whilst also generalising to novel object instances. This efficiency enables us to teach a robot 1,000 distinct everyday tasks in under 24 hours of human demonstrator time. Through 2,200 additional real-world rollouts, we reveal MT3's capabilities and limitations across different task families. Videos of our experiments can be found on at this https URL.
Learn More
Project Page: https://www.robot-learning.uk/learning-1000-tasks
ArXiV: https://arxiv.org/abs/2511.10110
Science Paper: https://www.science.org/doi/abs/10.1126/scirobotics.adv7594
Github: https://github.com/kamil-dreczkowski/learning_thousand_tasks
This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit robopapers.substack.comEp#93: CHORUS: Decentralized Multi-Embodiment Collaboration with One VLA Policy
Many tasks cannot be completed by one robot alone. But coordinating multiple robots performing complex manipulation tasks is very difficult. Many solutions rely on complicated centralized control, which tends …
Notas do episódio
Many tasks cannot be completed by one robot alone. But coordinating multiple robots performing complex manipulation tasks is very difficult. Many solutions rely on complicated centralized control, which tends towards bespoke methods that do not scale well with team size.
Enter CHORUS by Ria Doshi, Jeanette Bohg, and team. They train a single VLA to control multiple, diverse robots, given only local observations and identifying information. This allows robots to collaborate on tasks like using a tape measure, handing over a book, or lifting a laundry basket.
To learn more, watch Episode #93 of Robopapers now, with Michael Cho, Jiafei Duan, and Chris Paxton!
Abstract
Multi-robot collaboration allows robots to efficiently take on a wide range of tasks, from moving a couch through a doorway to assembling structures on a construction site. However, achieving such coordination in mobile multi-robot settings remains challenging: centralized methods conditioned on the combined observations of a team scale poorly with team size, and decentralized methods that train one policy per robot often require explicit alignment procedures or information sharing at inference time to overcome partial observability. Our key insight is that the visuomotor priors of pretrained vision-language-action (VLA) models should enable reactive, decentralized collaboration from each robot's local observations alone, without these inference-time assumptions. We propose CHORUS, a framework that adapts a single VLA backbone to control diverse, multi-robot teams. At inference time, each robot runs an independent copy of CHORUS, conditioned only on its own observations and a robot-identifying prompt. In real-world experiments including mobile tape measurement, library book handovers, and laundry basket lifting, CHORUS achieves a 64% point improvement over decentralized, from-scratch models, improves reactivity to teammate behavior by 40% points, and outperforms centralized baselines. Together, these results show that a shared VLA backbone is capable of achieving decentralized multi-robot collaboration, without per-robot policies or inter-robot communication at inference.
Learn More
Project Page: https://chorus-model.github.io/
ArXiV: https://arxiv.org/abs/2606.12352
This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit robopapers.substack.comEp#92: Human Universal Grasping
How can we learn robot grasping from egocentric human video alone? General-purpose dexterous manipulation learning will require a lot of data, and yet robot data is hard to find at scale. Better leveraging …
Notas do episódio
How can we learn robot grasping from egocentric human video alone? General-purpose dexterous manipulation learning will require a lot of data, and yet robot data is hard to find at scale. Better leveraging human data, then, will be key to general-purpose dexterous manipulation.
Kevin Wu, Irmak Guzey, and Dandan Shan join us to talk about HUG: Human Unversal Grasping. They collect a dataset of 1M frames (27.8 hours) of egocentric human grasping data, and use it to train a new flow-matching algorithm to predict hand pose. They can then predict human hand poses and retarget them to robot hands, allowing for a massive improvement on a wide variety of zero-shot robot grasping tasks in everyday scenes.
To learn more, watch Episode 92 of RoboPapers with Michael Cho and Chris Paxton today!
Abstract
Humans can grasp objects effortlessly, whereas multi-fingered robots are far from this level of generality. We argue that the most natural source of robot grasping data is from humans, who pick up thousands of objects every day. We present HUG, a flow-matching model that generates diverse human grasps for any user-specified object in a single RGB-D image captured from a stereo camera. Using smart glasses, we first collect 1M-HUGs, an egocentric dataset of human grasps spanning 1M frames (27.8 hrs) and 6,707 object instances across 41 buildings. Next, to model the distribution of natural human grasps, our novel flow-matching model fuses RGB and depth observations to output a grasp parameterized by wrist translation, wrist rotation, and MANO hand pose. Predicted grasps can be retargeted to various robot hands, enabling zero-shot grasping in everyday scenes. To standardize evaluation, we build a new simulated benchmark, HUG-Bench, of 90 unseen objects from five geometric categories and various sizes, with metric-scale 3D meshes. We evaluate HUG in the real world on the 30-object test set of HUG-Bench across multiple stereo cameras, robot embodiments, and household environments. HUG outperforms the state-of-the-art grasping baselines by +23% and +34% on our challenging object set. Code, data, benchmark, checkpoints, and an interactive demo are released on our website: this https URL
Learn More
Project Page: https://grasping.io/
ArXiV: https://arxiv.org/abs/2606.17054
Github: https://github.com/KevinyWu/hug
This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit robopapers.substack.comEp#91: RoboMME: Benchmarking and Understanding Memory for Robotic Generalist Policies
Memory is one of the most important problems in robotics. Long horizon memory is key for a variety of robot manipulation problems. However, there exist no good benchmarks for understanding progress in how well …
Notas do episódio
Memory is one of the most important problems in robotics. Long horizon memory is key for a variety of robot manipulation problems. However, there exist no good benchmarks for understanding progress in how well generalist robot policies can understand language.
Yinpei Dai and Yuejiang Liu made RoboMME as a solution: it’s a large benchmark which shows 16 different robot tasks, like counting objects or mastering timing. They show 14 different memory-augmented generalist policies across these different benchmarks. It’s an incredibly thorough and interesting result, aimed at driving forward this core robotic capability.
To learn more, watch Episode 91 of RoboPapers with Michael Cho and Chris Paxton!
Abstract
Memory is critical for long-horizon and history-dependent robotic manipulation. Such tasks often involve counting repeated actions or manipulating objects that become temporarily occluded. Recent vision-language-action (VLA) models have begun to incorporate memory mechanisms; however, their evaluations remain confined to narrow, non-standardized settings. This limits their systematic understanding, comparison, and progress measurement. To address these challenges, we introduce RoboMME: a large-scale standardized benchmark for evaluating and advancing VLA models in long-horizon, history-dependent scenarios. Our benchmark comprises 16 manipulation tasks constructed under a carefully designed taxonomy that evaluates temporal, spatial, object, and procedural memory. We further develop a suite of 14 memory-augmented VLA variants built on the π0.5 backbone to systematically explore different memory representations across multiple integration strategies. Experimental results show that the effectiveness of memory representations is highly task-dependent, with each design offering distinct advantages and limitations across different tasks.
This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit robopapers.substack.comEp#90: From Capable Controllers to Deployable Humanoid Systems
We want humanoid robots to be able to perform complex, long-horizon tasks in the real world — putting away the groceries or cleaning a room, for example. This requires diverse loco-manipulation skills, which …
Notas do episódio
We want humanoid robots to be able to perform complex, long-horizon tasks in the real world — putting away the groceries or cleaning a room, for example. This requires diverse loco-manipulation skills, which can be easily parameterized to handle object affordances and interact with the world around it safely.
In HANDOFF, Lizhi Yang proposes a 10-D learned whole body controller, which can be converted to whole body actions, and which can be used by a VLM-driven agentic planner to perform complex, multi-step manipulation actions in the real world.
We then discuss how it’s possible to deploy such controllers in the real world, how to make them safe around people via controlled barrier functions and safety functions. This enables humanoids which can move safely through dynamic, crowded environments in the real world.
Learn more in Episode 90 of RoboPapers, with Michael Cho and Chris Paxton.
Abstract
For a humanoid robot to be deployed in the real world, the choice of command space (i.e., the interface between task planning and whole-body control) is crucial. Existing whole-body controllers typically demand dense kinematic or spatial references that planners struggle to synthesize from task semantics. We instead propose a compact, explicit interface that is intuitive, general, modular, and expressive enough for diverse loco-manipulation skills. To this end, we introduce HANDOFF, a single humanoid whole-body controller that follows this interface and is distilled via multi-teacher KL distillation under a context-conditioned gating scheme into a mixture-of-experts student from three complementary specialists: whole-body motion tracking with safety-filtered data, locomotion, and fall-recovery. On the Unitree G1, HANDOFF matches state-of-the-art velocity tracking and offers one of the largest robust manipulation workspaces. We further demonstrate hardware feasibility through multiple natural-language-driven task roll-outs, powered by a VLM-driven agentic planner with no task-specific data or controller fine-tuning.
Learn More
Project page for HANDOFF: https://lzyang2000.github.io/HANDOFF/
Code: https://github.com/lzyang2000/HANDOFF
ArXiV: https://arxiv.org/abs/2606.06493
Safe-SAGE on ArXiV: https://arxiv.org/abs/2603.05497
SHIELD on ArXiV: https://arxiv.org/abs/2505.11494
This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit robopapers.substack.comEp#89: Contact Grounded Policy
Contact-rich manipulation is still very challenging for robotics. Problems like opening a jar, or in-hand reorientation of an object, require making repeated contact with different parts of a robot’s hand, and …
Notas do episódio
Contact-rich manipulation is still very challenging for robotics. Problems like opening a jar, or in-hand reorientation of an object, require making repeated contact with different parts of a robot’s hand, and this is hard to do with pure vision. Instead, research is moving towards using tactile sensors in combination with visual policies. But what’s the best way to learn how to handle multi-point contact?
Zhengtong Xu and Yeping Wang tell us about their new work Contact-Grounded Policy (CGP). CGP predicts future robot state and tactile feedback, and predicts this into actions for a compliant robot controller so that a four- or five-finger robot hand can perform complex tasks involving precise manipulation, delicate grasping, and tool use.
To learn more, watch Episode #89 of RoboPapers, with Chris Paxton and Jiafei Duan.
Abstract
Contact-rich dexterous manipulation with multi-finger hands remains an open challenge in robotics because task success depends on multi-point contacts that continuously evolve and are highly sensitive to object geometry, frictional transitions, and slip. Recently, tactile-informed manipulation policies have shown promise. However, most use tactile signals as additional observations rather than modeling contact state or how their action outputs interact with low-level controller dynamics. We present Contact-Grounded Policy (CGP), a visuotactile policy that grounds multi-point contacts by predicting coupled trajectories of actual robot state and tactile feedback, and using a learned contact-consistency mapping to convert these predictions into executable target robot states for a compliance controller. CGP consists of two components: (i) a conditional diffusion model that forecasts future robot state and tactile feedback in a compressed latent space, and (ii) a learned contact-consistency mapping that converts the predicted robot state-tactile pair into executable targets for a compliance controller, enabling it to realize the intended contacts. We evaluate CGP using a physical four-finger Allegro V5 hand with Digit360 fingertip tactile sensors, and a simulated five-finger Tesollo DG-5F hand with dense whole-hand tactile arrays. Across a range of dexterous tasks including in-hand manipulation, delicate grasping, and tool use, CGP outperforms visuomotor and visuotactile diffusion-policy baselines.
Learn More
Project page: https://contact-grounded-policy.github.io/
ArXiV: https://arxiv.org/abs/2603.05687
This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit robopapers.substack.comEp#88: DexSkin: High-Coverage Conformable Robotic Skin for Learning Contact-Rich Manipulation
Human skin plays an important role in how we interact with the world and robustly manipulate objects. It’s not just important when we can’t see things with out eyes, but when we want to pick up something heavy …
Notas do episódio
Human skin plays an important role in how we interact with the world and robustly manipulate objects. It’s not just important when we can’t see things with out eyes, but when we want to pick up something heavy, or apply a very specific amount of force. So, it makes sense to want to give robots skin.
Enter DexSkin: a soft, deformable electronic skin which can be applied across different surfaces and used to cover robot hands or fingers. Suzannah Wistreich and Baiyu Shi talk to us about their work building DexSkin, showing how it’s useful for policy learning, including online reinforcement learning, and how it' can be calibrated and policies transferred across sensors. They also open sourced their code and methods for building the sensors.
To learn more, watch Episode #88 of RoboPapers now, hosted by Chris Paxton and Jiafei Duan!
Abstract
Human skin provides a rich tactile sensing stream, localizing intentional and unintentional contact events over a large and contoured region. Replicating these tactile sensing capabilities for dexterous robotic manipulation systems remains a longstanding challenge. In this work, we take a step towards this goal by introducing DexSkin. DexSkin is a soft, conformable capacitive electronic skin that enables sensitive, localized, and calibratable tactile sensing, and can be tailored to varying geometries. We demonstrate its efficacy for learning downstream robotic manipulation by sensorizing a pair of parallel jaw gripper fingers, providing tactile coverage across almost the entire finger surfaces. We empirically evaluate DexSkin's capabilities in learning challenging manipulation tasks that require sensing coverage across the entire surface of the fingers, such as reorienting objects in hand and wrapping elastic bands around boxes, in a learning-from-demonstration framework. We then show that, critically for data-driven approaches, DexSkin can be calibrated to enable model transfer across sensor instances, and demonstrate its applicability to online reinforcement learning on real robots. Our results highlight DexSkin's suitability and practicality for learning real-world, contact-rich manipulation. Please see our project webpage for videos and visualizations: this https URL.
Learn More
ArXiV: https://arxiv.org/abs/2509.18830
Project Page: https://dex-skin.github.io/
Github: https://github.com/sdwistreich/dexskin
Datasets: https://huggingface.co/datasets/swistreich/dexskin
This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit robopapers.substack.comEp#87: MolmoAct 2: An open foundation for robots that work in the real world
There are few truly open models in the world, including both weights and data. However, these models are crucial for research and development of new systems — they help us learn which data is important and help …
Notas do episódio
There are few truly open models in the world, including both weights and data. However, these models are crucial for research and development of new systems — they help us learn which data is important and help develop new capabilities for deploying robots in the real world.
MolmoAct2 provides a foundation for open research into robotics. It is associated with its own open dataset, an open-data action tokenizer, and a reasoning variant which predicts depth tokens. And people have actually been using it across the community, running experiments in their own labs or homes.
Haoquan Fang and Jiafei Duan tell us more. Watch Episode 87 of RoboPapers, with Michael Cho and Chris Paxton, now!
Abstract
Vision-Language-Action (VLA) models aim to provide a single generalist controller for robots, but today’s systems fall short for real-world deployment. Frontier models are closed; open-weight alternatives are tied to expensive hardware; reasoning-augmented policies pay prohibitive latency for their grounding; and fine-tuned success rates remain below the threshold for dependable use. We present MolmoAct2, a fully open action reasoning model built for practical deployment, advancing its predecessor, MolmoAct along five axes. (1) MolmoAct2 is built on top of our new Molmo2-ER, a VLM backbone specialized for spatial and embodied reasoning, trained on a 3.3M-sample corpus with a specialize-then-rehearse recipe. (2) We release three new robot datasets spanning low-to-medium cost platforms: MolmoAct2-BimanualYAM Dataset, 720 hours of teleoperated bimanual trajectories that constitute the largest open bimanual dataset to date; MolmoAct2-DROID Dataset, a quality-filtered Franka subset of DROID; and MolmoAct2-SO100/101 Dataset, a quality-filtered SO-100/101 subset. (3) We train and release MolmoAct2-FAST Tokenizer, an open-weight, open-data action tokenizer trained on millions of trajectories across five embodiments. (4) We design a new VLA architecture to graft the discrete-token VLM into the flow-matching continuous-action expert via per-layer key-value (KV) conditioning. (5) we propose MolmoAct2-Think, an adaptive-depth reasoning variant that re-predicts depth tokens only for scene regions that change between timesteps, retaining geometric grounding at a fraction of prior latency. In the most extensive empirical study of any open VLA to date, spanning 7 simulation and real-world benchmarks, MolmoAct2 outperforms strong baselines including π0.5, while Molmo2-ER surpasses GPT-5 and Gemini Robotics ER-1.5 across 13 embodied-reasoning benchmarks. We release model weights, training code, and complete training data.
Learn More
Project page: https://allenai.org/blog/molmoact2
Code: https://github.com/allenai/molmoact2
ArXiV: https://arxiv.org/pdf/2605.02881v1
And check out our episode on the original MolmoAct:
This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit robopapers.substack.comEp#86: RISE: Self-Improving Robot Policy with Compositional World Model
Robot policies must be both reliable and highly capable to be useful; the best way to achieve this level of performance is with reinforcement learning. However, for reinforcement learning you are usually stuck …
Notas do episódio
Robot policies must be both reliable and highly capable to be useful; the best way to achieve this level of performance is with reinforcement learning. However, for reinforcement learning you are usually stuck between two difficult options: reinforcement in the real world is often risky and expensive, while reinforcement learning in a traditional simulator takes a lot of engineering work and has a persistent sim-to-real gap. What if instead you could train your robot purely in a world model?
RISE by Jiazhi Yang et al. uses a compositional world model to predict the future and evaluate progress. This allows for a self-improving pipeline, which learns a world model from real data and then learns how the robot should perform different tasks. This pipeline results in a data-driven way to improve policy performance from real data but without real-world reinforcement learning.
Watch Episode #86 of RoboPapers, with Chris Paxton and Jiafei Duan, to learn more!
Abstract
Despite the sustained scaling on model capacity and data acquisition, Vision-Language-Action (VLA) models remain brittle in contact-rich and dynamic manipulation tasks, where minor execution deviations can compound into failures. While reinforcement learning (RL) offers a principled path to robustness, on-policy RL in the physical world is constrained by safety risk, hardware cost, and environment reset. To bridge this gap, we present RISE, a scalable framework of robotic reinforcement learning via imagination. At its core is a Compositional World Model that (i) predicts multi-view future via a controllable dynamics model, and (ii) evaluates imagined outcomes with a progress value model, producing informative advantages for the policy improvement. Such compositional design allows state and value to be tailored by best-suited yet distinct architectures and objectives. These components are integrated into a closed-loop self-improving pipeline that continuously generates imaginary rollouts, estimates advantages, and updates the policy in imaginary space without costly physical interaction. Across three challenging real-world tasks, RISE yields significant improvement over prior art, with more than +35% absolute performance increase in dynamic brick sorting, +45% for backpack packing, and +35% for box closing, respectively.
Learn More
Project Page: https://opendrivelab.com/RISE/
ArXiV: https://arxiv.org/abs/2602.11075
This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit robopapers.substack.com
Aqui estão os 40 episódios mais recentes de 115. Os mais antigos estão no feed do podcast.
Podcasts parecidos
Mais podcasts de Tecnologia
Este podcast é seu?
Confirme com uma entrada no seu feed: você recebe um código aqui, adiciona como podcast:txt no canal do seu feed e nós verificamos. Depois disso, seu podcast aparece como “verificado” no SoundNode.