Ep#111: SOLE-R1: Video-Language Reasoning as the Sole Reward for On-Robot Reinforcement Learning

Reinforcement learning is often seen as key to achieving scalable, highly reliable real-world robot performance, yet it requires good reward functions that can supervise the wide range of tasks that need to be performed. The obvious thing to try would be a large vision-language model (VLM), but these models often fail under common real-world conditions like partial observability and occlusion.

As a solution Philip Schroeder and Thomas Weng propose SOLE-R1, a video-language reasoning model trained to serve as a reward signal for online RL. Given real video, SOLE-R1 will reason over input video frames, generates reasoning traces, and predicts a reward signal. This allows online RL from random initialization on new manipulation tasks.

To learn more, watch Episode 111 of RoboPapers, with Michael Cho and Jiafei Duan!

Abstract

Vision-language models (VLMs) have shown impressive capabilities across diverse tasks, motivating efforts to leverage these models to supervise robot learning. However, when used as evaluators in reinforcement learning (RL), today's strongest models often fail under partial observability and distribution shift, enabling policies to exploit perceptual errors rather than solve the task. We introduce SOLE-R1 (Self-Observing LEarner), a video-language reasoning model explicitly designed to serve as the sole reward signal for online RL. Given only raw video observations and a natural-language goal, SOLE-R1 performs per-timestep spatiotemporal chain-of-thought (CoT) reasoning and produces dense estimates of task progress that can be used directly as rewards. To train SOLE-R1, we develop a large-scale video trajectory and reasoning synthesis pipeline that generates temporally grounded CoT traces aligned with continuous progress supervision. This data is combined with foundational spatial and multi-frame temporal reasoning, and used to train the model with a hybrid framework that couples supervised fine-tuning with RL from verifiable rewards. Across four different simulation environments and a real-robot setting, SOLE-R1 enables zero-shot online RL from random initialization: robots learn previously unseen manipulation tasks without ground-truth rewards, success indicators, demonstrations, or task-specific tuning. SOLE-R1 succeeds on 24 unseen tasks and substantially outperforms strong vision-language rewarders, including Robometer, RoboReward, ReWiND, GPT-5, and Gemini-3-Pro, while exhibiting markedly greater robustness to reward hacking. We release all models, data, code, and demos at the anonymous page: this https URL

Learn More

Project page: https://philip-mit.github.io/sole-r1/

ArXiV: https://arxiv.org/abs/2603.28730

Transcript

Transcripts are automatically generated by Riverside and are lightly edited by ChatGPT.

Michael Cho [00:06]

Hello guys, welcome to another episode of RoboPapers. Today we are really happy to have Philip Schroeder and Thomas Weng, two of the co-authors of this paper, SOLE-R1: Video-Language Reasoning as the Sole Reward for On-Robot Reinforcement Learning. Thank you so much for making the time, guys. Maybe as a start, Philip, you can share a bit about yourself and then Thomas.

Philip Schroeder [00:27]

Yeah, so my name is Philip. I’m a final-year PhD student at MIT in computer science. My work has mainly focused on improving the reasoning capabilities of language and vision-language models in challenging embodied settings. And over the last couple years, my work has mainly focused on reward reasoning over video, which is what we do for this paper.

Thomas Weng [00:55]

Cool. I’m Thomas. I’m a research scientist at RAInstitute. And more recently my work has been focusing on the pieces required to achieve like post-training and deployment for robotics. So reward modeling is one piece of that. On-robot reinforcement learning where you have data from deployment is also a piece of that. But that’s what I do.

Michael Cho [01:22]

All right, great to have both of you on. Yeah, maybe Philip you can take us through.

Philip Schroeder [01:27]

Yeah, yeah. Okay. So the goal of this paper was to develop a vision-language model that has strong enough reward reasoning that we can train policies from scratch through online reinforcement learning. So the model we developed is called SOLE-R1. And we show that it can serve as the sole reward signal for learning over 20 different unseen manipulation tasks through online RL. And I guess the motivating problem for this paper was that if you use just general vision-language models or VLM-based reward models that have recently come out, you get this reward hacking behavior when you try to train a policy from scratch using the predicted rewards.

So here, the task is to open the drawer, and you can see that these VLM-based reward models give very high rewards just for the policy fiddling with the drawer and just slightly opening it. And then here’s another example with the Pick Cube task in RoboSuite. Again, you see that the policy is getting these very high rewards by just finding these odd camerangles and positions that trick the VLM into thinking the task is being accomplished without actually doing the task. And you can see here the SOLE-R1 predicted rewards are plotted here in blue, and the ground-truth rewards are plotted in red. And you can see that solar SOLE-R1. Much more robust to this type of reward hacking.

And you kind of think of this paper as a recipe for how to train a reward model that is robust to reward hacking. And there’s kind of two major components to this that differentiate our work from prior work and really help with this reward hacking robustness. And I can quickly mention those and then go over them in more detail later. So the first is our training data mixture. We do I think three unique things with our training data mixture. One is that we include a lot of general spatial reasoning data. And a lot of this data is new. It came out with NeurIPS 2025 and ICLR 2026 papers. So at the time of this work it was data sets that were just released.

So it includes just general spatial re spatial reasoning data SSR chain-of-thought And some robotics data sets, some recent one called From Seeing to Doing, and some very famous ones like Embody chain-of-thought and Robo2VLM. And we combine this with data that forces the model to reason over multiple images within the same space, capturing different moments in time. So this includes general spatial reasoning data or spatiotemporal reasoning data spot the diff and robot-specific video data that we generate and then. Also curate from Open X-Embodiment. And our video data includes failure datand I think much more failure data than prior work because we generate a lot of it ourselves in RoboCasa. So that’s one big thing.

And then another big thing is that we annotate the time each time step of the videos with dense progress values and dense chain-of-thought reasoning traces to help the model explicitly reason about progress with these failure trajectories or suboptimal trajectories. And then we mix this with these artificial failure data from Open X So that’s the training data mixture. And then another novel thing we do is have the model do this chain-of-thought reasoning at each time step of the video. And this has been shown to significantly improve language model performance and visual language model performance. So and we show ablations where it helps significantly.

So the models con it at each time step is reasoning about what’s changing over time between the current and previous time step and predicting a dense progress value. Relative to the previous progress value that it predicted at the last time step. And then we do a lot of experiments that I can that I can show. But let me know if you have any questions so far. I can go over in more detail the both how we generate our training datand then how we train this chain-of-thought reasoning over video.

Jiafei Duan [05:56]

Can you let me yeah. Yeah, I’m gonna go ahead. Yeah, no, you go ahead.

Michael Cho…

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit robopapers.substack.com