Ep#108: VLS: Steering Pretrained Robot Policies via Vision–Language Models

Pretrained vision-language-action models for robots often fail when a task is performed in a slightly different environment. With Vision-Language Steering, you can get better generalization performance out of a frozen robot policy, by steering the sampling process of your pretrained diffusion or flow-matching policy to handle out-of-distribution changes like obstacle position or changing visual appearance. Shuo Lin and Ishneet Sukhviner Singh join us to talk about how to better generalize robot policies with no new training data.

Watch Episode 108 of RoboPapers today, with Jiafei Duan and Ruijie He!

Abstract

Why do pretrained diffusion or flow-matching policies fail when the same task is performed near an obstacle, on a shifted support surface, or amid mild clutter? Such failures rarely reflect missing motor skills; instead, they expose a limitation of imitation learning under train–test shifts, where action generation is tightly coupled to training-specific spatial configurations and task specifications. Retraining or fine-tuning to address these failures is costly and conceptually misaligned, as the required behaviors already exist but cannot be selectively adapted at test time.

We propose Vision–Language Steering (VLS), a training-free framework for inference-time adaptation of frozen generative robot policies. VLS treats adaptation as an inference-time control problem, steering the sampling process of a pretrained diffusion or flow-matching policy in response to out-of-distribution observation–language inputs without modifying policy parameters. By leveraging vision–language models to synthesize trajectory-differentiable reward functions, VLS guides denoising toward action trajectories that satisfy test-time spatial and task requirements.

Learn More

Project Page: https://vision-language-steering.github.io/webpage/

ArXiV: https://arxiv.org/pdf/2602.03973

Transcript

Transcripts are lightly edited by ChatGPT.

RJ He (00:01)

Okay. Hi everyone. Welcome to another episode of Robot Papers. Today we are super excited to have with us Shuo Liu and Ishneet, who will be sharing their paper on Vision-Language Steering: Steering Pre-Trained Robot Policies via Vision-Language Models. Shuo, Ishneet, do you all want to introduce yourselves?

Shuo Liu (00:29)

Yes. Hi everyone. My name is Shuo Liu. For now I’m a research engineer at Cortex Robot AI and previously I graduated from University of Washington, where I was also a student researcher at Allen Institute for AI, advised by Professor Ranji Krishna and Professor Jiafei Duan, focus on robot learning from develop foundation models to deploying them into real world.

RJ He (01:01)

Isn’t it?

Ishneet (01:02)

Yeah. Hi everyone, my name is Ishneet. I work with Professor Jiafei on this paper, VLS, with Shuo. I’m currently at Menlo Research, where I also work on Asimov, and I’ll be going to Oxford for my undergrad later this year.

RJ He (01:17)

Super exciting. Amazing that you’re already having publications at this young age. It’s awesome. Okay, without further ado, Shuo, why don’t you take us through the paper?

Shuo Liu (01:26)

Okay, sure, yeah. Let me share the screen. Okay, so just make sure it’s it’s good to say right.

RJ He (01:38)

Yeah, it’s good.

Shuo Liu (01:39)

Yeah. Hi everyone. Today I want to share our work vision language theory. This work is finished when I was a master student at the University of Washington and advised by Professor Ranjik Krishna and Professor Jiafei Duan. And so today I want to introduce motivation of this work and the method we solve this problem and the experiment results and also some future works maybe could be done from this direction. So the problem is from the observation of some small front twining of the Like models like π0 or π0 Po, we found that current generative robot policies are can’t are kind of tightly coupled of the training specific configurations, so which cause some failing to decouple the reusable motor skills from the normal task constraints. Like for example, here we have some cubes on the table and if you want to stack them up to a tower with specific numbers you may call the π0.5 pre-trained value a policy to stack some cubes like if you want to stack two cubes and using the termination signal like the pushing the red button on the table it can do very well because you Because for this specific task you can fine twin the base policy a bit, like stack two cubes and push the button or stack four cubes to push the button. And but what if at test time you ask the base policy to stack three cubes? It’s never it’s never contained in the fine-tuning data set, so the base policy

Shuo Liu (03:33)

Can learn how to stack two cubes or stack four cubes and push the button, but it will never know how to st how to stack three cubes and push the button. This is because the training configuration we combine the language instruction and the image observation with the task two cubes and four cubes together and the polic the base policy is never contained with task specific instruction stack three cubes. So the motor skills like moving the gripper to the cube and stack it on another one and going push the red button is already included in the base policy. But at test time if you given another out of distribution scenario or out of distribution language instruction, the base policy can always fail. So we think this is from the standard imitation learning reset. So for now if we have a base policy like take π0.5 or Momo Act 2 for example, the base policy can be trained on the behavior cloning target, which means given an expert data set which contains observation, action, and the language for each index and the optimization target is to maximize the likelihood of the base policy to be to predict the next action chunk based on current observation and the absurd language instruction. So the objective is stand static and also is very distribution dependent, which means after fun training your base policy can only capture the special and semantic correlations on the training manifold if you change the observation or language at test time the base policy may confuse. This observation is clear at our daily work because if you take one pre-trained VLA policy you just don’t want to like use it in the

Shuo Liu (05:53)

In distribution tasks, you may change the scenario, you may change the language to the VLA. So for example, sometimes we may face observation out of distribution signal which means we may made some visual or special shift and some semantic shift which means the change of language distribution. So at that time the world may drift away from the training distribution and how we can solve it. We for now we have some two solutions. The first one is we recollect a small dataset out of distribution scenario and run another training cycle. But we need to repeat this circle for every new deployment and make the base policy useful for this specific scenario. But this is kind of misaligned with the target of we train very large VLA policy. That’s because the target is to make the base policy general generalizable but and we can using the pre-train and post-train recipe to make the skills like pick and place or open close this kind of low level skills already inside the VLA policy but we are paying the cost to relear the existing skills at test time s simply because we are not selectively adapt to the base policy at test time. So in this work we want to train we want to try another way that is steering the base policy at inference time, which means we’re taking the we’re taking the ability of generative models like photomatching or diffusion models which are

Shuo Liu (07:46)

Broadly used in current VLAs like the action part action expert part. We can take the advan advantag advancement of their ability to denoise iteratively. So for example the diffusion model is denoising to the target distribution from Gaussian noise and using many diff denoising steps and the flow matching are learning denoising paths to from the Ga…

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit robopapers.substack.com