Ep#105: Beyond Imitation: Self-Improving Robot Policies via Off-Policy Q-Planning

Imitation learning, especially with interventions, has driven so much recent robotics progress. However, improving a policy via targeted interventions until it reaches a useful and deployable success rate is a time and labor intensive process. Instead, wouldn’t it be great if policies could improve on their own?

That’s what Varun Giridhar and Animesh Garg join us to talk about. In Q-Planning, they start with a large policy like pi-0.5, and add a Q-function estimator to predict value instead of just actions, then use both successful and failed rollouts to update this Q-function online, then use it to guide sampling and trajectory selection. With just a few rollouts they can dramatically improve policy performance online. This provides a way to do really difficult tasks like inserting a credit card into a wallet, increasing success rate from 25% to 80% in just a few iterations.

Learn more in Episode 105 of RoboPapers, hosted by Michael Cho, Chris Paxton, and Ruijie He!

Abstract

Behaviour Cloning (BC) has driven remarkable progress in robot manipulation, yet it is fundamentally limited by its inability to self-improve: a policy that fails cannot learn from that failure without additional human demonstrations. Reinforcement Learning fine-tuning offers a path to self-improvement but has proven difficult to scale to the multi-billion-parameter models underpinning modern robot policies. We propose Q-Planning, which equips a large visuomotor BC policy with a small off-policy Q-function. Because a Q-function estimates value rather than imitates actions, it can be trained on the same successful demonstrations as the BC policy and later absorb both successful and failed deployment rollouts, an asymmetry BC does not have. We exploit this asymmetry to enable value-guided action selection at inference (a single-step QQ-weighted average over BC draws) and online self-improvement that fine-tunes only the Q-function, leaving the BC weights untouched. On LIBERO and bimanual RoboTwin, ten iterations of self-improvement lift every benchmark score we tested (LIBERO-10 93 → 99%, RoboTwin 83.8 → 91.4%) and shorten successful episodes on the near-ceiling suites (LIBERO-Object, LIBERO-Goal). On two contact-rich bimanual real-robot tasks, the same loop (BC frozen, no human intervention) improves purely from its own deployment rollouts: stack-cups 40 → 90% and insert-wallet 25 → 80% in five iterations, whereas SFT on successful rollouts alone stalls at 55% and 30%. Under an identical online budget Q-Planning is the only method, among Best-of-N, filtered SFT, IBRL, DSRL, and DAWR, that improves stably from failures without training an auxiliary actor.

Learn More

Project page: https://q-planning.github.io/

ArXiV: https://arxiv.org/abs/2608.21204

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit robopapers.substack.com