Vector Policy Optimization: Training for Diversity to Improve Test-Time Search

This episode examines "Vector Policy Optimization," a proposed drop-in replacement for the GRPO advantage that trains models to produce diverse candidates rather than a single best answer, so that test-time search such as best-of-N and evolutionary code search has more to work with. It lays out why standard RL post-training collapses onto one high-reward mode. The evidence cited is that RL models beat base models at small k but fall behind at large k, and that KL-regularized RL is argued to collapse structurally. The method treats rewards as vectors, such as per-test-case or per-hop scores. It samples random scalarization weights from a uniform Dirichlet during training, so candidates can land at different points on the Pareto frontier. The discussion compares this with pass@k optimization, entropy bonuses and embedding-based diversity rewards, and with a multi-answer baseline that gives the model room to diversify but no incentive to do so. It also defines the evaluation metrics, pass@k, best@k and reward-space diversity. For listeners, the interest is the argument that giving up some single-shot accuracy can pay off once a search procedure sits on top of the model. Sources: 1. Vector Policy Optimization: Training for Diversity Improves Test-Time Search — Ryan Bahlous-Boldi, Isha Puri, Idan Shenfeld, Akarsh Kumar, Mehul Damani, Sebastian Risi, Omar Khattab, Zhang-Wei Hong, Pulkit Agrawal, 2026 http://arxiv.org/abs/2605.22817 2. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models — Zhihong Shao, Peiyi Wang, Qihao Zhu, et al. (DeepSeek-AI), 2024 https://scholar.google.com/scholar?q=DeepSeekMath%3A+Pushing+the+Limits+of+Mathematical+Reasoning+in+Open+Language+Models 3. Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? — Yang Yue, Zhiqi Chen, Rui Lu, et al., 2025 https://scholar.google.com/scholar?q=Does+Reinforcement+Learning+Really+Incentivize+Reasoning+Capacity+in+LLMs+Beyond+the+Base+Model%3F 4. Large Language Monkeys: Scaling Inference Compute with Repeated Sampling — Bradley Brown, Jordan Juravsky, Ryan Ehrlich, et al., 2024 https://scholar.google.com/scholar?q=Large+Language+Monkeys%3A+Scaling+Inference+Compute+with+Repeated+Sampling 5. Understanding the Effects of RLHF on LLM Generalisation and Diversity — Robert Kirk, Ishita Mediratta, Christoforos Nalmpantis, et al., 2024 https://scholar.google.com/scholar?q=Understanding+the+Effects+of+RLHF+on+LLM+Generalisation+and+Diversity 6. A Survey of Multi-Objective Sequential Decision-Making — Diederik M. Roijers, Peter Vamplew, Shimon Whiteson, Richard Dazeley, 2013 https://scholar.google.com/scholar?q=A+Survey+of+Multi-Objective+Sequential+Decision-Making 7. A Practical Guide to Multi-Objective Reinforcement Learning and Planning — Conor F. Hayes, Roxana Rădulescu, Eugenio Bargiacchi, et al., 2022 https://scholar.google.com/scholar?q=A+Practical+Guide+to+Multi-Objective+Reinforcement+Learning+and+Planning 8. A Generalized Algorithm for Multi-Objective Reinforcement Learning and Policy Adaptation — Runzhe Yang, Xingyuan Sun, Karthik Narasimhan, 2019 https://scholar.google.com/scholar?q=A+Generalized+Algorithm+for+Multi-Objective+Reinforcement+Learning+and+Policy+Adaptation 9. Rewarded Soups: Towards Pareto-Optimal Alignment by Interpolating Weights Fine-Tuned on Diverse Rewards — Alexandre Ramé, Guillaume Couairon, Corentin Dancette, et al., 2023 https://scholar.google.com/scholar?q=Rewarded+Soups%3A+Towards+Pareto-Optimal+Alignment+by+Interpolating+Weights+Fine-Tuned+on+Diverse+Rewards 10. Pass@k Policy Optimization: Solving Harder Reinforcement Learning Problems — Christian Walder, Deep Karkhanis, 2025 https://scholar.google.com/scholar?q=Pass%40k+Policy+Optimization%3A+Solving+Harder+Reinforcement+Learning+Problems 11. Inference-Aware Fine-Tuning for Best-of-N Sampling in Large Language Models — Yinlam Chow, Guy Tennenholtz, Izzeddin Gur, et al. (Google DeepMind), 2024 https://scholar.google.com/scholar?q=Inference-Aware+Fine-Tuning+for+Best-of-N+Sampling+in+Large+Language+Models 12. AlphaEvolve: A coding agent for scientific and algorithmic discovery — Alexander Novikov, Ngân Vũ, Marvin Eisenberger, et al. (Google DeepMind), 2025 https://scholar.google.com/scholar?q=AlphaEvolve%3A+A+coding+agent+for+scientific+and+algorithmic+discovery 13. Mathematical discoveries from program search with large language models (FunSearch) — Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov, et al., 2024 https://scholar.google.com/scholar?q=Mathematical+discoveries+from+program+search+with+large+language+models+%28FunSearch%29 14. ShinkaEvolve: Towards Open-Ended and Sample-Efficient Program Evolution — Robert Tjarko Lange, Yuki Imajuku, Edoardo Cetin (Sakana AI), 2025 https://scholar.google.com/scholar?q=ShinkaEvolve%3A+Towards+Open-Ended+and+Sample-Efficient+Program+Evolution 15. Illuminating search spaces by mapping elites (MAP-Elites) / Quality Diversity: A New Frontier for Evolutionary Computation — Jean-Baptiste Mouret and Jeff Clune (2015); Justin K. Pugh, Lisa B. Soros, Kenneth O. Stanley (2016), 2015-2016 https://scholar.google.com/scholar?q=Illuminating+search+spaces+by+mapping+elites+%28MAP-Elites%29+%2F+Quality+Diversity%3A+A+New+Frontier+for+Evolutionary+Computation 16. Reaching Beyond the Mode: RL for Distributional Reasoning in Language Models — Isha Puri, Mehul Damani, Idan Shenfeld, Marzyeh Ghassemi, Jacob Andreas, Yoon Kim, 2026 https://scholar.google.com/scholar?q=Reaching+Beyond+the+Mode%3A+RL+for+Distributional+Reasoning+in+Language+Models 17. KL-Regularized Reinforcement Learning is Designed to Mode Collapse — Anthony GX-Chen, Jatin Prakash, Jeff Guo, Rob Fergus, Rajesh Ranganath, 2025 https://scholar.google.com/scholar?q=KL-Regularized+Reinforcement+Learning+is+Designed+to+Mode+Collapse 18. The Best of N Worlds: Aligning Reinforcement Learning with Best-of-N Sampling via max@k Optimisation — Farid Bagirov, Mikhail Arkhipov, Ksenia Sycheva, Evgeniy Glukhov, Egor Bogomolov, 2025 https://scholar.google.com/scholar?q=The+Best+of+N+Worlds%3A+Aligning+Reinforcement+Learning+with+Best-of-N+Sampling+via+max%40k+Optimisation 19. Maximum Likelihood Reinforcement Learning (MaxRL) — Fahim Tajwar, Guanning Zeng, Yueer Zhou, Yuda Song, et al., 2026 https://scholar.google.com/scholar?q=Maximum+Likelihood+Reinforcement+Learning+%28MaxRL%29 20. GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimization — Shih-Yang Liu, Xin Dong, Ximing Lu, Shizhe Diao, et al. (NVIDIA), 2026 https://scholar.google.com/scholar?q=GDPO%3A+Group+reward-Decoupled+Normalization+Policy+Optimization+for+Multi-reward+RL+Optimization 21. A Survey of Multi-Objective Sequential Decision-Making / A Practical Guide to Multi-Objective RL and Planning — Roijers, Vamplew, Whiteson, Dazeley (2013); Hayes et al. (2022), 2013/2022 https://scholar.google.com/scholar?q=A+Survey+of+Multi-Objective+Sequential+Decision-Making+%2F+A+Practical+Guide+to+Multi-Objective+RL+and+Planning 22. Lexicase selection (Spector 2012; La Cava et al. 2019; Ni et al. 2024, DALex) — Lee Spector; William La Cava, Thomas Helmuth, et al.; Andrew Ni, Li Ding, Lee Spector, 2012/2019/2024 https://scholar.google.com/scholar?q=Lexicase+selection+%28Spector+2012%3B+La+Cava+et+al.+2019%3B+Ni+et+al.+2024%2C+DALex%29 23. Inference-Aware Fine-Tuning for Best-of-N Sampling in Large Language Models / BOND / BoNBoN / InfAlign — Chow et al.; Sessa et al.; Gui, Gârbacea, Veitch; Balashankar et al., 2024-2025 https://scholar.google.com/scholar?q=Inference-Aware+Fine-Tuning+for+Best-of-N+Sampling+in+Large+Language+Models+%2F+BOND+%2F+BoNBoN+%2F+InfAlign 24. AlphaEvolve / FunSearch / ShinkaEvolve / OpenEvolve — Novikov et al.; Romera-Paredes et al.; Lange, Imajuku, Cetin; Sharma, 2023-2025 https://scholar.google.com/scholar?q=AlphaEvolve+%2F+FunSearch+%2F+ShinkaEvolve+%2F+OpenEvolve 25. Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment — Rui Yang, Xiaoman Pan, Feng Luo, et al., 2024 https://scholar.google.com/scholar?q=Rewards-in-Context%3A+Multi-objective+Alignment+of+Foundation+Models+with+Dynamic+Preference+Adjustment 26. Rewarded Soups / Directional Preference Alignment — Rame et al. (2023); Wang et al. (2024), 2023/2024 https://scholar.google.com/scholar?q=Rewarded+Soups+%2F+Directional+Preference+Alignment 27. DARLING (diversity-aware RL) and entropy-preserving GRPO variants (DAPO clip-higher) — Li et al.; Yu et al., 2025 https://scholar.google.com/scholar?q=DARLING+%28diversity-aware+RL%29+and+entropy-preserving+GRPO+variants+%28DAPO+clip-higher%29 Interactive Visualization: Vector Policy Optimization: Training for Diversity to Improve Test-Time Search