Model Soups: Averaging Fine-Tuned Weights Beats Ensembles' Cost

This episode examines "Model soups," a paper showing that averaging the weights of several fine-tuned models can raise accuracy without adding any inference cost. It explains why this should be risky: the loss landscape has many basins, and averaging across a barrier breaks the network. Models fine-tuned from the same pre-trained checkpoint tend to share a basin, which is what makes the averaging work. The discussion covers the uniform soup and the greedy soup. The greedy soup adds checkpoints in order of validation accuracy and keeps one only if accuracy doesn't drop. It also compares soups against ensembles, which need k forward passes, and against stochastic weight averaging, which averages along a single training run. Listeners get a clear account of the ImageNet distribution-shift benchmarks, the oracle baseline, and an open question about whether the gains on shifted data come from soups simply pulling weights back toward the initialization. Sources: 1. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time — Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S. Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, Ludwig Schmidt, 2022 http://arxiv.org/abs/2203.05482 2. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time — Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S. Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, Ludwig Schmidt, 2022 (ICML) https://scholar.google.com/scholar?q=Model+soups%3A+averaging+weights+of+multiple+fine-tuned+models+improves+accuracy+without+increasing+inference+time 3. Robust fine-tuning of zero-shot models (WiSE-FT) — Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo-Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, Ludwig Schmidt, 2021 (CVPR 2022) https://scholar.google.com/scholar?q=Robust+fine-tuning+of+zero-shot+models+%28WiSE-FT%29 4. Averaging Weights Leads to Wider Optima and Better Generalization (Stochastic Weight Averaging) — Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, Andrew Gordon Wilson, 2018 (UAI) https://scholar.google.com/scholar?q=Averaging+Weights+Leads+to+Wider+Optima+and+Better+Generalization+%28Stochastic+Weight+Averaging%29 5. Merging Models with Fisher-Weighted Averaging — Michael Matena, Colin Raffel, 2022 (NeurIPS) https://scholar.google.com/scholar?q=Merging+Models+with+Fisher-Weighted+Averaging 6. Do Better ImageNet Models Transfer Better? — Simon Kornblith, Jonathon Shlens, Quoc V. Le, 2019 (CVPR) https://scholar.google.com/scholar?q=Do+Better+ImageNet+Models+Transfer+Better%3F 7. Learning Transferable Visual Models From Natural Language Supervision (CLIP) — Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, Ilya Sutskever, 2021 (ICML) https://scholar.google.com/scholar?q=Learning+Transferable+Visual+Models+From+Natural+Language+Supervision+%28CLIP%29 8. Scaling Vision Transformers — Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, Lucas Beyer, 2021 (CVPR 2022) https://scholar.google.com/scholar?q=Scaling+Vision+Transformers 9. What is being transferred in transfer learning? — Behnam Neyshabur, Hanie Sedghi, Chiyuan Zhang, 2020 (NeurIPS) https://scholar.google.com/scholar?q=What+is+being+transferred+in+transfer+learning%3F 10. Linear Mode Connectivity and the Lottery Ticket Hypothesis — Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M. Roy, Michael Carbin, 2020 (ICML) https://scholar.google.com/scholar?q=Linear+Mode+Connectivity+and+the+Lottery+Ticket+Hypothesis 11. Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs — Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry Vetrov, Andrew Gordon Wilson, 2018 (NeurIPS) https://scholar.google.com/scholar?q=Loss+Surfaces%2C+Mode+Connectivity%2C+and+Fast+Ensembling+of+DNNs 12. The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks — Rahim Entezari, Hanie Sedghi, Olga Saukh, Behnam Neyshabur, 2021 (ICLR 2022) https://scholar.google.com/scholar?q=The+Role+of+Permutation+Invariance+in+Linear+Mode+Connectivity+of+Neural+Networks 13. Sharpness-Aware Minimization for Efficiently Improving Generalization — Pierre Foret, Ariel Kleiner, Hossein Mobahi, Behnam Neyshabur, 2020 (ICLR 2021) https://scholar.google.com/scholar?q=Sharpness-Aware+Minimization+for+Efficiently+Improving+Generalization 14. Averaging Weights Leads to Wider Optima and Better Generalization (SWA) — Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, Andrew Gordon Wilson, 2018 https://scholar.google.com/scholar?q=Averaging+Weights+Leads+to+Wider+Optima+and+Better+Generalization+%28SWA%29 15. Git Re-Basin: Merging Models modulo Permutation Symmetries — Samuel K. Ainsworth, Jonathan Hayase, Siddhartha Srinivasa, 2023 https://scholar.google.com/scholar?q=Git+Re-Basin%3A+Merging+Models+modulo+Permutation+Symmetries 16. Editing Models with Task Arithmetic — Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, et al., 2023 https://scholar.google.com/scholar?q=Editing+Models+with+Task+Arithmetic 17. TIES-Merging: Resolving Interference When Merging Models — Prateek Yadav, Derek Tam, Leshem Choshen, Colin Raffel, Mohit Bansal, 2023 https://scholar.google.com/scholar?q=TIES-Merging%3A+Resolving+Interference+When+Merging+Models 18. Ensemble of Averages: Improving Model Selection and Boosting Performance in Domain Generalization (Model Ratatouille / DiWA / Rewarded Soups line) — Alexandre Ramé, Matthieu Kirchmeyer, et al., 2022 https://scholar.google.com/scholar?q=Ensemble+of+Averages%3A+Improving+Model+Selection+and+Boosting+Performance+in+Domain+Generalization+%28Model+Ratatouille+%2F+DiWA+%2F+Rewarded+Soups+line%29 19. Deep Ensembles: A Loss Landscape Perspective — Stanislav Fort, Huiyi Hu, Balaji Lakshminarayanan, 2019 https://scholar.google.com/scholar?q=Deep+Ensembles%3A+A+Loss+Landscape+Perspective 20. LoRA: Low-Rank Adaptation of Large Language Models — Edward Hu et al., 2021 https://scholar.google.com/scholar?q=LoRA%3A+Low-Rank+Adaptation+of+Large+Language+Models Interactive Visualization: Model Soups: Averaging Fine-Tuned Weights Beats Ensembles' Cost