Soup-of-Experts: Pretraining Specialists via Parameter Averaging

This episode examines "Soup-of-Experts," an Apple paper on pretraining one bank of parameters that can be averaged into a small specialist model for any data mixture, with no retraining. It sets up the background: clustered importance sampling over 4096 RedPajama-v2 clusters, model soups from fine-tuning, per-domain expert baselines, and the hypernetwork framing. It also notes closely related prior work, Branch-Train-Merge and c-BTM, that the paper doesn't compare against. The discussion then moves to the mechanism: shared parameters plus 128 experts, combined by a small MLP that maps a domain histogram to mixing coefficients. Listeners get a clear look at the amortization argument, which is paying one large pretraining cost so each later specialist is nearly free. They also get a skeptical read on whether the headline comparison holds up. Sources: 1. Soup-of-Experts: Pretraining Specialist Models via Parameters Averaging — Pierre Ablin, Angelos Katharopoulos, Skyler Seto, David Grangier, 2025 http://arxiv.org/abs/2502.01804 2. Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Models — Margaret Li, Suchin Gururangan, Tim Dettmers, Mike Lewis, Noah A. Smith, Luke Zettlemoyer, 2022 https://scholar.google.com/scholar?q=Branch-Train-Merge%3A+Embarrassingly+Parallel+Training+of+Expert+Language+Models 3. Scaling Expert Language Models with Unsupervised Domain Discovery (c-BTM) — Suchin Gururangan, Margaret Li, Mike Lewis, Weijia Shi, Tim Althoff, Noah A. Smith, Luke Zettlemoyer, 2023 https://scholar.google.com/scholar?q=Scaling+Expert+Language+Models+with+Unsupervised+Domain+Discovery+%28c-BTM%29 4. Hypernetworks — David Ha, Andrew Dai, Quoc V. Le, 2017 https://scholar.google.com/scholar?q=Hypernetworks 5. HyperTuning / Hypernetwork-based task adaptation (e.g., HyperFormer, Hyper-X) — Various (Mahabadi et al.; Üstün et al.; Phang et al.), 2021-2023 https://scholar.google.com/scholar?q=HyperTuning+%2F+Hypernetwork-based+task+adaptation+%28e.g.%2C+HyperFormer%2C+Hyper-X%29 6. Mixture of Experts Models / Soft MoE (From Sparse to Soft Mixtures of Experts) — Joan Puigcerver, Carlos Riquelme, Basil Mustafa, Neil Houlsby, 2024 https://scholar.google.com/scholar?q=Mixture+of+Experts+Models+%2F+Soft+MoE+%28From+Sparse+to+Soft+Mixtures+of+Experts%29 7. Lory: Fully Differentiable Mixture-of-Experts for Autoregressive Language Model Pre-training — Zexuan Zhong, Mengzhou Xia, Danqi Chen, Mike Lewis, 2024 https://scholar.google.com/scholar?q=Lory%3A+Fully+Differentiable+Mixture-of-Experts+for+Autoregressive+Language+Model+Pre-training 8. Sheared LLaMA / structured pruning to a target-task model — Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, Danqi Chen, 2023 https://scholar.google.com/scholar?q=Sheared+LLaMA+%2F+structured+pruning+to+a+target-task+model 9. DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pre-training — Sang Michael Xie et al., 2023 https://scholar.google.com/scholar?q=DoReMi%3A+Optimizing+Data+Mixtures+Speeds+Up+Language+Model+Pre-training 10. Editing Models with Task Arithmetic / TIES-Merging / DARE — Ilharco et al.; Yadav et al.; Yu et al., 2022-2024 https://scholar.google.com/scholar?q=Editing+Models+with+Task+Arithmetic+%2F+TIES-Merging+%2F+DARE 11. Language Models Scale Reliably With Over-Training and on Downstream Tasks — Samir Yitzhak Gadre et al., 2024 https://scholar.google.com/scholar?q=Language+Models+Scale+Reliably+With+Over-Training+and+on+Downstream+Tasks 12. Model Merging in LLMs, MLLMs, and Beyond / Merging pretrained models survey — Various, 2024-2025, 2024 https://scholar.google.com/scholar?q=Model+Merging+in+LLMs%2C+MLLMs%2C+and+Beyond+%2F+Merging+pretrained+models+survey Interactive Visualization: Soup-of-Experts: Pretraining Specialists via Parameter Averaging