#156 Bayesian Experimental Design & Active Learning, with Adam Foster
Show notes
Support & Resources
→ Support the show on Patreon
→ Bayesian Modeling Course (first 2 lessons free)
Our theme music is « Good Bayesian », by Baba Brinkman (feat MC Lars and Mega Ran). Check out his awesome work
Takeaways
Q: What is Bayesian experimental design and what problem does it solve?
A: It's the practice of using a Bayesian model to decide how to collect data before you collect it. Most statistical thinking starts with a fixed dataset. Bayesian experimental design sits upstream -- you have control over experimental parameters (which questions to ask, which reagents to mix, which conditions to test) and you want to choose them optimally. The Bayesian angle is to ask: what new data would most reduce my current uncertainty?
Q: When should you actually use Bayesian experimental design?
A: When two conditions hold: you have active control over how data is collected (not just passive observation), and you have a Bayesian model whose prior predictive distribution gives a reasonable picture of what typical data might look like. It's especially valuable when data collection is expensive or irreversible -- when the "committal step" of running an experiment has real cost, it's worth doing the analysis first.
Q: What is expected information gain (EIG) and why is it central to Bayesian experimental design?
A: EIG is the score you assign to a candidate experimental design -- the amount of information you expect to gain about your model parameters by running an experiment with that design. You compute it by simulating datasets from your prior predictive, doing Bayesian inference on each, and averaging how much the uncertainty decreased. What's remarkable is that you can derive the same quantity from two completely different starting points -- reducing parameter uncertainty, or maximizing outcome uncertainty while correcting for noise - and arrive at the same formula. That convergence is why EIG keeps being re-discovered independently across fields.
Full takeaways here
Chapters:
00:00 What is Bayesian experimental design and why does it matter?
00:06:02 What problem does Bayesian experimental design actually solve?
00:08:54 When should practitioners use Bayesian experimental design?
00:12:00 Is Bayesian experimental design changing how scientists work in practice?
00:15:04 What are the limitations of Bayesian experimental design?
00:17:55 What is expected information gain (EIG) and how does it work?
00:21:05 How do you compute expected information gain in practice?
00:23:48 What is active learning and how does it connect to Bayesian experimental design?
00:41:02 What is active learning by disagreement?
00:48:57 What is deep adaptive design and when should you00: use it?
00:56:02 How is Bayesian experimental design applied in protein dynamics and quantum chemistry?
01:01:58 What does a practical Bayesian experimental design workflow look like?
Thank you to my Patrons for making this episode possible!
Links from the show