#167 Amortized Inference & GPUs for Stan, with Paul-Christian Bürkner & Steve Bronder

Support & Resources
→ Support the show on Patreon
→ Bayesian Modeling Course (first 2 lessons free)

Our theme music is « Good Bayesian », by Baba Brinkman (feat MC Lars and Mega Ran). Check out his awesome work!

Takeaways:

Q: What is amortized Bayesian inference and how does it differ from MCMC?

Instead of running MCMC on each dataset, you train neural networks on data simulated from the model, so you don't need likelihood or prior densities. Once training is done, inference on a new dataset is essentially instant: new data in, posterior out. The price is everything neural networks bring with them, from unstable training to a lack of guarantees. Paul's stated goal is to make these methods as reliable and robust as MCMC while staying orders of magnitude faster, and he's candid that he doesn't know whether the field will ever get there.

Q: Why would a Bayesian model not have a usable likelihood?

A: Usually not because the likelihood can't be written down in principle, but because computing it is so expensive or numerically unstable that it's intractable in practice. Paul gives two cases. First, mechanistic models built on complicated differential equations, as in physical chemistry or biology, where you can't solve through the equations to get densities and gradients in a stable, fast way. Second, big latent variable models with thousands of latent variables between the parameters and the data, where marginalizing them out is hopeless. In both cases, simulating from the model is easy: simulate the parameters, the latents and the data, throw the latents away, and learn the relation between parameters and data from the simulations.

Full takeaways here!

Chapters:

15:33 What is amortized Bayesian inference and when should you use it instead of MCMC?
23:18 Why would a Bayesian model not have a usable likelihood?
26:03 What's holding back GPU computing in Stan?
31:14 What are sum-to-zero constraints and why do they make models sample faster?
35:55 What is a Jacobian adjustment in Bayesian statistics?
39:17 How do you diagnose failures in amortized Bayesian inference?
42:01 What is leave-one-group-out cross-validation for hierarchical models?
45:57 Do undertrained neural posteriors overestimate or underestimate uncertainty?
49:06 How do Stan developers use AI tools in their work?
55:47 Should Stan and brms hide the inference algorithm from users?

Thank you to my Patrons for making this episode possible!

Full links from the show here!