What DeepSeek figured out, and why you shouldn't copy it (w/ Julien Launay & Sesh Nalla)

For years the recipe for better AI was simple: more data, bigger models.

Then DeepSeek's R1 showed models could learn to reason by practicing on problems with checkable answers, and Reinforcement Learning became the way forward.

But what actually is RL? And is it something every startup should be doing, or just a practice for big AI labs?

Julien Launay, cofounder of Adaptive ML, and Sesh Nalla, VP of Engineering at Datadog (which bought Adaptive), made big bets on RL before it was obvious.

In this episode, Julien, Sesh and Megan Reynolds discuss:
- What RL adds to pre-training ("pre-training is a block of marble")
- How an RL run actually works, step by step
- Why measuring an RL run is "an observability problem"
- Whether you should train your own model ("for 99% of people it doesn't make any sense")
- What scares the people building autonomous agents

Presented by Railway, your agent’s favorite cloud - get started at https://go.technically.dev/railway.

Original music by The Jazz Boys.

Show notes and the full transcript: https://technically.dev/podcast/all-about-reinforcement-learning

• (00:00) - Why is everyone talking about RL?
• (06:57) - What does RL add to pre-training?
• (11:36) - Why does Datadog care about RL?
• (22:07) - How does an RL run actually work?
• (29:36) - What limits RL?
• (40:51) - Where do models practice?
• (47:27) - Is it the weights or the harness?
• (54:00) - Should you train your own model?
• (01:04:22) - What should worry us?