Skip to content

How to Build Better AI Evals with Claude Code in 5 Steps | Shreya & Hamel

August 23, 2026 · 54 min

Show notes

📌 Copy the AI system I use to run a one-person $1M+ business: https://www.behindthecraft.com

🤝 Subscribe to my practical AI newsletter read by 150K+ professionals: https://creatoreconomy.so


Shreya and Hamel teach the industry standard course for AI evaluations taken by 4,500+ students from OpenAI, Google, and more. In our interview, I asked them to give live feedback on the AI evals that I built for my skills and demo how anyone can run evals with ChatGPT/Claude and their free evals skill. Don’t miss this up to date primer on evals from the real experts. 🙂


P.S., I asked Shreya and Hamel to give my audience 25% off the next cohort of their AI evals course. It has 4.7 stars with 900 reviews which is almost unheard of on Maven. Sign up via the link here: https://maven.com/parlance-labs/evals?utm_campaign=peteryang&utm_medium=affiliate&utm_source=maven&promoCode=PETERYANG


We talked about:

(00:00) Why AI evals are completely different now with the latest models

(02:30) Demo: Live audit of the evals that I built for my AI skills

(05:14) Top down vs. bottom up evals and why AI sucks at the latter

(08:16) Spinning up AI agents to grade each eval criteria

(14:32) Demo: Using Shreyas AI skill to run evals in Claude

(23:46) How to review output labels to identify failures

(34:37) How to turn failures into reusable evals

(43:19) Where human judgement is still needed


Resources mentioned:

Free eval skills from Shreya: https://github.com/ai-evals-course/evals-skills

Do automated evals work?: https://parlance-labs.com/blog/posts/auto-evals/


Thanks to our sponsors:

Wispr Flow — 4x faster than typing with your voice — https://ref.wisprflow.ai/peteryang


📌 Get the takeaways: https://creatoreconomy.so/p/how-to-build-better-ai-evals-with-claude-code-shreya-hamel


Where to find Shreya and Hamel:

Shreya: https://www.sh-reya.com/

Hamel: https://hamel.dev/


Subscribe to this channel - more interviews coming soon!

All episodes of Behind the Craft

Settings

Display

Language

Decides which stations you see first on the home page.

History

Recently played stations and episodes and your latest searches are only stored in this browser.

Notifications

A short notice when a podcast you follow has a new episode. Turn off single podcasts with the bell on their page.

Account

Save favorites – in the app and on the website.

Keyboard shortcuts

/
Open search
Space
Play or pause
← →
In the player: back 15 s, forward 30 s (episodes) · in a row: next tile
↑ ↓
In a row: target within the tile
Home / End
In a row: first or last tile
Esc
Close window or menu
?
This overview
Now playing

Go to station page

Play next

    Add episodes with “Play next” – they play after the current episode.