The Quest for Embedded Evaluators
Shownotes
Anthropic has committed to embedded evaluators: outsiders placed inside the lab with employee-level access, reporting on what they see. There is only one problem: who will the evaluators be? Zvi looks at a public letter setting out minimum standards for credible evaluators, Anthropic’s partnership with Accenture and its plans to include METR, Drake Thomas’s ranking of the week’s takes from worst to best, and OpenAI’s new call for international frontier standards.
The Don’t Worry About the Vase Podcast is a listener-supported podcast. To receive new posts and support the cost of creation, consider becoming a free or paid subscriber.
This has been an Askwho Casts audio conversion. If you would like your own private feed of audio conversions of any blog posts you would like to listen to, You can sign up For Askwho Casts Pro at https://app.askwhocasts.com/, Where you can give any post the multi-voiced podcast treatment, into your own podcast feed. Thanks for listening.
* 00:00 - Introduction
* 02:03 - Table of Contents
* 02:39 - Look, All I’m Asking For Is That You Find A Highly-Qualified, Experienced, Trustworthy, Non-Conflicted Source of Embedded Evaluators That Will Work Entirely For Free, Without Government Assistance or Money from EA Sources Not Chosen By the Lab
* 06:10 - Anthropic Partners with Accenture for Embedded Evaluation, also Plans to Include METR
* 12:46 - Reading the METR
* 16:40 - OpenAI Suggests Doing The Least We Can Do
Get full access to DWAtV Podcast at dwatvpodcast.substack.com/subscribe