Mapping and Managing AI Agents’ Compliance Risk
Show notes
The award-winning Compliance into the Weeds is the only weekly podcast that takes a deep dive into compliance-related topics, literally going into the weeds to explore them more fully. Looking for some hard-hitting insights on compliance? Look no further than Compliance into the Weeds! In this special episode of Compliance into the Weeds, Tom Fox and Matt Kelly discuss a Darrow AI report on whether leading LLMs can build privacy-sensitive websites while recognizing compliance obligations.
Darrow had seven “frontier” models build three hypothetical sites involving personal data (pediatric lab results, credit scores, and mental health services) and scored them against a privacy/compliance checklist. With no compliance prompting, models averaged about 28/100; with generic privacy prompts, about 63.5; and with extensive documentation and technical specifications, about 88.6 (the top model around 95), with none reaching 100. They highlight “ghost controls,” where front-end compliance features (e.g., opt-out buttons) lack back-end functionality. The conversation stresses that prompting quality, technical specifications, and clear human oversight are essential, especially amid layoffs and “vibe coding,” and warns that GDPR/CCPA risks scale with AI speed. They note insurers requested the research and may push governance requirements.
Key highlights:
- Darrow Study Setup
- Workplace Oversight Risks
- Ghost Controls Explained
- Governance at Scale
- Beyond Websites to Projects
- Insurance Pressure Coming
Resources:
Tom
A multi-award-winning podcast, Compliance into the Weeds, was recently named one of the Top 25 Regulatory Compliance Podcasts, Top 10 Business Law Podcasts, and Top 12 Risk Management Podcasts. Compliance into the Weeds has been awarded a Davey, Communicator, and W3 Award, all for podcast excellence.
Learn more about your ad choices. Visit megaphone.fm/adchoices