Preparing for the Worst: Tabletop Exercises, Unit Tests, and Everything In Between
Show notes
In this episode of "How Many CTOs Does It Take?" podcast, hosts Scott Porad and Brad Hefta-Gaub discuss TaskRabbit's overnight AWS region migration from us-west-1 to us-west-2, describing the months of preparation, detailed runbook, large war-room call, minor cleanup issues, and why periodic region moves can serve as valuable "hygiene" to relearn and document complex systems while advancing disaster recovery and business continuity. They then explain tabletop exercises as structured incident-response drills designed to uncover gaps in process and documentation, sharing scenario ideas like data deletion, token leaks, cloud outages, ransomware, and internal actors, including an embezzlement case where the perpetrator joined incident response. The conversation touches on an alleged AI sandbox escape involving Hugging Face, then debates claims that "unit tests are dead," arguing AI makes end-to-end testing easier but unit, integration, contract, and mutation testing remain important—especially for edge cases and dependency failures—and that agents can now generate extensive test coverage. 00:00 Cold Open Banter 00:35 Light Bulb Struggles 01:12 TaskRabbit Region Move 05:23 Tabletop Exercises Explained 10:06 Designing Scenarios 13:37 Insider Threat Tabletop 16:51 AI Sandbox Escape Story 21:19 Unit Tests Are Dead 25:16 Why E2E Misses Edge Cases 28:30 Mutation Testing Explained 31:25 Integration Tests Basics 33:23 Contract and API Testing 38:27 Third Party Outage Example 41:55 Golden Age of Validation 43:59 Wrap Up and Credits Resources:
- How Many CTOs Pod: https://howmanyctospod.com
- Scott Porad: https://www.linkedin.com/in/scottporad/
- Brad Hefta-Gaub: https://www.linkedin.com/in/bradheftagaub/