S3E6: AI Tech Debt

Nick and Chris talk about the security, data privacy, and operational challenges of enterprise AI adoption, highlighting how executive pressure driven by “Fortune front page FOMO” often leads to hasty deployments. Drawing from a presentation by ARM’s Richard Grisenthwaite on trustable AI, they emphasize that the majority of AI security relies on foundational security practices that organizations should already have in place. They explore the critical concept of the “blast radius” and the “cost of wrong,” illustrating how tools like ChatGPT Enterprise and Model Context Protocol (MCP) agents can inadvertently grant models excessive, godlike access to sensitive workspaces and shared drives, bypassing principles of least privilege. They draw parallels to historical trials with the Google Search Appliance and early web service API sprawl, showing how unchecked connectivity creates massive security liabilities if data flows and roles are not rigorously mapped and monitored.

To mitigate these risks, the hosts propose a tiered compute model that prioritizes local LLMs and private data centers for processing and classifying unstructured data, keeping the vast majority of workloads off frontier lab servers. They also discuss auditing enterprise ChatGPT compliance APIs to manage retention and risk. Shifting to software engineering, Nick and Chris reflect on DevOps Research and Assessment (DORA) findings, warning against “development team phrenology” and noting that while AI coding assistants boost individual productivity, they can degrade overall team efficiency by generating complex, “sloppy pull requests” that lack proper test coverage. Ultimately, they view these AI tools not as magical solutions to legacy systems, such as aging CICS COBOL code, but as evolving utilities that require structured platform engineering and automated testing to prevent accumulating even more technical debt.