Arrested DevOps

1. Modern Backups with Gilles Chehade

   11 de noviembre de 202630 minEpisodio 211A continuación

   Nobody Talks About Restoring Matty opens by admitting that, as someone who spent decades as a sysadmin, the topic drags up Arcserve and tape libraries, and a third-shift computer operator who would call in the …

   Notas del episodio

   Nobody Talks About Restoring

   Matty opens by admitting that, as someone who spent decades as a sysadmin, the topic drags up Arcserve and tape libraries, and a third-shift computer operator who would call in the middle of the night to say the backup failed and ask what to do. "The same thing you do every time. Go and rerun the job." The operator, Matty figures, was bored and wanted to talk to somebody. Gilles started doing backups about 30 years ago as a student with sysadmin gigs on the side, and says nobody called him until a machine had died. His diagnosis hasn't changed: "Most people think of backups as a one-shot thing you do, and you forget about it until you are actually in a very stressful situation." Matty puts the same point from the other direction. Everybody talks about backups. "Nobody ever talks about restoring the data."

   Terabytes Are the New Low Unit

   Gilles describes the old landscape: dedicated servers in a data center, hardware you knew existed for real, a tar copied to disk or magnetic tape, and data measured in gigabytes or hundreds of gigabytes. Virtualization then brought compute that spawns and despawns on demand, so you can no longer assume the thing you back up today will exist tomorrow. Now "terabytes is the new low unit," some companies are at petabyte scale, and some old habits turn into physical limits. Even with the network and the hardware, you can't finish inside the window, so daily backups become a problem. The old tools survive anyway, in Gilles's telling, because backups are "absolutely not a sexy area," and whoever gets handed the job wants to be rid of it as fast as possible. Matty adds that file-level thinking was wrong even decades ago, and he reaches for Microsoft Exchange: backing up the whole data store didn't help when a user deleted their own mailbox.

   Disaster Recovery Versus Rewinding Time

   Matty splits the reasons to restore into two. One is disaster recovery: "What would we do if our main data center in Chicago went south and we had to restore service to a data center in Columbus, Ohio?" His team ran drills and pretended. The other is putting data back because of something people did, which he calls "rewind time." Gilles has the user-level version from his student days, working for small companies where the sysadmin was the only techie and gave access to whoever asked. Including a non-technical director, who typed a command he shouldn't have and wiped an entire set of data. Gilles was the one who came in to restore it.

   The Agent Is Already Inside

   Gilles lists what changed since then. Ransomware was a curiosity when he was younger, when cryptography first got plugged into viruses, and now he says a company in the French news gets locked up every few months. On top of that, AI lets people administer infrastructure without understanding the pieces, delegating a lot of tokens and letting agents do far more than they should. He brings up the head of AI security at Facebook posting publicly that her mailbox was wiped while she was at lunch, because an agent decided to delete the data. The threat model flips. Before, you stacked layers to raise the cost for an outside attacker. Now the agent runs inside, so you have to guard against it as you would a user deleting your data, which is "a very, very different threat model." Admins he talked to recently told him a heatwave caused a lot of disks to malfunction, and he has been contacted by people with data in "data centers that were struck by missiles." He calls it "Interesting, nonetheless."

   Matty notes that an agent dropping a production table is not something backups solve, and that the next line of those stories is usually "I didn't have it backed up because it was a POC." He calls getting teams to productionize their apps an episode of its own. Gilles agrees backups aren't the fix, but says they have to be reasoned about differently. The odds used to be low if you had enough redundancy and a high enough bar for attackers. Now it's not "I have low odds that it's going to happen." It's "It's going to happen."

   Resilience as Code

   Matty reassures everyone that the show isn't here to tell them to retire to Montana and raise chickens, and Gilles offers two ways to think about the problem. The first is categorization. Most teams back up machines at one uniform level, which drives cost up and means the sensitive data can't be backed up as often as it should be. He suggests sitting down with an inventory of every data asset and putting a weight on each. Losing an hour of some data is fine, and other data can't lose ten minutes. The second is that infrastructure and DevOps already learned to run a reconciliation loop: an inventory, a statement of what should exist, and a fix when reality drifts. Data protection usually has a cron job and a report a human reads. Restoring is worse. Validating a restore costs far more than taking the backup, and when Gilles asks meetup rooms how many people check their backups, "two hands raised in the room" is a good turnout.

   His model is a standard for what he calls resilience as code. The inventory driving backups should be the same one that spawns the machines, whether that's an Ansible inventory, a scan of Kubernetes PVCs, or the resources in an AWS account. A second, hand-written list drifts, and the result is "the wrong inventory being used at the worst time." On top of the inventory sit policies for what gets backed up, how often, and how long it's kept, then a loop comparing reality against those policies, with a fix that is either automated or a human. In one of his talks, that loop was wired to GitHub CI so the build breaks when expectation and reality disagree. Matty, a believer in infrastructure as code and policy as code, says backups are still "too much click next." He tracks his lifting in Liftosaur, an app that treats workouts as code, and Gilles asks if that means that when you skip a workout, "you break a CI." Matty files it as a feature request.

   Plakar, From Home Directories to Control Plane

   Plakar began as Gilles's own open source project. Every time he changed machines he copied the old home directory onto the new one, ending up with nested copies of the same data, which he backed up to two or three places and paid providers to store again and again. After building three or four tools for himself, he released Plakar, which compresses, encrypts end to end, and deduplicates at the source, so he can use a provider like AWS for storage without trusting it with his data. He didn't want every dataset forced to look like a file system, so Plakar has an abstract representation and connectors, about 30 of them, covering the file system, S3, databases, and more. "I can back up my Kindle on it." It's under the ISC license, which he likens to BSD, with a public SDK, and "You can actually build your business on top of it."

   The second project is a proprietary control plane, the only closed-source piece. It exists for larger organizations, where some users can run backups, others can't even look at them, and both need to verify that a backup happened. It is API-first, with the UI as just one client, so it plugs into an Ansible collection, a Terraform provider, or a Kubernetes operator. CI can ask whether a resource has a policy, whether a job covered it, and flip lamps green or red. There's a free plan for up to one terabyte of processed data. Matty scans the connector list for Microsoft Exchange and doesn't find it, and Gilles points out they do have one for Active Directory. Matty says nobody runs Exchange anymore, then admits one of his most-read posts is a nearly 20-year-old walkthrough of a SharePoint Server 2003 one-way trust, and that people still reading it makes him sad for them.

   An S3 Bucket Is Not a Backup

   Gilles's closing advice is to have backups whether or not you ever touch Plakar. He has horror stories, including a company that closed because it had no copy of its most critical data, and says it has happened to a lot of people around him. The cloud-era assumption that providers store your backups, that an S3 bucket is one, is wrong, and losing something there isn't unheard of. There are so many users that the stats look like it happens to no one, until it happens to you. The providers say so themselves: "We are not going to back up your data. Shared responsibility." His exercise is to look at your data, down to the pictures in Google Photos or a personal bucket, and ask what happens if you lose it and how depressed you'd be. That's enough to know you want your own copies, even if you never use them.

   Matty adds that technologists are as exposed as anyone. Someone he knows keeps her photos in iCloud, which feels like a backup but is closer to a point-in-time copy, and a photo deleted three weeks ago may be gone. He wraps by saying the conversation maybe scared him a little, but that's a good thing. Gilles: "I did my job then."

   Links to Resources Mentioned

   • Plakar, the open backup platform Gilles describes
   • Plakar on GitHub, the open source repo
   • Plakar control plane docs, for the proprietary control plane Gilles describes
   • Liftosaur, the workouts-as-code app Matty mentions

2. AI and Incidents with Sylvain Kalache

   28 de octubre de 202633 minEpisodio 210A continuación

   What an AI SRE Actually Does Sylvain Kalache runs the AI lab at Rootly, and he starts by explaining the two kinds of memory behind the tools the industry has settled on calling AI SRE. The long-term one is a …

   Notas del episodio

   What an AI SRE Actually Does

   Sylvain Kalache runs the AI lab at Rootly, and he starts by explaining the two kinds of memory behind the tools the industry has settled on calling AI SRE. The long-term one is a knowledge graph built from your code base, past incident reports, knowledge base, and Slack channels, the machine version of the instinct a veteran gets when something breaks and they think "I think I know where it is." The live one is whatever just happened: recent deploys, telemetry, traces. From the two, the tool forms hypotheses and starts investigating. Sylvain's claim is that the unglamorous part, collecting the data, is already "a huge time saver," since the tool can chase ten hypotheses at once and put what it finds on one dashboard.

   Matty has been waiting for this for a while. Years ago he talked with Damon Edwards, Rundeck's founder, about having a robot run the same first-look checks while the responder is still getting out of bed. That never needed AI. What AI adds is the guessing: it doesn't have to be told which logs to pull for this particular shape of incident.

   Easy Incidents, Jagged Results

   For toil-level incidents, Sylvain says the tools get close to a hundred percent accuracy on sev 3s and sev 2s. Past that it gets "jagged." His best example comes from an AI and reliability conference in Berlin, where an Anthropic reliability engineer described an outage in which a request carrying 22 images crashed the service. Claude didn't stop at the bug. It noticed the same request coming from 4,000 accounts, concluded this was an attack and not just an outage, and suggested contacting the safety team about abuse. A human patching the bug, Sylvain notes, would likely never have looked past it.

   The same talk had the opposite case: the model grabbing a red herring, a cache hit rate graph that told a very persuasive story and turned out to have nothing to do with the problem. Matty points out that humans fall for this constantly, which is a large part of why incident commanders exist.

   Could Claude Be the Incident Commander?

   Matty admits his skin shivers a little when he says it, but he floats the question anyway. Everything you have to watch out for with an LLM responder is what a good incident commander already does for human ones: sitting there asking "is that what we need to do right now to get service restored?" So either side of the role might work with the right setup. What he finds dangerous is the person who says "it can do every damn thing" if you just write the right agent description.

   He also brings the PagerDuty-era version of this argument. Back when it was machine learning and not LLMs, the pitch was to train on all your incident data so it knows what to do. He and a colleague there kept running into two problems. Organizations don't have enough data, and the cost of a wrong answer matters: a bad Netflix recommendation wastes an episode of your time, a bad restaurant recommendation costs you hundreds of dollars, and a model that decides something probably wasn't a sev 1 when it was has the same problem. Sylvain agrees, and says that among Rootly's customers using the AI SRE, "the human is always involved." Nobody he sees runs it blind.

   Children of Magenta

   The bigger risk, Sylvain argues, is what happens to the people. If the tool takes every easy incident, you lose the small ones responders use to practice, to understand how their systems work, and to build the instinct they'll need when the huge one arrives and the AI can't solve it. He points to aviation, where American Airlines pilots in the 1990s coined "children of magenta," after the magenta line on the screens they were watching. Autopilot tends to be engaged in exactly the situations it can't handle, so the industry's fix was required simulator training every year for emergencies a pilot may never see.

   Matty piles on from his "Fight, Flight, or Freeze" talk: practicing in a calm moment is how procedures get into muscle memory for the 3am version. He quotes Rein Heinrichs, "If you're bad at having incidents, have you tried having more of them?" He also notes that better systems and bigger on-call rotations mean you can go months without a page, then wake up unable to remember how to log into the tool. Game days are the obvious answer, and the obvious problem has always been that someone has to build and inject a realistic failure. LLMs could do that part. Rootly has already gone one step further with Rootly Academy, built with Uptime Labs, where a human plays incident commander and directs a cast of LLM personas playing the responders.

   Watching Is Not Doing

   Sylvain's prediction is that the industry will push this too far, get bitten, and pull back. Incidents will get shorter on average, but he expects bigger major incidents that run for hours or days, because the engineers have lost touch with their systems. He spent half a decade running a software engineering school on the bet that people learn by doing instead of listening, and its students got hired at Facebook, LinkedIn, Meta, Google, and Nvidia. Applied to incidents: "just watching it is not the same as doing it." There are two ways this goes, he says. Either the models get so good that the lost knowledge doesn't matter, or there's a reckoning. "I don't have an answer to this."

   Matty adds two things. Product owners make excellent incident commanders partly because the experience makes them care about reliability, and that visceral exposure goes missing when the entire story is that Claude fixed it overnight. And if your coding agents don't get the context of the incidents that happened, you're skipping the same feedback loop you'd expect human developers to get from postmortems, which were already write-only before LLMs showed up.

   More Incidents, Less Help

   Sylvain brings data. Looking at the average number of incidents per Rootly customer, 2026 is running at three times 2023. "So far it's not becoming easier, it's becoming harder." Responders are also getting less backup: the old move of tapping the developer who wrote the thing on the shoulder now gets you "Hey, like, sorry, I didn't write it. I just prompted my agent." His summary of the on-call job today is more incidents with less help, and the trend could go either way.

   Matty ties it to the humane ops work from his PagerDuty days, when burnout and responder health were a bigger part of the conversation. One CTO or CIO he talked to answered concerns about on-call load with "that's why I pay him so much money." Sylvain's response, in full: "It's dumb."

   On-Call Health and Etsy's Sleep Tracker

   Rootly's AI lab built a methodology to catch overworked responders, and it combines two kinds of data. The observed kind is how many incidents someone handled, whether they worked nights or off hours, how severe the incidents were, and, newly added, how many tokens they're using, since "some people are token maxing and actually getting addicted." The self-reported kind borrows from medicine: just ask "how do you feel?", the way an Apple Watch does, and track the trend. It isn't a diagnostic tool, Sylvain stresses, but it gives you a signal. They released it as an open source project called On-Call Health, and he says large organizations are already using it.

   Matty goes digging mid-conversation and finds the ancestor: Etsy's Ops Weekly, from the days when Etsy was a monitoring company that also sold tea cozies. Its GitHub repo was archived three years ago and the last commit is nine or ten years old, but it came with a sleep tracker and a metric called MTTS, mean time to sleep. Matty says he can't describe how good it makes him feel to see the idea carried forward, and he lobbies Sylvain on the spot to come back for an entire episode on caring for the humans on call. Sylvain: "I'm down."

   Matty also points back to a few earlier episodes on the same ground: Incident Retrospectives, Incidents and Accidents, Cognitive Neuroscience, and Let's Be Careful Out There.

   Links to Resources Mentioned

   • Sylvain's post on engineers losing touch with their systems
   • Sylvain's post on whether Claude can fix itself
   • Rootly Academy
   • SRE Skills Bench, Rootly's open source benchmark for LLMs on SRE-type skills
   • On-Call Health on GitHub and the hosted version
   • Etsy's Ops Weekly
   • How Do You Infect Your Organization With Humane Ops?, Matty's talk
   • Fight, Flight, or Freeze: Releasing Organizational Trauma, Matty's talk

3. Meaningful Control with Jacquie Capur

   14 de octubre de 202641 minEpisodio 209A continuación

   Matty and Jacquie open with the caricature below, drawn at a vendor's caricature station at the first post-pandemic KubeCon. It's supposed to show the two of them fighting, since she was at Hashi and he was at …

   Notas del episodio

   Matty and Jacquie open with the caricature below, drawn at a vendor's caricature station at the first post-pandemic KubeCon. It's supposed to show the two of them fighting, since she was at Hashi and he was at Pulumi at the time. Neither of them works at those places now, so Matty figures they can be better friends.

   The Holy Shit Moment and the Albatross

   Jacquie's definition of meaningful control has three parts: you understand the dependencies you've created and what they're tied to, you can verify that they meet your needs consistently, and you have a realistic way to change course. "How easily can we pivot if we need to, and how clear are we on what it does?" Her own first magic moment was configuration management, back when she was an intern and three people were SSHing into a few hundred VMs to run four to six commands on each, except sometimes it was seven. "I can just do this thing one place, and it happens everywhere."

   Matty has the same origin story with a Windows vulnerability (maybe SQL Slammer, he doesn't remember): multiple days across three shifts in the data center, physically logged into servers over KVM, with a different login for each of something like 250 Windows NT domains. Then patch management tools showed up and you could just group machines and tell them to go. The trouble is what comes after the magic. The tool becomes load-bearing, then an albatross, because its opinions are now baked into how you work. It was great because the competition was Do Nothing Incorporated.

   Everything Changes Every Two Weeks

   The new twist is speed. Matty describes starting the year by building a plugin and skills for his team, then watching people adopt it at wildly different rates, some still working up to opening ChatGPT. When the way of working is defined by the tool, and the tool changes every two weeks, "it all becomes yak shaves." He wonders how anyone preps a conference talk three months out when it will be stale by the time they give it. Jacquie says DevRel has the same problem: every release means being expected to be an expert on it, with demos, within 48 hours.

   Her description of the process is a pipeline from discovery to something dependable: try it, break it, break it again, then ask whether it works three times in a row or once out of ten, and where the guardrails go. She also names the social side. Everyone's posting highlight reels, so "you're comparing your own friction points with everyone else's highlights," and there are too many launches to try them all the day they land. Python to Go to Rust felt fast at the time, she says, but "it objectively wasn't."

   Matty's favorite counterexample is a Chef meetup in Austin, where a presenter explained how migrating from Chef to Ansible proved Ansible was better. Matty, who worked at Chef, sat there thinking: "No, you're just smarter now." He gets the same feeling when people insist a different model should do the code review so it isn't reviewing its own work. The analogy to humans breaks down, since Opus is not a person.

   Abstraction All the Way Down

   Matty points out this isn't purely an AI problem, since every layer of abstraction removes some understanding of what's underneath it. He remembers a Gartner session from the early days of cloud predicting that the sysadmin's job would become vendor manager. You don't know Apache's internals the way you used to either, because you run the module that encodes best practice. Now Claude Code can write a whole website in a JavaScript framework he doesn't understand, and he suspects the people who build reliable systems this way are the ones who already had first principles.

   The counterargument is the photographers who never learned in a darkroom. Photoshop's metaphors came from that world, dodge and burn included, and a whole generation learned it another way anyway. The difference, Matty says, is that photography didn't move as fast as this.

   Meetups With No Slides

   Jacquie remembers a more vibrant meetup and conference scene a decade ago, when she could go to a DevOps one, a DevOps Enterprise one, and a Polyhack one, her favorite and now gone. They were places to scratch beneath the surface and hear what actually came out of a project, not a highlight reel. She misses that kind of under-the-hood writing too, citing the Netflix chaos engineering blog.

   Matty thinks meetups drifted into mini conferences and mini webinars, and that funding now tends to mean vendors who want to talk about their product. His version of the original: the first Chef meetup in Chicago, seven or eight people at a bar, "nobody had slides, nobody had a mic." The Chef Community Summit was two to three days of open spaces and no presentations, and he doesn't think you could fund one today.

   They also talk about the cost of being visible. Matty did a livestream at Pulumi where he did something dumb with Docker, someone politely pointed it out, and he spent the rest of the day feeling like he should have known. Jacquie has done upwards of 40 livestreams of learning in public and still runs into the same pull, because after 11 years people expect her to have answers. Her reasoning for doing it anyway: "If I can't do it, how do I expect anyone else to?" Matty adds the economy as a factor, since being on the record as imperfect feels riskier when jobs are scarce. Her line for the rest of it: "No one cares as much as you think they will."

   Data Residency Made It Real

   When Matty asks when meaningful control clicked for her, Jacquie goes to a healthcare job in Canada. Data residency rules meant they used AWS but not S3, which she says is a global service that could replicate to any other region. Control meant answering where the data goes, who can access it, how it's retained or used, and "how do we know what we're using is what we think we're using?" Those questions led her to a lot of Kubernetes, to learning the CAP theorem, and eventually to building their own managed database services in Kubernetes. Later, some of the industry tools grew data residency features of their own, and the question became how to make it easy to pivot out of what they'd built.

   The other half of control is who supports the tool. Jacquie points to the ingress-nginx retirement as the kind of thing happening more often: tools that were stable for years no longer being stable. She ties it to the economy, since open source has less funding and everyone has less time to give away, and the floodgates of PRs make communities harder to get involved in. She mentions Linus Torvalds gatekeeping which AI commits are allowed into Linux, and asks the same list of questions about every new tool: the maintainers, the documentation writers, the community knowledge base, the sponsors, the investors. "I don't have any answers here so much as it's just things that we look at."

   Make the Right Way the Easy Way

   Matty says the thing that makes all of this hard is that the boring parts, the plumbing, are what make the magic dependable, and there's no time to bake those practices in while the sand is shifting. AI-generated code is quick to make, but who maintains it? His internal marketing tool is the most heavily operationalized AI-built tool in his company, with ridiculous tests and CI and CD pipelines, only because he's been in ops for so long. The agent may not think of any of that, and neither will someone who doesn't live in that world.

   Jacquie says Terraform is the apt comparison. Plans can run to 70,000 lines of changes, so her teams flagged every delete for close human eyes, and she thinks the open question is what the deletes are in the agent world. Matty's skills and Agent.md files are built around what could go wrong, which he thinks isn't the normal instinct. He also worries that the guardrails he writes are all Claude-specific, so they go away if the tool does. He wrote up the practical version in a post about a designer on his team who can ship to production without knowing what CD means.

   His answer comes from the book Switch: make the right way the easy way. The story he always tells from it is the machine with a blade that kept cutting people's hands. Instead of signs and safety training, the plant redesigned it so you needed both hands to turn it on. If you have to actively opt into the safety, "it doesn't happen."

   Open Models and the TiVo Problem

   Jacquie thinks the questions are familiar from ten years ago: multi-cloud, disaster recovery, vendor lock-in, being able to take a backup and pivot. With proprietary models that can change pricing, push you out, or change how everything works, she expects a move toward open models, open weights you can adjust yourself, and small models trained to do one thing at a time.

   Matty pushes back on the friction. People use the central services because they're easier, and he'd happily buy an appliance for this, in the spirit of buying a TiVo because "I just wanna record the Cubs game." Jacquie hadn't planned to bring it up (it's literally not in her notes), but she works at Nebius, where one of the products is Token Factory, a set of open models the company has done its own accuracy and training work on. You switch between them by changing the endpoint. She runs them through Hermes and OpenCode for things like video editing and making video games for fun, and her sign-off is the Nebius builder program, with credits, and an online hackathon with up to $50,000 in prizes that runs through the end of October. Matty promises to ship the episode before then.

   Wrath Babies

   The conversation ends with a long tangent on World of Warcraft. Jacquie was a competitive player who paid her rent doing it, and started in Wrath of the Lich King, when veterans complained about "all these Wrath babies." Matty uses it to make a point about DevOps: when the movement was three years old, being there for one year wasn't much of it, but when it's twenty years old and you missed only the first two, that's almost all of it. Jacquie jokes about a follow-up called "What playing video games competitively taught me about working in tech," and Matty suggests a joint episode with Kat Cosgrove, who didn't play competitively but sure plays a lot of damn video games.

   Matty also points back to his recent episode with Marino Wijay, WTF Is Going On, for more on keeping up with all of this.

   Links to Resources Mentioned

   • Ingress NGINX Retirement: What You Need to Know, the Kubernetes blog announcement
   • AI Coding Assistants, the Linux kernel's policy on AI-assisted contributions, the closest public document to the Linus Torvalds gatekeeping Jacquie mentions (she doesn't cite a specific source)
   • How My Coworker Who Didn't Know 'cd' Shipped to Production, Matty's post
   • Switch: How to Change Things When Change Is Hard
   • Nebius builder program, with credits and the online hackathon Jacquie mentions

4. WTF Is Going On with Marino Wijay

   5 de octubre de 202636 minEpisodio 208A continuación

   No Topic, On Purpose Matty and Marino Wijay went into this one without a plan, and the plan that emerged was the honest one: what the hell is actually going on in tech right now. Both have been around long …

   Notas del episodio

   No Topic, On Purpose

   Matty and Marino Wijay went into this one without a plan, and the plan that emerged was the honest one: what the hell is actually going on in tech right now. Both have been around long enough to remember big shifts before (DevOps itself, the rise of Kubernetes and cloud native) but neither can remember change compounding this fast. Marino traces it to a specific hinge point: a predictable decade of innovation through the 2010s, then 2021 hits, zero interest rate money floods in, "you were limitless at that point," and a wave of startups and advocates get built on cheap capital. When rates rise, those ideas fizzle, the people promoting them lose their jobs, and the industry lurches toward the next hot thing. "Something called Loops was a huge thing two weeks ago, and now no one talks about it."

   Three Layers Become Two

   Marino draws on his data center days, back when a SAN install or rack expansion needed three distinct layers of expertise: contractors doing the physical racking and cabling, a team doing the finer cabling and bring-up, and a business layer figuring out what the infrastructure was actually for. "Those three layers, interestingly enough, have collapsed into two layers now." Physical labor gets automated away like it did in manufacturing, and what's left is a single human-plus-AI layer making the calls. The specialties that used to sit between those layers, especially networking, don't disappear so much as go invisible: "No one really talks about networking as much anymore, despite the fact that every single one of these models is running on hardware that's running on high-performance computing networking." The unsexy, boring infrastructure work is where the real money quietly lives now.

   Every App Is the Same Paint Swatch

   That collapse pushes everyone toward the same shortcut: "I need to just build, build, build with AI," and the result is that every AI-built product looks identical. Marino's line for it: open enough of these sites and "I'm looking at the same set of swatches," because the models are all referencing the same patterns. It's a preference problem as much as a technology one. Everyone wants the instant gratification of shipping a result, and nobody wants to be the one still doing the boring, careful thing while a competitor ships fast, so the shortcut becomes the default.

   Tech as the World's Longest-Running Accounting Hack

   Marino's most unfiltered riff of the episode: a lot of what looks like innovation is actually VCs hedging bets across five companies solving the same non-problem, and a tax and accounting structure that rewards hiring tech people and building software regardless of whether anyone needed it. "Tech has been the largest accounting hack ever." Data centers get sited for cheap water and power with real cost to the communities around them, models get bigger because bigger is fundable, and somewhere in the OpenAI-NVIDIA-Oracle circularity, "we're all in this endless cycle of tax evasion, tax accounting, tax hacks." His read isn't purely cynical, though: underneath the noise there's real opportunity in the unglamorous infrastructure work nobody wants to do, if you're willing to look past the hype for it.

   You're Not Talking to the Whole Industry

   Matty pushes back with a story from his PagerDuty days, arguing with a co-founder convinced that manual, dedicated bug-fix teams were extinct: "Alex, you talk to people who know what PagerDuty is in the first place and decided they wanna talk to you." A week at a manufacturing trade show reinforced the point for both of them: plenty of shops, tech and non-tech alike, are quietly not participating in any of this, "we still are building shit, and we do it the way we do." Whatever conclusion you're drawing about AI adoption from your own timeline and network is a sample of one bubble, not the industry.

   Finding Signal in the Noise

   Closing out, Marino's advice for keeping up: lean on the people who've kept their credibility instead of trading it for sponsorship money, because those are the ones who'll actually tell you when something's overhyped or underhyped. He points to Keith Townsend as an example of someone pragmatic who gets his hands dirty before rendering a verdict. Matty ties it back to a conversation he had with Emily Freeman after his own layoff: the useful answer usually sits between "I will not touch a single thing that uses AI" and "the only way you should communicate is through the agent."

   If you want more on how to actually keep learning in the middle of all this, Matty points back to three earlier episodes: Learning to Learn, Learning Stuff, and Managing Your Mental Stack.

5. CI/CD as Control System with Naga Sujitha Vummaneni and Sundeep Bobba

   8 de septiembre de 202632 minEpisodio 207A continuación

   Control Theory Had a Name for This All Along Naga Sujitha Vummaneni and Sundeep Bobba co-authored CI/CD as a Control System, which takes Jez Humble and Dave Farley's Continuous Delivery, now pushing 20 years …

   Notas del episodio

   Control Theory Had a Name for This All Along

   Naga Sujitha Vummaneni and Sundeep Bobba co-authored CI/CD as a Control System, which takes Jez Humble and Dave Farley's Continuous Delivery, now pushing 20 years old, and reframes it through control theory. As Sujitha puts it: "The pipelines are activators, observability is the feedback signal, policy is the constraint, and deployment strategy is how you regulate the risk." Once you see a CI/CD pipeline that way, a lot of what look like tooling problems turn out to be system-behavior problems, and most pipelines today are open loop: "They measure everything and on nothing." Sujitha traces the pattern back to a moment from the Arrested DevOps episode with Hannah Foxwell and Robert Warner: enterprise clients insisting continuous delivery would never work at their company, until it was just how everyone shipped. The premise of the book is that this framing was always available, but AI agents moving at machine speed finally make it urgent.

   Bonded Automation and the Four Questions

   Sundeep's answer to "should the agent handle this?" isn't a yes/no: it's a bounded operating envelope. Low-risk, reversible actions like restarting an unhealthy service or quarantining a known-bad artifact are fair game for automation. Changing security policy, touching production data, or anything with real customer blast radius pushes the threshold for human involvement way up. His framework is four questions: "How confident are we in the signal? What is the blast radius? Is the action reversible? And who owns the risk if the decision is wrong?" Sujitha adds the control-theory language underneath it: signal quality is itself a gate, because an automated rollback that fires on a noisy metric is worse than no automation at all, "you get flapping." Bounded blast radius, rate limits, and cooldowns exist so the system doesn't correct itself into a new outage.

   Writing "Always Run GitLeaks" in Your CLAUDE.md Doesn't Make It True

   Matty pushes on the gap between guardrails you write down and guardrails that actually run. Telling an agent to always scan for secrets before committing is no different than the developer who says "you're right, I should have run that" after skipping a check themselves, "it's the same trust-but-verify thing we've been doing forever, except now it's exacerbated." The book's answer is to stop treating this as purely a tooling problem: platform controls enforce the non-negotiables, prompts tell the agent what good behavior looks like, and runtime feedback tells you whether the controls are actually producing the outcome you expected. Sundeep frames it as fundamentally organizational: "Who owns the control, who can change it, what evidence proves it ran, and what happens when it fails, and who is allowed to accept an exception."

   The Approve Button You Click 200 Times a Day

   Sujitha's closing point reframes what a bypassed control actually means. "The bypass is a signal, not a violation. When engineers route around a control, the control was misdesigned. It might be in the wrong place, or too slow, or solving a problem they don't even have." She names the familiar shapes: the emergency-change process used for 40 percent of changes, the security scan everyone has a documented exception for, the staging environment nobody deploys to because it's never in a usable state, the approve button clicked 200 times a day without being read. Each one means leadership believes there's a control while the dashboard shows green and nobody actually knows what's happening behind it, control theory's version of losing observability of your own control layer.

   You Don't Need a Platform Team to Start

   For smaller, scrappier teams, both guests argue the model still applies, maybe more cleanly. Sujitha notes that small teams often close feedback loops faster because the control boundary and the team boundary line up. Sundeep's advice: "Start with one service or one delivery path and make the loop visible. Know what signals tell us the system is healthy, what decisions we're making from those signals, and what actions we can safely automate." Observability is foundational to all of it, since without actionable feedback there's nothing to close the loop with. The through-line for organizations of any size: "It's whether you have a closed loop of feedback, decisions, constraints, and action, rather than a pile of automation."

6. Industrial DevOps with Doug Pagnutti

   17 de agosto de 202642 minEpisodio 206A continuación

   The Wall Nobody Talks About Long before "DevOps" had a name, manufacturing plants had already split into two tribes: the automation and manufacturing engineers who built up their own scrappy plant-floor IT, and …

   Notas del episodio

   The Wall Nobody Talks About

   Long before "DevOps" had a name, manufacturing plants had already split into two tribes: the automation and manufacturing engineers who built up their own scrappy plant-floor IT, and the corporate IT department issuing laptops and running the domain a few buildings over. They grew up independently, with different tools and different goals, until, inevitably, they collided. IT wanted plant data flowing to corporate; OT wanted to reach out for data too. Add different incentive structures on top, and, as Matty points out, it's the same "wall of confusion" that gave rise to DevOps in the first place, just wearing safety glasses instead of a hoodie.

   "I Broke the Rules Just to Get It Going"

   Doug's clearest example: a robot on the plant floor goes down because of a network issue, and the routers are owned by IT: a help desk routed through Bogotá with zero context on an explosives manufacturing line. The ticket crawls through triage while production sits at zero and the CEO starts calling. Doug's fix was to log into the router console himself and get it running again. "Everyone's like, 'Ooh, yeah, you're a hero.' Like, no, I'm not a hero. I broke the rules just to get it going." It's shadow IT, industrial-plant edition: the AWS-credit-card moment of manufacturing.

   Converging, Whether Anyone Planned It or Not

   What actually helped, in Doug's experience, wasn't process; it was knowing people. "It's always gonna be personal relationships," he says, and his pitch to leadership has been simple: IT needs someone who's had OT experience, and OT needs a rep who understands IT's constraints, because the requirements on both sides have already converged. Neither side can keep pretending plant data stays on the plant floor; predictive maintenance and machine learning mean data has to leave the building, sent out to third parties for analysis, whether or not the old "peace treaty" between IT and OT accounted for it. That same shift is dragging OT into IT's security concerns for the first time, too. Doug is blunt that OT folks "just don't have a good concept of the security requirements," a gap he's only closed by talking directly to security engineers in IT.

   Manufacturing Already Solved This: In Manufacturing

   Matty pulls in The Phoenix Project and its inspiration, The Goal, both built on the theory of constraints and value-stream mapping from actual manufacturing. The irony: software loves borrowing metaphors from manufacturing (Andon cords, the Toyota Production System) while the people doing real manufacturing haven't necessarily absorbed the lessons about visibility and bottlenecks that DevOps eventually learned from them. Doug agrees the CAMS/CALMS framework (culture, automation, measurement, learning or lean, sharing) maps cleanly onto OT/IT, especially the parts industrial teams already live: lean practices are baked into manufacturing, but the value an OT change delivers is far easier to point to (100 widgets became 110) than the value of infrastructure work that only shows up when it's absent.

   Incentives Explain Almost Everything

   The conversation lands on a favorite Matty theme: people work to the incentives you give them, not the ones you intended. A mandate for developers to write ten tests a sprint produces assert(true). A cash bonus for testers finding bugs and developers fixing them produces a black market in planted bugs. And "litigating severity" on an incident call is really about whoever's compensation is tied to P1 counts. Matty points to the STELLA Report as the same pattern in postmortems generally: after a well-handled incident, it's easy to reason "nothing bad happened, so you shouldn't have shut it down," the same Y2K cognitive dissonance in miniature. Doug's closing advice for anyone straddling the OT/IT divide is almost embarrassingly simple: find out the name of your IT person. Treat them like a teammate with real expertise, not a request queue ("not AWS," as Matty puts it) and the rest gets a lot easier.

7. How AI is changing the SDLC with Hannah Foxwell and Robert Werner

   1 de octubre de 202540 minEpisodio 205A continuación

   The Trust Problem Returns Hannah Foxwell, who has spent over a decade in DevOps and platform engineering, draws a striking parallel to earlier transformations: "It used to be that testers didn't trust …

   Notas del episodio

   The Trust Problem Returns

   Hannah Foxwell, who has spent over a decade in DevOps and platform engineering, draws a striking parallel to earlier transformations: "It used to be that testers didn't trust developers and ops didn't trust testers and there were all these silos. Now we're putting AI agents in the mix. Can we trust them? Should we trust them?"

   This isn't just déjà vu—it's a fundamental challenge that resurfaces with every major shift in how we build software. As Robert Werner points out, management had to give up control and push trust to the edges of organizations during the agile transformation. With cloud adoption came self-service and automation. Now, with AI, we're dealing with non-deterministic black boxes that we need to trust to be "right often enough."

   The Fluency Gap

   One of the biggest challenges isn't the technology itself—it's the lack of shared understanding. Hannah launched "AI for the Rest of Us," a community now with over 1,000 members, after realizing that AI fluency is essential for making good decisions about where and how to use these tools.

   "I went to a talk at a conference thinking I'd learn about AI in one talk and become an expert by tomorrow," Hannah recalls. "It just didn't happen like that. There's a whole new domain with new vocabulary, new concepts, new techniques."

   The community focuses on making AI accessible without dumbing it down—providing talks and content that explain complex concepts in simple language so more people can participate in the conversation about AI's role in software development.

   The Speed-Responsibility Paradox

   The technology is evolving so rapidly that best practices barely have time to solidify before they're obsolete. Robert describes how hiring strategies at startups are changing every few weeks as new capabilities emerge. "Things that weren't feasible last week are suddenly possible," he notes.

   But this speed creates a dangerous tension. Organizations are pushing hard for AI adoption while the guardrails, workflows, and cultural practices needed to use it safely are still being figured out. As Matty observes, this leads to perverse incentives—developers required to "use AI" who find ways to tick the box without actually deriving value, just like teams that once added meaningless tests to meet sprint requirements.

   Who Owns the Code?

   A critical question emerges: if AI generates the code, who owns it? Who's responsible when something goes wrong?

   Hannah frames it in familiar DevOps terms: "Does anybody really want to own a service if they didn't write it and they don't understand how it works? It's the ops challenge again—AI throwing code over the wall to us."

   Robert's answer is pragmatic and honest: humans will need to take responsibility for validating AI-generated code, even if it's tedious work most developers won't enjoy. His company, Leap, is building tools specifically to make that verification process as convenient and enjoyable as possible, because he believes there's simply no other way to do it safely.

   The Documentation Double-Bind

   There's an ironic twist in how AI agents work best: they need excellent documentation. Organizations improving their documentation to support AI-powered development are inadvertently following DevOps best practices that benefit human developers too.

   But as Matty discovered building his own project, AI-generated documentation can be dangerously unreliable. The tools will confidently document features that don't exist, pulling from incomplete PRDs or speculative notes in the codebase. Great documentation trains better agents, but agents shouldn't write that documentation—creating a challenge that requires human judgment and oversight.

   Lessons from Past Transformations

   The parallels to earlier shifts are instructive. Hannah remembers enterprise clients who insisted continuous delivery would "never work here." Now it's common practice. The same resistance appeared with cloud adoption and agile methodologies.

   What worked then still matters now:

   • Guardrails enable freedom: Constraints and safety nets let people explore confidently
   • Make the right way the easy way: Transformations succeed when good practices are more convenient than bad ones
   • Community and shared learning: Success stories and failures shared openly help everyone navigate change faster
   • Start with good practices: Teams with solid engineering fundamentals—blue-green deployments, A/B testing, safe-to-fail production environments—are better positioned to benefit from AI-assisted development

   Practical Advice for Explorers

   For developers and teams trying to navigate this transformation, Hannah and Robert offer grounded guidance:

   Keep your eyes open: Watch for patterns of success and failure. Who's making this work, and what do they have in common?

   Build community: Find or create spaces where people can share honestly about what's working and what isn't, without the pressure to pretend everything's perfect.

   Be selective about information sources: With so much noise and hype, focus on quality outlets. Ignore things for a few weeks, and if they keep coming up, that's when to invest your time.

   Practice regularly: The technology evolves so fast that hands-on experience goes stale quickly. Even if it's not your main job, refresh your skills every few months.

   Be specific and constrained: AI coding assistants work best with clear, narrow requests. Frustration comes from asking too much or being too vague.

   The Future We're Building

   We're in the Nokia phone stage of AI-assisted development, as Robert puts it—the technology will look completely different in just a few years. But unlike waiting passively for that future to arrive, developers and teams are actively creating it through the choices they make today about how to integrate these tools.

   The question isn't whether AI will transform software development—it already is. The question is whether we'll learn from past transformations to build better practices, stronger safety nets, and more trustworthy systems. Or whether we'll repeat old mistakes at unprecedented speed.

   As Hannah emphasizes, having more people with AI fluency means better conversations and better decisions at a pivotal moment in history. The rollercoaster is moving whether we're ready or not. The best approach is to keep your eyes open, stay connected to community, and remain thoughtfully critical about what works and what doesn't.

   Learn more about Hannah's work at AI for the Rest of Us. Use code ADO20 for 20% off tickets to their London conference on October 15-16, 2025.

8. Digging Into Security with Kat Cosgrove

   25 de agosto de 202529 minEpisodio 204A continuación

   Matty talks with returning guest Kat Cosgrove, now head of developer advocacy at Minimus, about why security stays in the news, how to live with an endless stream of CVEs, and which tools belong in every …

   Notas del episodio

   Matty talks with returning guest Kat Cosgrove, now head of developer advocacy at Minimus, about why security stays in the news, how to live with an endless stream of CVEs, and which tools belong in every production environment. Kat's employer builds container images with very few vulnerabilities, which comes up in the tooling section.

   Worse Than Chain Emails

   Kat's opening point: "Security is like a never not hot topic," and the stakes have risen as more of life and government runs on computers. The worms and Trojans of the 1990s look charming next to Meltdown, Spectre and Heartbleed, and a chain email that crashed your computer has been replaced by something that can take down much of the planet, as with the CrowdStrike outage, which Kat notes was a bug and not a vulnerability. Matty goes back to the Morris worm of the 1980s, when about 100 computers were on the internet and sysops hopped on conference calls because they were the only people affected. Today the whole thing is held together with "baling wire and good intentions."

   Three Dependencies Deep

   Matty says that complexity makes it worse, since a vulnerable package may sit 17 layers below the thing you use, and brings up left-pad and the earlier Who Owns Your Availability episode. Kat uses the Equifax breach as the example: a wildly outdated version of Apache Struts, which anyone could imagine they'd never let happen, until it's three dependencies deep and nobody knows it's there. Kat has credit monitoring for life because of it. The breach is also how the industry got software bills of materials and a pile of security tooling.

   Matty notes that an SBOM only helps after the fact, while something like Dependabot tells you about known vulnerabilities in what you already have, which is a bit more proactive. Both agree teams pull in far more dependencies than they need.

   Most CVEs Stay Put

   Kat says companies spend an outrageous amount of time and money mitigating CVEs, sometimes with whole teams, and still only handle the very high and maybe high ones. If the fix costs more than the potential exploit is worth, it stays: "Most CVEs just get left chilling there." So anyone who thinks their software is secure is probably wrong, and nothing is immune. Kat cites a Kubernetes Ingress NGINX vulnerability scored 9.7 or 9.8, which affected 38 percent of cloud environments, and says it was not fun from a maintainer's perspective.

   Matty asks about the dependency of a dependency whose maintainer left five years ago. Kat's answer is to fork it and fix it: "It's open source, baby. PRs welcome," and if the fork is public you risk becoming the new person in the XKCD comic. Kat still thinks open source is the best and most secure way to build software, since flaws are found and fixed faster, though they're also exploited faster, and a community of dedicated nerds can outrun proprietary teams.

   What Belongs in the Kit

   Kat's list for anyone who should pause the episode and install something starts with observability and monitoring, such as OpenTelemetry or Prometheus, because "you cannot just be like raw-dogging production": you need to notice weird traffic from a weird location hitting a weird endpoint. Next is alerting smart enough that engineers aren't firehosed with nonsense, then dependency management, meaning you know your dependencies and theirs. Kat adds infrastructure as code, instead of clicking through the AWS console, since most DevOps tooling has a security angle, and an SBOM, which Kat admits everyone is tired of hearing about after roughly three years of KubeCon talks but which still matters.

   Kat's employer makes very small container images, most with zero CVEs, rebuilt automatically when an update lands, which gives an application a clean starting point, though it can't fix questionable code a team adds. Along the way Kat and Matty decide that raw-dogging alone wouldn't earn the explicit tag, and then Matty swears and it does.

   Touch Grass

   Kat says security and computers are a burnout factory and begs listeners to have a hobby that doesn't involve computers: "literally go outside and touch grass," especially anyone on the "mitigating CVEs treadmill," a much less fun cousin of the loot treadmill in games like Diablo. Matty recalls a friend who took up piano and kept it off social media for the same reason, and admits that all of Matty's own hobbies used to be reframings of work. Kat floats a miniature rage room with racks at conferences, like the therapy dogs and baby goats that have shown up at events, and both complain about status LEDs that can't be turned off on printers and routers. Kat's router lives in the bedroom, under a t-shirt.

9. AI, Ethics, and Empathy with Kat Morgan

   3 de junio de 202540 minEpisodio 203A continuación

   Matty talks with Kat Morgan about AI from the point of view of two people who use it daily and have mixed feelings about it: where it helps, where it's a mess, and how to work with it responsibly. Matty …

   Notas del episodio

   Matty talks with Kat Morgan about AI from the point of view of two people who use it daily and have mixed feelings about it: where it helps, where it's a mess, and how to work with it responsibly. Matty currently works in a marketing-adjacent role that involves some coding, at a company that makes Steampipe, and uses Cursor mostly for prototyping. The cold open is Kat, "very strongly opposed to abusing the robots."

   Nuance in a Minefield

   Matty frames the question by saying people are bad at nuance and asking someone's opinion on AI is like asking what they think about computers, since some people picture generative art and others picture assistive tools or agents. Kat lists the ethical threads that deserve attention: intellectual property and whether creators can keep a roof over their heads, accessibility and whether LLMs open the digital world to people who need it, and the ecological and academic impacts. Kat compares it to security and documentation, which end up being everyone's responsibility, and expects everyone to need enough AI knowledge to tell where the tools stop and where humans have to start. Even abstaining means gaining that awareness. Matty adds that the less educated we are the more likely we are to be steamrolled, and that for some uses "the juice is not worth the squeeze," like burning resources on a cute cartoon of a friend.

   Kat notes that any line drawn today can move tomorrow as the tools change, and that LLMs have changed how long a tech career seems feasible given a tendency toward carpal tunnel. Kat also points out that significant models already run on a MacBook, which Kat compares to the room-sized computer that became a pocket one.

   What Worked and What Didn't

   Matty uses ChatGPT for alt text on social media, which makes doing it well faster. With Cursor, Matty's good experiences come when Matty already knows what to build. One was a private website for friends to watch old videos, with S3 and signed URLs, in a well-trodden React setup where Matty had the architecture in mind and Cursor implemented it. The bad one was asking Cursor to write a Steampipe plugin for Bluesky: "oh boy was that terrible." Matty concludes vibe coding has not made Matty worried that engineers will go away.

   Kat says context is everything: about 30 to 50 percent of a context window goes to building context, including recent library versions, docs for new functions, and the project's structure and hygiene. Kat recently spent about 90 minutes on context and planning, with a task file where a markdown checkbox marks not started, in progress and done. When Kat told the agent to knock out the tasks, the work Kat expected to take four or five hours finished in a few prompts, leaving time to review the code line by line and check whether the first version was the user experience the team was after.

   One More Thing

   Both describe the rabbit hole: "just this function" becomes "just this entire feature." Matty ended up awake until 5:30 in the morning on a refactor of a podcast Hugo theme. Matty notes an upside, which is that an agent waits patiently with its context, so returning after two weeks costs nothing. Kat says burnout from layoffs and volatility has been real, and that Kat uses an LLM as an executive decision-making regulator, asking whether a tangent moves toward the milestone, which has helped with planning, estimating and sticking to a plan without difficult conversations with a manager.

   Treat It Like a Junior Colleague

   Matty compares working with agents to a senior engineer working with a junior, where mistakes are expected and the process of plan, develop, test, iterate, document and commit exists to catch them. Kat adds that LLMs were trained on GitHub and respond well to work framed as GitHub issues: have the agent write an issue, edit it to the real requirements, then tell it to pull the issue and work it. That demands good code hygiene, with modular code, documented interfaces and an easy data model so a context window stays coherent for a one to two hour pairing session. Kat also uses the GitHub MCP server to have the agent comment on issues when the plan pivots and to pick up from the issue history on Monday mornings, and thinks a service account would help distinguish what a person wrote from what the agent wrote. Matty keeps issues in solo repos and has one with over a hundred comments that are all Matty talking to Matty.

   Guardrails and Private Setups

   Kat is uncomfortable with data centers powered by gas turbines and plans to run a local setup, and cites a DevOpsDays Chicago talk by Paul Czarkowski, which wasn't recorded, about running open models locally on something like a Mac Mini. Kat doesn't want secrets in a context window, so secrets go in a file in the home directory that git commands can't reach, and the agent runs in a dev container, never directly on the host. Matty notes that an env file in the agent's workspace is exactly what it can read. "It's important to try and make it safe to make mistakes," Kat says.

   Matty's last caution is that an agent will troubleshoot things that aren't broken: it once tried to uninstall a plugin because it ran the wrong CLI command, and it will "fix" a function that was already fixed, since the context window is small. Kat says healthy skepticism is warranted because it messes up a lot and isn't replacing people.

   Be Polite to the Robots

   Matty says you should always be polite to agents, partly as a joke about who they'll remember and partly because being polite to them makes us more inclined to be polite to everyone. Kat's version: neural networks are loosely aligned with how our brains work, and reinforcing demeaning behavior in a chat reinforces it in us, making it harder to tell the difference between treating computers that way and treating people that way. "We actually have to respect our own presence enough to appreciate that what we put out in the world will also change ourselves."

10. Open Communities with Andrew Zigler

    1 de febrero de 202435 minEpisodio 202A continuación

    Matty talks with Andrew Zigler, developer advocate at Mattermost, about what it takes to run a developer community that is open first, and why it's harder than it sounds. Matty says neither of them knew the …

    Notas del episodio

    Matty talks with Andrew Zigler, developer advocate at Mattermost, about what it takes to run a developer community that is open first, and why it's harder than it sounds. Matty says neither of them knew the topic going in, then draws on time at PagerDuty, Chef and Pulumi. The cold open is Andrew: "being default open doesn't mean you have to sacrifice standards or sacrifice productivity or what you're aiming for as a company. It just means that you have to do the work to align people to it."

    The Echo Chamber

    Andrew says the first hurdle in working default open is the echo chamber feeling: you post things you'd normally keep in closed channels, and either the same few people respond or nobody does, and you can't tell whether the message is landing or being read passively. DevRel also ends up as a counterbalance between what the community thinks of the project and what is happening inside the company. Building trust takes a long time, and the rule is to listen and incorporate the feedback you ask for, because otherwise you lose that trust.

    The Gravity of the Company

    Matty adds that the company behind a project always has more gravity, since the people paid to work on it attend every roadmap meeting, while volunteers and even people at another company's open source program office can only look in now and then. Andrew agrees that the loudest and most contributory voices are usually paid staff. Andrew's answer is to give other contributors gradients of engagement and a way to validate their experience and reward them, whether with swag, a platform, shared stage time, or simply being heard. People on the edges who never see contributors like themselves celebrated don't find a way in, so part of the job is teaching people to contribute where they are.

    Matty compares Chef and Pulumi. At Chef, Matty says, "I work at Chef because I believe in Chef, not the other way around": employees were members of the community first, and community summits had staff and outsiders on equal footing. At Pulumi, when a community event was planned, it was hard to get employees to see that they were part of the community too and to join on a workday. Matty is careful to say it wasn't malicious, and that leadership has to reinforce it, since engineers with deliverables won't otherwise take the day. "No one plans for it to go bad," Andrew says, and adds that this is a universal experience in open source. Andrew suggests sharing the tools too, such as cloud workspaces that let outsiders build the software the way staff do, since internal build systems are where open projects quietly stop being open.

    What Can Stay Closed

    Matty raises the open roadmap problem: customers can't be named, and a feature may matter because one large customer wants it. Matty notes almost no company would give the community an equal say over the project it stewards. Andrew says "being default open doesn't mean that everything has to be open. It just means that you have to default to being open," and that it's fine to ask at the start whether something needs to be open. The key is to communicate clearly about what isn't, to keep trust. If community-built features make a product sticky in the enterprise, Andrew adds, some of what comes back should go to the community, for meetups, lightning talks and swag, "by sharing the spoils."

    Opening Up Advocacy

    Matty asks how to open source advocacy itself, since people whose title is developer advocate can't do all of it. Matty describes a friend who asked how engineering blogs work, and the pattern of nothing for three months and then 12 posts in a day, since engineers write when they can. Matty's fix is to have people whose job is content work with subject matter experts. Andrew loves an engineering blog run by engineers, since the community wants to hear from the React developer about a new design library, not executive marketing, and says it needs pairing with a marketing team to tie back to strategy.

    Andrew's core idea: "The biggest thing that you can do as a developer advocate is to create more developer advocates," meaning people without the title who have opportunities to engage. Open source communities are seasonal, since people contribute when they have free time, and "you can't control the waves, but you can control what the waves do when they wash up on the shore." Advocates should share strategy and plans, and write retrospectives about conferences, demos and talks, so contributors get excited and come along. Andrew has had contributors ask for feedback on their conference talks, which means trust has been earned and that advocacy created another advocate.

    Andrew's example is GitLab's DevRel team, who plan events in the open on GitLab with epics and milestones and post retros with photos, talks and contributors. A contributor in Brussels can see the team is coming in three months and sign up on the issue. Planning for sponsorships, collateral and demos tends to happen invisibly on Slack, Discord or Asana, and fixing that invisibility matters.

    Writing Programs and Contact Pages

    Matty says a contributor writing program is "so much work" that contacts at other companies who ran one mostly wished they hadn't, since it multiplies content production problems. It doesn't have to be a formal program: signal boost a post about your product, or invite the author to collaborate. Matty also pleads with bloggers to put a way to reach them on their blogs, because Pulumi wanted to syndicate community posts with canonical tags and couldn't find the authors, and some authors turn out to have moved on from the tool years ago, though the post is still good. Matty's own most visited page is an old post about SharePoint 2010 search in a one-way trust, written to document a fix for a team.

    Andrew ran a community writing program at Mattermost, which is rewarding for contributors, especially those in emerging tech hubs, and draining for staff, and can drift into a side project that doesn't connect back to company goals. Community members often assume the company won't notice their work, Andrew says, when in fact staff are right on top of anything that comes in and want to lift it up, and helping them get past that feeling builds confidence.

    Links

    • Matty's blog post about SharePoint (including broken images!)
    • Community Pulse podcast

11. Machine Learning Ops with Chelsea Troy

    18 de enero de 202448 minEpisodio 201A continuación

    Jessica Kerr talks with Chelsea Troy, a staff data engineer on the machine learning operations team at Mozilla, about what staff engineering actually involves, how MLOps differs from DevOps, and when to use …

    Notas del episodio

    Jessica Kerr talks with Chelsea Troy, a staff data engineer on the machine learning operations team at Mozilla, about what staff engineering actually involves, how MLOps differs from DevOps, and when to use machine learning at all. The team exists to help Mozilla's other teams get models into production, and Mozilla has around 750 people. The promised topic of surfing doesn't come up in the recording. The cold open is Chelsea: "all jobs, if you do them long enough, become either management or marketing or some combination of management and marketing."

    Every Job Becomes Management or Marketing

    Chelsea started out planning to stay on the senior engineering track forever and write code for a whole career, and has since concluded that the skills early-career developers dismiss as soft become nearly the entire job: coordinating within and across teams, making sure everyone knows a product exists and how to use it. The system turns out to include people, regulations and corporate bureaucracy as well as the repositories. It's a lesson people seem to have to reach themselves, like advice to a friend in a bad relationship, and Jessica's version is that you don't break up with your technical skills, you open the relationship. Chelsea adds that you don't get to build things unless people want them, at least not if you want to avoid shelfware.

    Knowledge Work Is Not an Assembly Line

    Chelsea says measuring engineers on productivity metrics inherited from an assembly line, where nails accumulate linearly through the day, is a disservice. Software is knowledge work: "We don't create value by doing the same thing over and over." It creates value by gathering context into an understanding of how to solve a problem now while keeping as many likely directions open, which Chelsea credits to Kent Beck's term optionality. From outside, that looks like nothing, nothing, nothing and then a big release, after months of understanding the problem, talking to users and socializing changes, since changes that torch people's context take power away from them. Writing the code comes last and is the easiest step. "We are treating lines of code like nails." Jessica adds that "our job is not what we do. It's what we know," and Chelsea says much of the day job is finding knowledge for someone and routing it to them, and that early in a career, how much code you can write depends on how good the decision makers about the repository are.

    Buying MLOps Tools

    Mozilla's values include data privacy and ethical use of machine learning, which are hard to prioritize when teams spend their energy on getting any model into production through individual heroic efforts. In the fall a team began evaluating products to give data scientists and machine learning engineers a turnkey path to production. Chelsea notes the product landscape is young, with upstarts of around 15 employees, so the larger asks sometimes can't be met and documentation doesn't always match behavior.

    Chelsea's advice on evaluating them is to separate optimizing metrics, where more is always better, from satisficing metrics with a good-enough threshold. Engineers tend to treat everything as an optimizing metric and end up in decision deadlock, for instance over scale, when a small startup isn't going to see a billion users at once. Jessica: "Problems you want to have." The optimizing metric that often isn't on the grid is the availability and flexibility of support. Chelsea fought to include it, and found that "the products we chose that have really, really responsive support teams are the products that we have managed to get into production at this point." A support engineer on Slack, for example at Weights and Biases, will take a custom chart problem and reproduce it in their own project. Free and open source tools the team liked ideologically didn't make the cut, since there's nobody whose job is making sure it works for the people using it, which Jessica sums up: "Wow, it's almost like the code isn't the whole thing." Jessica adds that at Honeycomb, good support gets renewals.

    How MLOps Differs from DevOps

    Chelsea came to MLOps from software engineering through data science, not through DevOps, and describes a data engineer as someone thrown into the model, the data science code or the app code as needed. Chelsea's view is that it isn't necessarily different in kind but that machine learning models add special considerations. Jessica compares deterministic program execution to the internal combustion engine, now one category among vehicle types. If the same input produces different output, that would worry a DevOps person and not necessarily an MLOps person, though there's a different decision tree, since some causes are fine and some are awful, and diagnosing a malfunctioning model is harder because the internals are automated.

    Chelsea tells of someone distraught that ChatGPT claimed to have run code and reported wrong output, and of spending too long explaining the mechanism when the question was really "how could ChatGPT do this to me?" The person wanted to imagine a world where the tool had an interpreter, and Chelsea's answer was that one can imagine that world, but it's not the one we're in. A software engineering background alone isn't enough to debug these systems, and tools built to operationalize deterministic code lack needed features.

    One such feature is checking that production data still matches the test data the model was evaluated on. Chelsea cites Andrew Ng's Machine Learning Yearning and a contrived example of a cat-photo model trained on high-quality images that then meets blurry phone photos. At Mozilla, Chelsea works on a system that sanitizes search data, discarding anything that might contain personal information. It drops anything with numerals unless it's on a human-made allow list, and uses spaCy's named entity recognizer to filter names. The team has to be sure the people using the feature are in the population the recognizer was trained and tested on, so it monitors an aggregate distribution of languages in search data, without storing the searches, to catch a shift before anything is stored long term.

    Chelsea says MLOps and DevOps share a focus on catchability. There are three risk amplifiers: catastrophicness, likelihood, and insidiousness, "how likely is it that this thing goes uncaught if it happens." Security, DevOps and MLOps focus on insidiousness more than other roles do.

    When to Use Machine Learning

    Chelsea's view is that the most effective systems are a series of steps with humans in the loop, not one generalized system doing everything. The example is who-to-follow recommendations on social media. A popularity-based model produces "the Beyoncé problem," which spirals up the already popular and doesn't connect niche audiences. A person can often tell when someone has gamed the engagement system, and human judgment is hard to automate and, Chelsea adds, hard to beat with a model, even though it's spotty at best. Better would be to break it into three simpler steps: find what topics people discuss, figure out who is knowledgeable on them, probably with a human in the loop, and recommend those people to people who want to learn. Chelsea notes that Follow Fridays and hashtags were user inventions, and that classical models on tabular data are often enough. Jessica's summary: for any problem where you'd reach for machine learning or generative AI, break it down into parts where a model helps, parts where a deterministic rule works, and low-volume, high-value parts where a real person should be asked.

    Chelsea writes at chelseatroy.com, linked below.

    Read more of Chelsea Troy's writing here!

12. It's Been Ten Years of ADO, Charlie Brown

    4 de enero de 20241 h 49 minEpisodio 200A continuación

    Joe, Matty, Trevor, Bridget and Jessica record the tenth anniversary episode, which is also episode 200, as a decade-end wrap-up. This is a second take, which Joe opens with "Here we go, take 2. Slower, more …

    Notas del episodio

    Joe, Matty, Trevor, Bridget and Jessica record the tenth anniversary episode, which is also episode 200, as a decade-end wrap-up. This is a second take, which Joe opens with "Here we go, take 2. Slower, more intense," and Matty notes that "it's not a year-end wrap-up, it's a decade-end wrap-up." The hosts cover how the show started, how each host joined, conference and live-recording stories, guest and download stats, and what each is up to now. The recording ends with a supercut of the show's cold opens, which the episode's existing link also points to. The cold open is Joe's take-two line.

    Sponsors and the First Sponsor Deal

    Matty tells the story of the show's first sponsor. PagerDuty reached out, and when Matty and Trevor didn't know what a sponsorship costs, they said $50 an episode, and Matty's one smart move was limiting it to a six-episode deal. Later Matty heard PagerDuty's ad on another podcast and learned at a conference bar that the other host had been paid $1,000 an episode for a similar audience. The renewal went to a number between the two. Years later, after Matty worked at PagerDuty, a sponsorship of the show drew lawyers who needed to be sure it wasn't double dipping. Matty adds that some sponsors come for a couple of episodes and some keep returning, and that the long back catalog helps.

    How It Started

    Arrested DevOps began as a blog Matty never wrote, named by a friend who is a writer. Matty had learned DevOps from podcasts like the Ship Show, DevOps Cafe and Food Fight, and wanted a show for people who didn't know who John Allspaw was, the ones whose boss "read about DevOps in the in-flight magazine." Matty met Trevor at an Azure meetup in Chicago and invited Trevor to co-host, since Matty knew operations and wanted someone with a developer background. The two then got on a Hangout with Nathan Harvey and a fellow podcaster for advice, where the banana stand tagline was coined. Matty also has a lost episode 0, a five-minute recording testing Hangouts. Jessica says the "banana pants" variant came from a first episode where a script-reading panic and a lot of Dora the Explorer produced "in the banana pants."

    The first episode with a cold open is 44. Joe picked them, and Joe's rule was: "If there's interesting profanity, that's the cold open." Matty also runs through the guest spiel: make peace with the explicit tag, stop and restart if you slip, and clap loudly if you blow an NDA, which has happened exactly once in ten years.

    How the Hosts Joined

    Bridget appeared first as a guest on an episode Matty remembers as being about conferences, whose art was a photo of Bridget's badges, published September 23, 2014, and Matty's checking turns up that the first episode Bridget hosted was November 18, 2014, DevOps in the Enterprise. They disagree about whether the invitation came at the DevOpsDays Chicago afterparty. Jessica had been on the show, liked it, and asked to join, after the two met at the first Redeploy conference, where Matty was a fan of Jessica's other podcast, Greater Than Code. Jeff Smith joined as an occasional host, and Joe, who started as the editor, also became a host, with a different editor before Joe. Since Joe went back to full-time work, Matty does the editing, which explains the quality, and the tail of each edit is worse because Matty got tired as an editor.

    Live Episodes and GOTO Chicago

    Bridget recalls curating a GOTO Chicago track as five live podcast panels in a row, with Matty, to an audience, in a breakout room with an air wall, and a guest loud enough that a proctor asked them to stop yelling. The first of them was Old Geeks Yell at Cloud with Andrew Clay Shafer and Bryan Cantrill. Bridget's takeaway: "If you're going to take on way too much, just have great guests. That'll cover a multitude of sins." Joe's tip for live episodes is to line up the AV crew ahead and pass a single mic, and says live episodes are easier to edit because fewer people talk over each other. The first live one may be eating sushi with Andrew Clay Shafer at DevOpsDays Minneapolis 2015, and Matty adds that the title was aspirational: the sushi took years to happen. Matty also notes that the DevOpsDays site doesn't handle a speaker called ADO well.

    Memorable Episodes and Fans

    Bridget highlights Who Owns Your Availability from 2016, recorded the day of the left-pad incident, as still current for software supply chain discussions. Matty calls out 2023's three platform engineering episodes (Daniel Bryant, Pete Cheslock and Matt Kuritz) and says "I still think that Pete is right, that platform engineering really is DevOps with better marketing," with the earlier platforms episode with Kelsey Hightower and Andrew Clay Shafer always linked. Jessica picks the one-on-one conversation with an enterprise practitioner, since those stories show real companies struggling and let listeners know they're not alone.

    Fan stories: a stranger at KubeCon Chicago told Bridget that a computer science course uses the show. Matty recalls an early fan letter that went unanswered on LinkedIn for years. Matty credits the show for much of a career, since "I owe a fair amount, if not all of my career, to this podcast," and notes how easy it is to be a guest on someone else's show. Matty also mentions using the same pun twice, in database episode titles in 2019 and 2023.

    Stats

    The most frequent guest is Andrew Clay Shafer with eight episodes, about 4 percent, then Nicole Forsgren and Sasha Rosenbaum with seven each. Matty says downloads stood at 1,911,233 plus about 35,000 on Spotify, since Spotify hosts its own copy and doesn't show in the regular stats, so two million is near. Episode 1 is most downloaded, then episode 92, CI/CD Oh My with Jez Humble. The most viewed YouTube video is Old Geeks Yell at Cloud at about 16,000 views, then Bryan Cantrill's fireside chat at about 8,000, and Matty jokes that the show should have Bryan on more often.

    Matty also fixes episode data live and discovers guest entries were lost in migrations: the site started on Jekyll for about seven episodes, moved to WordPress with a custom plugin, then to Hugo. After Bridget found a Google alert for another podcast using the theme, Matty turned it into Castanet, a podcast theme that is now poorly maintained. Jessica's verdict: "our memory is terrible and our data is also crap."

    Where Everyone Is Now

    Trevor is job hunting, looking at solution architecture or DevRel, and painting miniatures. Jessica is in a second year as engineering manager of developer relations at Honeycomb, is at a conference run by the Linux Foundation that added AI content to a Cassandra event, and would like the show to cover DevOps for LLM integrations. Bridget works in product at Microsoft, which means listening to other people's stories more than telling them. Joe travels for work to hotel ballrooms in Nashville and Cleveland. Matty leads developer relations and growth at Aiven, a Finnish data platform company, did not reach United's top tier this year, pays around $100 a day for dog care when traveling, and runs the team with a social contract about staying off Slack on holiday. Matty adds that because the company's field isn't Matty's expertise, leading the team works better than trying to do the work.

    Bridget's closing idea, for anyone tired of telling the same stories: step back, help a colleague prep a talk, and "we can always make room to inspire someone else or to teach someone else or to go to a conference that we're not even speaking at."

    • Every ADO Cold Open Ever
    • "Episode 0" of ADO
    • "Old Geeks Yell At Cloud" video

13. So You’re In Charge Now… with Ben Greenberg

    22 de diciembre de 202348 minEpisodio 199A continuación

    Matty talks with Ben Greenberg, who had been head of DevRel at Fuel Labs for about three weeks at the time of recording and also runs a small consultancy, Yalla DevRel. The topic is what to do when you're …

    Notas del episodio

    Matty talks with Ben Greenberg, who had been head of DevRel at Fuel Labs for about three weeks at the time of recording and also runs a small consultancy, Yalla DevRel. The topic is what to do when you're suddenly in charge of a team, or starting any senior role. Ben had guested before, but that episode was recorded in May and shelved because its social networking content had aged, and Matty says this one is aimed at something timeless. The cold open is Matty on the classic interview question: "What's your biggest weakness, Ben? I work too hard."

    Nobody Wants You to Change Everything

    Matty's warning for new senior hires: no matter how often people say they want a change agent, "here's the secret: they do not." Ben calls it the hero complex of hiring, the lowercase-m messiah, and describes doing something new in interviews: sharing a real, still-raw failure instead of a humblebrag, on the theory that if Fuel Labs still wanted Ben afterward, the fit would be real. Matty recalls the first manager job at Apartments.com, a role that was half manager and half sysadmin, where Matty arrived fired up about process and after about four weeks realized it was time to stop and listen. Matty's rule: "you have 2 ears and 1 mouth," and "every process, every way of doing things in a company is organizational scar tissue." The likelier explanation for something being done a certain way is that there's a reason, not that nobody has heard of Salesforce.

    Ben contrasts the invading colonizer who uproots things with the person who sits with people, listens to their stories and helps shape direction from within, and admits that's hard when you were hired to do things.

    The First Weeks

    Matty describes starting a role while knowing a new boss would arrive in three months. A coach suggested using the time for discovery so that the new boss could be handed what Matty had learned and a plan, without having made changes yet. Matty's onboarding technique: in every get-to-know-you meeting, mostly about the person and not the job, schedule a follow-up four to six weeks out, when there's enough context for a substantive conversation. Ben adds asking each person who else to meet, which widens the circle quickly, and meeting ecosystem partners and developers who build on the company's infrastructure. Matty recommends The First 90 Days, with the caveat that it reads like Harvard Business Review articles stapled together, and that it helps with figuring out what kind of organization you're in.

    Matty also uses a model from an executive coach, Bill Joy, who taught engagement and skill as two axes: a new starter is high engagement and low skill, even a principal engineer or a new CEO. "You could be the new CEO. You are low skill." Activity for its own sake looks busy but doesn't win collaboration, and Ben says DevRel suffers from filling calendars without knowing what moved the needle.

    Showing Value

    Matty's standing advice is to learn how the company makes money. In DevRel interviews Matty's philosophy is that "if you don't have a way of demonstrating value, a way of demonstrating value will be assigned to you and you won't like it." Take care of the things tied to business outcomes and nobody will question the rest. Matty jokes about wanting a Nicole Forsgren for DevRel. Ben says the same applies to SDK engineers, technical writers and DevOps: do less, more completely and at a higher standard, instead of covering everything.

    Ben's Plan

    Ben inherited a team of three, dispersed across the Americas, each there about a year, and wants a team "that knows more than you." Ben's first career was as a director at nonprofits, and Ben's earlier stint as a young manager was marked by opinions, talking more than listening and buying into the change agent paradigm, which cost Ben. This time Ben has been listening and taking notes, then drafted a quarterly plan, shared it with Ben's own boss, then the team with an invitation to tear it apart, and scheduled an offsite to build Q1 objectives and key results together. Ben's method is to be a facilitator and consensus builder, and says "ownership is how you create a sense of culture of staying." A key goal is career paths for the team, since nobody had held the role for a while, recognizing what people did in the past year and getting them to where they should be.

    Team Lead Is Not a Manager

    Matty points to Lindsay Holmwood's post "It's Not a Promotion, It's a Career Change," and tells how a new CTO suggested moving Matty from director to infrastructure architect because technology, not managing people, seemed to be what got Matty out of bed. Matty's conditions were no pay cut and a letter making clear it wasn't a demotion, which Matty says illustrates the problem. Both believe individual contributors should be able to earn more than their managers, and Ben cites Aaron Bassett's talk that promotion shouldn't require becoming a manager. Matty says team lead and people manager are very different jobs: leading an open source project is not performance management, career coaching or time-off requests, and as Ben puts it, "you're never putting an open source contributor on a PIP."

    Matty credits Apartments.com for heavy manager training, since many managers were first-timers from internal promotion, and Bill Joy's framing that the hardest transition is the first one, from individual contributor to manager, because you must stop doing what you're used to. Matty also says that leading a DevRel team in data, and not in a space like Kubernetes or infrastructure as code where Matty knows the material, has been easier to do humbly, since "I couldn't get a job on my own team," while still knowing how to do DevRel. Ben calls that humility a good topic on its own.

    • The First 90 Days
    • "It's not a promotion - it's a career change" (Lindsay Holmwood)
    • "Not All Leaders Are Managers" - (Aaron Bassett)

14. DevOps Isn’t a Department with Jeremy Duvall

    7 de diciembre de 202330 minEpisodio 198A continuación

    Matty talks with Jeremy Duvall, founder of Seven Factor Software, a software engineering consultancy in Atlanta, about why DevOps keeps ending up as a department and what to do instead. Jeremy cut teeth at …

    Notas del episodio

    Matty talks with Jeremy Duvall, founder of Seven Factor Software, a software engineering consultancy in Atlanta, about why DevOps keeps ending up as a department and what to do instead. Jeremy cut teeth at Danger, which built the T-Mobile Sidekick, then worked at Microsoft, and has been doing DevOps for a long time. The episode is part of the show's tenth-year look back, and Matty promises the anniversary episode in a month. The cold open is Jeremy on the Agile and DevOps values: "focus on your people and stop worrying about the stupid shit you use to get things done."

    How DevOps Became a Department

    Jeremy's first exposure was a DevOps Days in Atlanta where John Willis spoke on burnout, which showed that DevOps wasn't just about playing with Jenkins. Before that, Jeremy says, nobody thought you needed a department to deploy Jenkins servers: two people on a tools team built the machinery the rest of engineering used to ship. The real contribution of the movement was refocusing on the idea that developers are humans. Then, as with big data and digital transformation, big companies took the ideas, put them in a department with pay scales, and turned infrastructure teams into DevOps engineers doing the same things. Jeremy sees platform engineering as "the next logical evolution" of what DevOps should have been.

    Matty adds that early DevOps deliberately avoided being prescriptive, with no manifesto, just the CAMS acronym of Culture, Automation, Measurement and Sharing from John Willis and Damon Edwards at the first US DevOpsDays in Mountain View, to which Jez Humble later added Lean, for CALMS. People still have to do work, and automation is where vendors make money, so DevOps drifted toward meaning Jenkins and Puppet. Jeremy compares it to Agile: a manifesto that gave rise to SAFe, which Jeremy calls garbage, and also to good things like Kanban and Lean. Matty adds that "SAFe is how to have Agile but let project managers still have a job."

    Measurement and Incentives

    Jeremy says the business community got its claws into both movements with Scrum, velocity metrics and somebody-has-to-get-fired accountability, which produces walled gardens, bureaucracy and a pathological culture instead of a generative one. Matty says measurement doesn't equal Taylorism, contrasts Nicole Forsgren's work on developer productivity at GitHub with McKinsey's, and names Freakonomics as the most important DevOps book: "Go learn about incentives and then you will understand DevOps."

    Jeremy says what engineers do is hard to measure, so organizations reach for features shipped or hours against velocity points, and "A measurement ceases to be a good measurement when it becomes a target." Jeremy points to the developer productivity work of Abhinav and DX, which focuses on happiness, and cites Antifragile for thinking about teams that survive someone leaving or breaking an arm. Jeremy says Google, Amazon and Facebook get this right, with engineers empowered to deploy and build their own platforms, while Jeremy's retail clients are stuck in a 1990s CEO mindset, with Nordstrom as an example of a real turnaround. Matty adds that Courtney, who led that transformation at Nordstrom, has talked about internal platforms for the enterprise, linked below.

    The Frozen Middle

    Matty describes hearing at Chef that about 80 percent of CIOs said DevOps transformation was critical but only about 15 percent of enterprises had a plan, and says the C-suite can think big while middle managers whose job is executing one way of measuring will fight a radical change. At PagerDuty, banks had whole teams of incident managers, and they weren't bad people: "I would probably have done the same damn thing."

    What Success Looks Like

    Jeremy says most of the Fortune 500 now know DevOps is a thing, and successful teams have trust to make decisions and a paved golden road to production, with a platform or DevOps team, whatever it's called, building the frameworks while software engineering teams cross-skilled in things like Terraform get their apps to production without interference. The cloud means developers no longer wait two weeks for a Tomcat server. To avoid repeating the mistake, Jeremy says to avoid commoditization: "everything that's wrong with software engineering today is commoditization and value engineering." Jeremy builds teams with overlapping skills, "I don't want specialists in 2023," and says platform teams are software engineers, "They're not infrastructure people anymore." Matty wraps up with the 2014 episode How to Eff Up DevOps, linked below.

    • John Willis’s talk at DevOpsDays Atlanta 2016 on Burnout
    • https://platformengineering.org/talks-library/internal-platform-enterprise-courtney-kissler
    • ADO - How to Eff Up Devops with Pete Cheslock, Nathen Harvey, and Randi Harper

15. Runtime Analysis with Brian Kelly

    23 de noviembre de 202339 minEpisodio 197A continuación

    Matty talks with Brian Kelly of AppMap, a runtime analysis company, about what runtime analysis is and where it fits between static analysis and production observability. Brian is originally from Ireland, has …

    Notas del episodio

    Matty talks with Brian Kelly of AppMap, a runtime analysis company, about what runtime analysis is and where it fits between static analysis and production observability. Brian is originally from Ireland, has lived in the Boston area for over 20 years, and came to AppMap from distributed systems, SaaS and a cybersecurity company. The guest's employer sells the category under discussion, which the guest says plainly when it comes up, and AppMap also appears in the links below. The cold open is Brian: "We won't say the Log4j word."

    Where Runtime Analysis Fits

    Matty's guess at the definition is "analyzing during runtime," and Brian confirms it. The difference from the APM and observability tools the industry is used to is where in the workflow it happens: on the developer's laptop or in CI, before deployment. Static analysis has become a commodity for classes of problems like vulnerable dependencies, and Brian says a team that isn't using it is delinquent. But there are problems developers assume they can only catch by sending code to production and watching it with an observability tool. Brian says "I hate to say shift left, so I'm going to say shift closer."

    The cost of finding out late, Brian says, is context switching: a developer who has moved on to another pull request hears days later that a change is a problem, and the issue tends to land on a Jira backlog, where an SRE compensates by adding CPUs and RAM, which Brian ties to crazy upside-down hosting costs. Tests and manual QA already generate runtime data that many teams ignore. Matty adds that the point is getting 80 percent: fewer defects reach production, so the remaining ones might actually get fixed, and Matty compares the backlog to Homer Simpson balancing the garbage. Brian says Dependabot and static analysis didn't end pen testing or dynamic testing, and each of them is a signal, but teams over-rely on observability tools and crank up APM spend.

    What It Finds

    Brian's best-known example is the N+1 query from an ORM such as Hibernate or ActiveRecord, where a static analyzer sees that an ORM is in use but not the queries generated at runtime or how many identical ones are issued while paginating. Others are dependency injection and dynamic library loading: a static tool sees a config file and can't see what gets loaded. Runtime analysis watches where "the infinite gets constrained down to the finite," and can show that seven or eight listed libraries are never used and whether the ones in use are used in a vulnerable way. Matty connects this to the Sysdig report finding of loaded but unused JavaScript packages in the Cloud Native Security episode.

    What It Looks Like in a Workflow

    Brian says tools in this class have to be automated, fast and fit the existing workflow: in VS Code, IntelliJ or PyCharm, pressing the run button instruments the application, watching bytecode in Java's case. That sounds like an APM, but it collects more specific data, between observability and a profiler, building a dataset of which functions, queries and libraries run. Analysis comes in automated form, like flagging an N+1 query, and human form. Brian's example: a unit test triggers six occurrences of an N+1 query that's a tiny fraction of CPU time, but the developer knows production data would make that six million. It's also definitive: "this really happened," not a fuzzy prediction.

    Brian expects the concerns to be noise, since static analyzers started out with many false positives, and expectation of magic, but runtime analysis only observes, and is bound by how long the tests take, which developers run anyway.

    The OWASP Top 10

    Brian points to the OWASP Top 10 as an illustration: around 2010 SQL injection was at the top, and then tools and frameworks commoditized the fixes. Brian recalls a 33,000-page automated pen test report for a big bank where prepared statements fixed about 27,000 pages. What replaced it are harder problems like broken authorization and authentication, cryptography and secrets handling, which static analysis mostly can't detect. A secret might pass through a framework that logs it, and a runtime analyzer can find the code paths.

    Getting Started and AI

    Matty mentions Adam Jacob's approach for Habitat, installing the gnarliest enterprise software to prove it, and says that's not the way to start. Brian's answer splits on whether you have automated tests. With tests, put runtime analysis in the editor and in CI. Without tests, instrument locally, on a laptop or a UAT environment, and look at the reports, or "personal observability," which is safe because nobody needs tests to run in production. Matty: "monitoring is simply testing with a time dimension." Brian says you should still write tests.

    On AI, Brian says most AI code reviewer apps just look at the same static diff, while runtime analysis data is a new dataset. In experiments, adding runtime data to an LLM prompt stripped out noise and produced answers like how to mitigate a known N+1 query. For anti-patterns, Brian can't think of one on the spot, and says the risk is overreach: a tool that becomes "a mosquito in your eardrum," as static analyzers did with their 79 vulnerabilities until Dependabot started opening pull requests. Brian ties the category to the DevOps hangover of overspending on observability.

    • OWASP Top 10
    • Stripe: The developer coefficient (quantifies the cost of bad code to companies to be $59B annually)
    • Facebook: FAUSTA: Scaling Dynamic Analysis with Traffic Generation (how runtime analysis was used at WhatsApp to catch design flaws before they reached production)
    • Dragan Stepanović - Async code reviews are choking your company’s throughput (from LAS 2022, a talk which highlights the systemic problems with developers trying to do manual code reviews of large PRs)
    • AppMap, the runtime analysis company which Brian works for
    • Cloud Native Security with Michael Isbitski ADO Episode

16. Complexity with Michael Stahnke

    9 de noviembre de 202347 minEpisodio 196A continuación

    Matty talks with Michael Stahnke about whether the systems we run need to be as complicated as they've become. Michael has spent 13 or 14 years on and off the DevOps circuit, was VP of engineering at CircleCI …

    Notas del episodio

    Matty talks with Michael Stahnke about whether the systems we run need to be as complicated as they've become. Michael has spent 13 or 14 years on and off the DevOps circuit, was VP of engineering at CircleCI, and now works at Flox, an 18-person company building tooling aimed at removing complexity. Matty notes the episode isn't meant as a pitch, and Michael says caring about the problem is why Michael works there. The cold open is Michael: "the original problem was I couldn't get my developer environments unified, and therefore I ended up with Kubernetes. What the fuck?"

    Are We Better Off?

    Michael's question is whether operational availability, debugging and troubleshooting are better than in 2004 or 2005, given that "what we keep doing is inventing new problems and then inventing new solutions." At CircleCI the availability struggles came from "doing really complicated shit," not bad engineers. Both recall 2003: Michael was writing software to replace spreadsheets and manual administration with SSH and for loops, managing thousands of servers, while Matty was moving from Exchange 5.5 to Exchange 2000 after an acquisition, and rebuilding dev servers at Allstate from an answer file on a floppy and a ProLiant CD while sitting in a cold data center with a book. Matty says you could hold the whole stack in your head, and asks whether microservices and distribution have left anyone better off. Michael says unequivocally yes in many scenarios and no in many others, and the first thing to understand is whether you actually have the scaling problems or just think the tools are cool. "Keeping one thing online is easier than keeping 25 things online."

    Build for the Problem You Have

    Matty recalls a Rails app being "up in 3 days, down in 3 months," and Cars.com handling Super Bowl traffic by renting servers for 48 hours instead of re-architecting, since the spike happened at the same time every year. Michael says to find users and product market fit before investing in a service mesh and service discovery: "don't underestimate the power of rsync and cron," and a colo, Linode or DigitalOcean node can go a long way. The test is whether you're delivering the value, and time spent working out whether a Kubernetes minor version will change an ingress controller API is something no customer pays for. Matty adds that "you" is doing a lot of work, from a 12-person startup to JPMorgan Chase, and tells the story of a retailer's ops team insisting on site stability until management said the job was selling things.

    How Containers Led to Kubernetes

    Michael walks back the chain: a developer environment needed to be consistent, so "we're going to package up your laptop, we're going to pass it around until it gets to production," which is what a container is. That led to a scheduler, service discovery, a service mesh, and businesses scanning containers for vulnerabilities, all layered on top instead of asking why the decision was made. If everyone developed in the same reproducible environment, Michael says, containers might not be needed, and then neither would the rest. Michael misses typing service start and strace, where a problem takes ten seconds to find, instead of launching a debug pod. Kubernetes "was originally designed to solve Google-scale problems. Unless you are Google, you do not have Google-scale problems," and Michael's image is that not everyone needs Everest climbing gear to cross the street. Matty: "What's the best container scheduler? The one you don't need."

    Organizations and Data

    Matty describes booth conversations at an AWS Summit where Kubernetes people said data was another team's job. Matty is clear that's an organizational pattern and not a failing of the engineers, and says a platform engineer should care because data is part of a platform. Matty used to accept that DevOps means never saying "that's not my job," and no longer does: "not my circus, not my monkeys" is fine, and if your job is limiting, "maybe you need a bigger circus."

    Michael adds that platform teams exist partly because the complexity was put inside the company, and asks whether they could have a smaller mandate. DevOps ideas of shared empathy and pain were good, Michael says, but operations expertise atrophied and developers reinvented tools, so a problem solved in 1995 gets rebuilt in 2018. "There's always a 27-year-old willing to redo everything you've already learned." Michael counts 150 AWS services, half competing with each other.

    What to Do About It

    Michael thinks simpler tools had a chance and missed: Docker Swarm beside Kubernetes, and a Rust tool like Docker Compose that runs plain processes without containers. Michael's wish is for a generation of tools that abstract the good patterns without all the complexity, solving the 80 percent case. Practical examples from Michael's company: a website behind a CDN and cache that could run on a Raspberry Pi with a cell modem and cost $5 a month instead of $600, and a monolith, with Knuth's line about premature optimization and "I hope that's a problem we have," since scaling problems mean users. One team with one microservice is a success; most places end up with more services than developers, and then Backstage to keep track. Michael adds that "only in software is legacy a bad word": a legacy system made the money, so don't be mad at it.

    Inside a large organization, Michael suggests shortening a workflow from 12 steps to 10, automating the repetitive debug steps, showing a decision maker two workflows and asking what you lose, and learning that "you're still a technical decision influencer." Matty recalls learning from a colleague's resume at Chase that treasury services processed $1.5 trillion in wires a day, information that sat on the business unit's intranet home page. Michael describes a Caterpillar division where every transaction ran through six servers, about $6 million a day, which made an $80,000 software upgrade an easy ask. Michael suggests reading an S-1's risk section to learn what matters to a business, and both lament that value stream mapping is discussed less than it used to be, with Matty blaming Steve Pereira no longer going to DevOpsDays.

    Michael's closing ask: understand the outcomes at the other end, even approximately, so that "my complexity is built because it achieves this goal," and send in stories of solving a problem with a simple solution like installing an RPM and hitting start. Sometimes, Michael notes, you can't do that 300,000 times, and then you actually do have the problems the tools were designed for.

17. The Database Calls are Coming from Inside the House with Grant Fritchey

    26 de octubre de 202342 minEpisodio 195A continuación

    Matty talks with Grant Fritchey, who last appeared on the show in December 2014 in The Database: The Elephant in the Room, about what has and hasn't changed for data and DevOps in the almost ten years since …

    Notas del episodio

    Matty talks with Grant Fritchey, who last appeared on the show in December 2014 in The Database: The Elephant in the Room, about what has and hasn't changed for data and DevOps in the almost ten years since. Grant is still at Redgate Software and has added PostgreSQL to the skill set, and Matty now works at Aiven, a data platform, which comes up as context. Both describe themselves as having "storied careers, which is a nice way of saying we are getting old." The cold open is that line from Matty.

    Nobody Has to Explain DevOps Anymore

    Matty frames the problem: code is close to immutable and easy to roll back, while data is living, and orchestration tools work well until someone asks about the data and it becomes somebody else's problem. Grant says that ten years ago every data person needed an explanation of what DevOps even was, and now "you don't have to talk about or explain what DevOps is anymore," which saves half an hour of every talk. Grant uses Donovan Brown's ordering of people, process and products.

    Matty asks whether the organizational divide has improved, based on conversations at an AWS Summit booth where infrastructure people treated databases and Kafka as another team's job. Grant says it's getting way better, with a split between companies that build DevOps in from the start and incorporate data management, and companies that add DevOps later. Grant also notices that database administrator is a term going away: people who do backups, availability and query tuning now call themselves data engineers. Matty compares it to sysadmins becoming DevOps engineers with a pay bump, and both say to change the title if it helps, since the work matters more.

    Still Struggling

    Grant admits being "a bit of a negative Nelly" here: "We are still struggling." The obstacle is persistence, since you can't toss the database and start over, and deployments have to happen without taking the server down for three hours. Grant thinks data management people haven't explained well enough to developers what they need, and developers often look at the database and see something scruffy. The joke that data work is like sweeping up behind a parade with elephants in it is, Grant says, how it sometimes feels, though the job is fun. Teams that have been bitten by data problems either embrace automation or avoid the topic altogether.

    Databases Are Code

    For the platform engineer who runs Kubernetes and leaves data to a data team, Grant's message is "Databases are code," and Matty adds "just big code." It's big, so it can't move fast and self-provisioned development databases need third-party tools or empty databases, but it can be treated like the rest of the code, with the special part being persistence. Grant says the biggest misconception is that the database can't be automated. It takes a bit more discipline than automating development because the data must be kept, but it's fully automatable, and wildly successful organizations automate their data management and deployments.

    Communities and History

    Grant finds the PostgreSQL community as welcoming as the SQL Server community is most of the time, and notices a split in Postgres between committers oriented around academics, theory and science, and everyone else who uses it and wants help. Grant says the same split is familiar from DevOps, where "Peggy does DevOps" doesn't make a DevOps company. Matty draws on the history-of-databases talk with Kat Cosgrove, from 1970s Ingres through Postgres, and says knowing why things are the way they are helps. Matty recounts that the relational model's creator disliked rows, columns and tables and wanted something more mathematical, like tuples.

    Grant says the old foundations persist because they work, "rebar inside of concrete," which the Romans used. Grant is building a LoRa and IoT project on Azure and still uses a relational store, since the data volume is small. Specialized databases such as Cosmos DB are like a specialty tool for taking antennas off a radio, and you still need a hammer and nails.

    What's Exciting

    Grant is excited by open source being everywhere: AWS, Azure and GCP all include or support it, and the divide between the commercial and open source camps is shrinking, though paid software will remain, as with Query Store in Postgres on Azure. Grant also expects people to try to put AI into production tooling.

    Grant's advice for growing a career is to "assume automation from the start," then decide whether you enjoy the why of things, such as query tuning and design, or the how, such as automation in Azure and AWS, Kubernetes and containers. The local job market matters too: in Tulsa, Oklahoma, SQL Server dominates, while on the coasts Postgres is growing. Grant has submitted to PGConf Europe in Prague. Grant's last word: "Treat your database like code."

    • Arrested DevOps - The Database: The Elephant in the Room
    • Arrested DevOps - Data! Data! Data! With Francesco Tisiot
    • Arrested DevOps - The New DevOps With Adam Jacob History of databases talk from Matty and Kat Cosgrove

18. Platform Engineering goes to Flavortown with Matt Kurtiz

    5 de octubre de 202347 minEpisodio 194A continuación

    Matty talks with Matt Kuritz, staff engineer and tech lead of the platform engineering team at The Farmer's Dog, a fresh dog food company, about what platform engineering looks like at a company that has been …

    Notas del episodio

    Matty talks with Matt Kuritz, staff engineer and tech lead of the platform engineering team at The Farmer's Dog, a fresh dog food company, about what platform engineering looks like at a company that has been doing it for about four years. The company has grown from a few engineers to roughly 50 to 100 and has shipped over 100 million meals. Matty frames the episode as a return to platform engineering after recent episodes with Daniel Bryant and Pete Cheslock, with the joke that none of them actually do it and just want to talk about it. The cold open is Matt Kuritz, on why tools are better: "infrastructure tools and software just becoming more like real software."

    Product Teams and Platform Teams

    Matty starts from the Charity Majors post on the Honeycomb blog, which has a table contrasting platform engineers and ops engineers, including a row where SSH is a no for platform engineers. Matt Kuritz says the idea came partly from a diagram in Lean Enterprise about self-service operations building a PaaS, which Matt never wanted to build, but its goal stuck: in an ideal world there are no handoffs between developers and operations, and product teams own their software end to end. At The Farmer's Dog there are two kinds of team, product engineering and platform engineering, and "the only difference is who the customer is." Everyone cares about reliability and delivery. Matt Kuritz adds a disclaimer that if SRE or DevOps works for a company, there's no reason to drop it, and that some companies have to go deep on infrastructure.

    Matty adds that nobody can talk about platform engineering without James Governor's line about "endlessly remaking remakes of Heroku," and says the point of Heroku was getting to value fast, with abstractions where they need to be, and not feeling like Dropbox. Matty repeats that "the best tool is the one you don't need. The second best one is a SaaS." Matt Kuritz says to "do what works at the right scale and then iterate and evolve from there," naming Vercel for front-end-heavy companies and Stripe as a business function nobody wants to run. The Farmer's Dog isn't building a PaaS but assembling a platform of preferred tools with glue where it pays off, which Matt sees as a pattern others could use.

    How the Team Measures Itself

    Matt Kuritz says the team has to be aligned with product teams on the goal, and their north star is ownership. The first milestone was continuous deployment, since every commit then has one clear owner who is responsible through the pipeline to customers in production. The team reached 100 percent continuous deployment of its applications after fixing a distributed monolith, and before any golden paths. Matt is frustrated by takes that say DevOps is dead and platform engineering is the future, and recommends Charity's DevOpsDays New York talk. The work, Matt says, wasn't different from what a DevOps team might do, but the language and structure changed: platform and product, not dev and ops. Ownership is still a multi-year goal, with each application having one clear owner, and "platform doesn't own anything."

    For the product management side, Matt prefers to work hands-on with internal customers: do a real use case manually, maybe one or two more times, and only then invest in a code generator or golden path. "We don't upfront really decide anything. It has to be done at least once and put into production." The mistake teams make, Matt says, is making decisions in their own echo chamber.

    Who Decides What

    Matty's rule of thumb for what to standardize is whether something is an interface point between groups: a shared source control tool matters, the JavaScript form validation library doesn't. Matt Kuritz says platform steps in as the decider for cross-team choices, such as asynchronous messaging, where RabbitMQ, SNS, SQS, Kinesis and Kafka could all show up in one app. The team collects opinions from all teams and asks whether a tool that becomes the standard would be worth maintaining, and challenges assumptions. Matt says Kafka is powerful and flexible, with many ways to get hurt, and the team evaluated Confluent but couldn't find use cases yet. Matt describes path dependence with driving on the right side of the road, and says to be honest about migration costs. Matty adds that people who've never heard of a tool will say it's fine for their team, so the platform team has to bring the context.

    Bounded Contexts, Not Silos

    Matt Kuritz says "functional silos are unhelpful," but not having boundaries means everyone has to know everything, so the answer is bounded contexts in the domain-driven design spirit: vertically integrated teams mapped to business domains, like fulfillment and signup, that rarely need to coordinate. Jess Kerr's blog post on better coordination or better software supports the idea: build technical components that let people work asynchronously instead of perfecting the handoffs.

    On the technical side, the team chose a monorepo, Node and one language, and not introducing another until necessary. The monorepo exists to practice trunk-based development and continuous deployment, and works only if there's one version, which is head. Upgrading Fastify across dozens of apps is one atomic commit that triggers the tests of dependents. The team has no Backstage or internal developer portal, since a code owners file in a repo organized by business domain is the directory, and Git is what developers already know. "What's our platform? It's a monorepo and a code generator. That's our platform."

    Start and Stop

    Matt Kuritz's thing to start is setting a long-term vision you can never fully achieve, an idea from Toyota Kata: for The Farmer's Dog it's that every dog lives its longest life. Then break it into 1 to 3 year challenges and near-term target conditions, like moving a 10-person team to three deploys a day. The thing to stop is copycatting, or cargo culting, which Matt ties to Feynman's Cargo Cult Science and to Lean being copied from Toyota by its artifacts like Kanban cards. In platform engineering it's launching Kubernetes and Backstage because other companies did. "Think about your problems, solve those problems from first principles, don't copy."

    • Arrested DevOps - DevOps With Better Marketing with Pete Cheslock
    • Arrested DevOps - Platform Engineering with Daniel Bryant
    • Arrested DevOps - Platforms with Kelsey Hightower and Andrew Clay Shafer
    • Lean Enterprise
    • The Future of Ops Is Platform Engineering
    • Charity’s talk from devopsdays NYC
    • Jess Kerr’s blog that Matt mentioned
    • Cargo Cult Science

19. What's Up With Open Terraform?

    21 de septiembre de 202339 minEpisodio 193A continuación

    Matty talks with Ohad Maislish, co-founder and CEO of env0, which manages Terraform, Pulumi and CloudFormation, and Cory O'Daniel, CEO and co-founder of Massdriver, a visual environment for cloud infrastructure …

    Notas del episodio

    Matty talks with Ohad Maislish, co-founder and CEO of env0, which manages Terraform, Pulumi and CloudFormation, and Cory O'Daniel, CEO and co-founder of Massdriver, a visual environment for cloud infrastructure that depends heavily on Terraform, about the community fork then called Open Terraform. Both guests run companies with a stake in the outcome, which Matty says up front while insisting it's not a criticism of their motives. Matty's own history is with Chef and Pulumi, and Matty says there is no dog in this hunt. The episode's URL and title predate a rebrand announced partway through. The cold open is Matty: "definitely GitHub is just effed right now."

    What Happened

    Ohad's account, as a participant: on August 10th HashiCorp changed the licenses of several projects, including Terraform, and a group of vendors and individuals wrote a manifesto asking that Terraform stay open source forever. When HashiCorp kept its decision, which Ohad calls "totally legit," the group started working on a fork. Cory adds that some CNCF projects began pulling away from HashiCorp tools, and that Massdriver wants to bet on a tool that the open source community keeps investing in, the "new lingua franca of infrastructure as code."

    Matty asks why a license change matters to an ordinary engineer. Matty's concern is vagueness: depending on how a lawyer reads the license, distributing a project that uses Terraform could be a violation, and you can't always know who is competitive with HashiCorp. Ohad says the license was followed by a binding blog post of clarifications on August 21st and changes to the Terraform Registry terms on August 24th, which shows the community didn't understand the implications. Every fundraising round, Ohad says, means sending the license to investors, and a BSL license with a binding blog post and more updates is harder to explain than a clear alternative. Ohad contrasts it with SSPL, which exists to stop others from reselling a vendor's product as a service, and calls BSL flexible and, in Ohad's opinion, still vague.

    Matty adds that enterprises already fight compliance just to use open source at all, and a complicated license makes that harder. Cory says Massdriver was affected and then wasn't, since it competes with Waypoint, which was carved out of the FAQ a week later, and that dynamism is what worries people. Cory gives examples of transitive dependencies: a CNCF project that uses Consul, projects pulling out Vagrant, and Jaeger considering removing a Go plugin, which is still MPL. "It is very much email us to figure out if you owe us money," Cory says, compared with how other projects handled BUSL changes.

    OpenTofu

    Asked what to call it, Ohad says OpenTF, and Cory breaks the news that will go public the day after recording: the Linux Foundation recommended a rebrand because TF could be confused with Terraform, so the project becomes OpenTofu and the binary is tofu. Ohad and Matty make the fork and tofu jokes. The project now belongs to the Linux Foundation, the natural step toward CNCF, and Ohad says it is meant to be a drop-in replacement: no code changes, just a different binary and registry. Gruntwork, the creators of Terragrunt and Terratest, and Harness are among the supporters, and Gruntwork has released a sneak peek of state encryption, which Matty recalls as a Pulumi differentiator since a Terraform state file can hold secrets in plain text.

    Matty asks what makes the fork more than a burst of pledges, recalling a maintainer maxim: "as a maintainer, no is temporary. Yes is forever." Ohad points out that Terraform core was open source but stopped accepting community pull requests about two years ago, while OpenTofu has a five-member steering committee under Linux Foundation and CNCF guidelines, and has already deferred a community change to a later version as a new capability. Ohad says RFCs and voting rules are a work in progress. Cory hopes that teams replacing HashiCorp tooling will contribute and speed up a project that "really has been bogged down for the past couple of years."

    The Registry

    For modules and providers, Cory says the first registry will proxy requests to GitHub and resolve module names to repositories, and Ohad adds that provider naming conventions map to GitHub URLs. So publishers won't have to submit anything. Cory sees an opportunity to support OCI-compliant registries later.

    How to Help

    Ohad suggests starring the repo, joining the community Slack, filing and voting on issues, and sending pull requests, and sees an opening for large vendors and cloud providers to shape the project under CNCF. Cory suggests putting OpenTofu into CI pipelines, for example with Terratest, once the alpha registry is out, to surface edge cases in loading providers and modules. Cory's closing point is that the group is "a consortium of competitors" that get along well because they love the language. Matty promises to file a first pull request to nominate Cory's Instagram-famous Australian Labradoodle, Ziggy O'Doodle, as mascot.

    • https://www.instagram.com/ziggy.odoodle/?hl=en
    • https://twitter.com/opentofuorg
    • https://github.com/opentofu
    • https://linkedin.com/company/opentofuorg
    • https://github.com/opentofu/opentofu

    DevOps World is back for 2023, and you won't want to miss out on this one-of-a-kind event! This year's program is packed with exclusive insights, immersive workshops, and unparalleled networking opportunities taking place across multiple cities in the US, UK, and Asia. Elevate your DevOps game and register using the following links: NYC area, Chicago, Silicon Valley, Singapore, and London.

20. The New DevOps with Adam Jacob

    7 de septiembre de 202358 minEpisodio 192A continuación

    Matty talks with Adam Jacob, CEO of System Initiative and, in a previous life, CTO of Chef and the person who wrote Chef originally, about what DevOps got right, where it fell short, and what a second wave of …

    Notas del episodio

    Matty talks with Adam Jacob, CEO of System Initiative and, in a previous life, CTO of Chef and the person who wrote Chef originally, about what DevOps got right, where it fell short, and what a second wave of tooling might look like. Adam counts DevOps from John Allspaw and Paul Hammond's 2009 talk at Velocity, and describes being on the "loves being a systems administrator" side of the field. The episode is sponsored by Uffizzi, and System Initiative is Adam's own company. The cold open is Adam: "It didn't matter how good you were at operations if the application didn't run, and it didn't matter what your application did if there was no infrastructure to run it on."

    What DevOps Got Right

    Adam starts with what went well by describing the year 2000: operations was separate from IT, siloed, and ran on long planning cycles, with developers requesting gear that took six to eight months to arrive and capacity planning a big deal. The organizations worked, Adam says, a lot like how people describe platform engineering today: operations stitches it together and builds systems so developers don't think about infrastructure, "and never the twain shall meet." The top achievement of the DevOps movement was recognizing a single continuum of work, and the pre-DevOps world was "objectively worse" in day-to-day work, tooling and what the systems could do.

    Matty adds a story from leaving Chef: a customer engineer at a large Chicago financial organization apologized for not getting much done, and Matty, who'd watched them in the thick of it for years, said they had come a long way. Adam agrees and says the flip side is real, with teams wanting 100 deploys a day with no friction and still mostly making pipelines and hoping it worked out. Following the DevOps Handbook roughly as written, Adam says, leads to mediocrity, deploying once every six months, which is better than before but rarely great.

    The Platform Engineering Smell

    Adam says the platform engineering rebrand comes from the feeling that DevOps failed because outcomes didn't arrive, and the proposed fix "does smell a lot like what life was like in 2001": "let ops be ops, let engineers be engineers" with some software in between. Matty puts it as accepting that the silos can't be removed, "So let's just make sure the silos are better." Adam's answer is that the people who succeeded collaborated better, and that the tooling was never designed for collaboration.

    Adam's history of automation runs from 1990s compute clusters and college labs, through ratios of 10 to 1, then 100 to 1, and up to 10,000 to 1 as EC2 and Facebook apps arrived, and each stage had to make up an answer for the world of the time. DevOps then "ossified the shape of the world into that shape." The Flickr talk shows it: a portal for deploying, feature flags, dark launches, configuration management, capacity planning and metrics, with deploys started from Subversion tags, so they "literally built a platform roughly the way that we describe it." Every piece of the stack has been rebuilt ten times in ten years without an appreciable change in outcomes, and Adam insists it isn't any one tool's fault.

    Tools and Culture

    Matty credits Adam with "tools influence culture, culture influences tools," which ends up in Matty's decks. Adam says regretting "the idea that it's about culture and not about tools was wrong from the jump," because culture is what you do, and a version of culture in DevOps was like being a lapsed Catholic: believing the right things hard enough. An organization will not become more collaborative without tooling that forces it, since individual people won't do it alone. "Tooling is culture because it's literally what we do all day."

    Matty ties it to the book Switch, where a manufacturing machine that kept injuring hands was redesigned so that turning it on required both hands: make the right way the easy way. Matty's Asana fight over a defined process is the same thing.

    Factories, Soccer and Collaboration

    Adam says DevOps inherited factory metaphors from lean, but software is closer to a professional sports team or an orchestra: highly motivated specialists unified on one objective and making tiny decisions together in real time. Adam's picture is a soccer team retrofitted with pipelines, review by other people and no coaching during a Scrum window. The result was process and tooling that sucked out the creativity and collaboration, where people end up "working near each other at best." The most important insight of DevOps was that what separates great teams from okay ones is "the rate of collaboration," and nobody designed tooling for it.

    Matty asks how much of this is outside the control of the people who can change it, comparing it to Agile needing changes in finance, sales and marketing. Adam answers that change is hard at scale but not insurmountable, and that the LivingSocial story of sales having already sold a removed experiment is a collaboration failure. At System Initiative, which Adam describes as "everything I believe at 12," one Miro board runs from the pitch deck down to an individual story, and Adam reads the whole strategy to everyone every Monday so engineers know why and for whom they're working.

    Scale and the Enterprise

    Matty asks about JPMorgan Chase scale rather than a 15-person company. Adam says large organizations can be convinced with data that DevOps is better, but they buy tools instead of building them, unlike Google and Facebook, whose tools are bespoke. The vendor's temptation is to adjust the tool to the customer's culture for a bigger check, which is "a tomorrow problem for tomorrow people," and Adam recalls a Cloud Foundry deployment that collapsed under its own integrations. The way out, Adam says, is to build a tool that teaches the right way, because enterprises reliably buy tools that promise better outcomes: "let the tool change them, not the other way around."

    Matty pushes back that the cynic sees churn, and asks whether the tool will get used by the people in the middle. Adam says it will if it's good, pointing to Salesforce as sticky because "it's actually kind of good," and admits not being sure this is the only answer. What Adam is certain of is that "the status quo is not a good enough answer," since a new Terraform-like or Ansible-like tool will have roughly the value of the old ones. The thing not yet tried is changing the shape of the system: why a pipeline in the middle, why only one layer of the application, and "what even is an application?"

    Simulations Instead of Code

    Matty asks how System Initiative reasons about this. Adam starts from automation, which has followed one line of thought since Mark Burgess's computer immunization paper in the 1990s, and argues that code is a poor medium for collaboration and that feedback loops of 20 minutes or more, with an opaque Terraform plan, are too slow. Looking at adjacent fields, Adam lands on digital twins: Formula 1 teams simulate every variable of the car so only promising changes are tried at the track. System Initiative is an attempt to build a high-fidelity simulation of infrastructure, removing glue code and building the workflow into the system, and Adam says more people need to pull on other threads too.

    Adam's closing hope is that people become willing to change the shape of the system and "throw some of the babies out with the bathwater." Matty adds that the past was the best available with the information at the time, and that maybe it was what got us able to learn these things.

    Links to Resources Mentioned

    • 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr
    • *The DevOps Handbook
    • The System Initiative
    • Switch: How to Change Things When Change Is Hard

    DevOps World is back for 2023, and you won't want to miss out on this one-of-a-kind event! This year's program is packed with exclusive insights, immersive workshops, and unparalleled networking opportunities taking place across multiple cities in the US, UK, and Asia. Elevate your DevOps game and register using the following links: NYC area, Chicago, Silicon Valley, Singapore, and London.

21. Purposeful Personal Brand with Cassandra Faris

    24 de agosto de 202352 minEpisodio 191A continuación

    Matty talks with Cassandra Faris, who runs community for KubeCampus, a free Kubernetes training product, about building a personal brand on purpose. Cassandra began in tech as a recruiter, has a philosophy …

    Notas del episodio

    Matty talks with Cassandra Faris, who runs community for KubeCampus, a free Kubernetes training product, about building a personal brand on purpose. Cassandra began in tech as a recruiter, has a philosophy degree, and went on to manage communities around SaltStack, a financial project and now people learning Kubernetes. The cold open is Cassandra: "the more you do that and the more you reach out and help other people as well, the stronger your brand is, the stronger your reputation is, the more success you have."

    You Already Have One

    Cassandra's practical case is that a brand helps a career: speaking invitations come from what people know of Cassandra's work, and for jobs "the resume is kind of a formality at this point." Even an individual contributor needs some online presence so people can see what they're about. Matty adds that you have a brand whether you want one or not, since it's what people think of when they think of you, such as the SRE everyone asks about replicas, or the colleague who is difficult to work with. Matty brings up the Mad Men line, "if you don't like what people are saying about you, change the conversation," and says that we have some responsibility to take ownership of it.

    From Accidental to Purposeful

    Cassandra's own brand started by accident. As a recruiter told not to be the one who doesn't know Java from JavaScript, Cassandra went to Agile and cloud meetups, then followed everyone tweeting the #CodeMash hashtag, listened for a while, and started to participate, posting job search tips and skills advice. Developers turned out to welcome a recruiter as long as Cassandra was authentic and not working an angle, and invitations to panels, talks and conference organizing followed. Cassandra realized that being deliberate about what was posted could work in Cassandra's favor, hence the talk on a purposeful personal brand.

    Matty's version: at Chef, Matty was known for infrastructure as code, and at PagerDuty, where incident response was the focus, nobody knew Matty for that despite years of doing it as a job. So for a year Matty took any stage to talk about incident response, blameless postmortems and learning from incidents. Matty insists that this wasn't inventing a persona, it was deciding what to be known for. When Matty's boss later asked for the same reputation in the ITIL and IT service management world, Matty's answer was that "you can't just decide to have authenticity." It takes time, and you have to be able to back it up. Cassandra says every role change means updating, not rebranding: the human skills of building relationships with a community's key leaders are the same whether the community is open source contributors or Microsoft MVPs.

    Your Brand Inside a Big Company

    Matty notes that almost nobody stays at one big employer for 20 years, so external brand matters even in roles that aren't public, and asks how it works inside an organization like Target or Visa. Cassandra says to choose opportunities that match who you are: not competitive, cutthroat cultures if you aren't competitive, and plenty of room for creativity if you need it. Matty adds that the tension is credit: talking about what the team did as well as what you did, since a story that is all about I doesn't show a team member, and ops roles are like a corporate lawyer, where nobody knows what you do until you don't do it. "If you don't, who's going to do it for you?" Nine times out of ten, Matty says, you think you're bragging and you aren't.

    Three Questions and Framing Accomplishments

    Cassandra suggests writing down answers to three questions, which are in the slides linked below. Who are you professionally: your top three technical specialties, three professional specialties and three contributions to the team and profession. Who are you personally: three interests and hobbies, three beliefs and values, and three parts of your social, family or community life you want to connect over. And how do you connect with people: shared interests, advice you're seeking and what you want to know about others. That gives you a bank of material. "Modesty isn't a career-enhancing trait, but that doesn't mean be an asshole." The trick is to frame accomplishments by who they helped. Cassandra's example is a KubeCampus Kubernetes Learning Day that sold out and gave 200 people an introduction to Kubernetes, which reads as "it helped all of these people" and not as bragging. Give credit to the team, and practice, even in a mirror.

    Show, Don't Tell

    Matty says the quickest way to get mocked on tech Twitter is to put thought leader in your bio, and recalls putting up a page of DevOps thought leaders a decade ago until J. Paul Reed asked to be taken off it. If you solve a problem, the story should be about the interesting way it was solved, and the expertise comes through. Matty also recommends keeping a brag book, an Evernote notebook of kind messages that started as a cure for imposter syndrome at Chef and turned out to help at performance review time and with seeing what's worth telling. Matty's DevRel team writes a monthly wrap-up for the company, framed around why it's useful to sales and others, and not as a pat on the back: "how do you brag by giving someone something useful?"

    Making Time and Choosing Platforms

    Cassandra keeps an hour blocked on the calendar every morning for social media, takes a day or two off the internet after a conference to avoid burnout, and pays attention to whether the people and things followed make things feel psychologically unsafe, since that makes content hard to produce. Platforms are shifting as Twitter usage declines: short-form video is growing, and Cassandra sees a generational split where Gen Z and younger millennials want to connect on Instagram while everyone else uses LinkedIn. The brand can stay the same while the tools change, so it helps to be on several.

    Matty suggests blogging may have a renaissance, especially documenting what you're doing or learning in the open. Matty's most visited page is a post on configuring SharePoint in a one-way trust, written to document a fix for a team, which became the answer on an MSDN forum. Matty tells of Annie, who learned an infrastructure testing tool in public on a blog until the product team said "Annie's blog is our documentation now." Beginner content is the hardest to create and the most needed, Matty says, since hundreds of thousands of people need to install Kubernetes and only a dozen need to optimize sharding at a million users per second. Teaching also helps you learn.

    Vulnerability, Asking and Paying It Forward

    Cassandra is seven months into a role at KubeCampus, is teaching the networking fundamentals lab, and plans a blog series on the learning journey. "It starts with admitting you don't know." Nobody has ever responded by doubting Cassandra knows what a cluster is, they've offered to explain it. Cassandra's tip for new brand builders is to ask a question or for resources about something you want to learn, since developers love to share knowledge.

    Matty adds that asking for help to learn is different from asking someone to do a task, and appreciation matters without genuflecting: "I'm not going to carry the bucket for you," but I'll give you a map, and sometimes you just need the bucket carried and can ask for that favor. Cassandra says helping people isn't transactional but pay-it-forward: Cassandra can't hire a former boss and mentor who is looking for a director of DevRel role, so the return on that help is paying it forward, and Cassandra passes the same knowledge on to others. Matty says to ask few favors and accept no, and that you don't need to go to the most famous expert every time, since thousands of people know Kubernetes. Cassandra's answer is meetups and conferences, where the hallway track has led to meeting an author whose Kubernetes book now helps, and a KubeCampus Slack for when learners get stuck.

    Links to Resources Mentioned

    • Keep a Brag Book
    • KubeCampus: Free Kubernetes Training
    • Purposeful Personal Branding - Nov 2018 Slides 13-15 have the 3 questions
      
      *DevOps World is back for 2023, and you won't want to miss out on this one-of-a-kind event! This year's program is packed with exclusive insights, immersive workshops, and unparalleled networking opportunities taking place across multiple cities in the US, UK, and Asia. Elevate your DevOps game and register using the following links: [NYC area](https://reg.rainfocus.com/flow/cloudbees/devopsnyc/webinar3/page/landing), [Chicago](https://reg.rainfocus.com/flow/cloudbees/devopschicago/webinar3/page/landing), [Silicon Valley](https://reg.rainfocus.com/flow/cloudbees/devopssiliconv/webinar3/page/landing), [Singapore](https://reg.rainfocus.com/flow/cloudbees/devopssingapore/webinar3/page/landing), and [London](https://reg.rainfocus.com/flow/cloudbees/devopslondon/webinar3/page/landing).*

22. Everything's a Product with Sarah Morgan

    10 de agosto de 202351 minEpisodio 190A continuación

    Matty talks with Sarah Morgan, senior product manager at Telemetry Hub, about applying product management thinking to DevOps, SRE and internal platform work. Sarah has about ten years in product after an …

    Notas del episodio

    Matty talks with Sarah Morgan, senior product manager at Telemetry Hub, about applying product management thinking to DevOps, SRE and internal platform work. Sarah has about ten years in product after an engineering background, and a career that goes back to the early 2000s. Matty's talk Everything's a Product, linked below, is the starting point. The cold open is Matty: "I think everything's a DevOps problem."

    Thinking Like a Product Owner

    Sarah says product people tend to stop at the business case and the features and forget reliability, stability and the rest of the user experience. Ownership has also gotten more siloed as systems have grown, so nobody sees the big picture. Matty recalls a product owner for the SRE team at PagerDuty and failing to find any posts or talks from that person. Sarah contrasts the two roles: supporting a SQL Server cluster meant caring about the health of the cluster and little else, while a product manager has to care about all of it, whether or not every piece is understood, because it all affects the end user.

    Matty's line from the talk is that you won't put an NPS score on your Jenkins pipeline, "but kind of are you," since it's about users and feedback loops, and that's DevOps. Sarah suggests a health rating for every piece of the system, covering whether it does what's needed to support the people paying the bills. Sarah has mostly worked in B2B software, where users often have no choice, and says internal stakeholders should be treated the same way: don't build things so hard to understand or so locked down that colleagues can't make sense of them.

    A PM on the DevOps Team

    Sarah was the product manager for a DevOps team at a company whose product was a messaging gateway for an IoT platform, a role the company hadn't had before. The team, spread across Boston and Budapest, managed development environments, AWS infrastructure, federation access control, security and disaster recovery, and was about six people before Sarah made it seven. After watching for a month, Sarah saw repetitive work that could be productized and automated, and spent much of the time running interference between the team and people used to going straight to a favorite engineer for access. The team's customers were the software developers, so the job was finding their pain points and building a strategic roadmap, where ops work is usually "pipeline-style ticketing" of whatever is oldest or loudest. The team liked having a direction.

    Roadmaps and Promises

    Matty's talk has a slide that says "this is why roadmaps are bad, and if you have one, you should feel bad," since a roadmap can be read as a promise, and asks how to keep transparency without that. Sarah calls it the bane of every product manager's existence and handles it sneakily: only the roadmap shared with engineers is called a roadmap, and that's where dates live. Everything else goes out as vague documents with names like "H1 priorities." Sarah tells sales to say "new alerting integrations" and not whether it's PagerDuty or a webhook, and adds detail as a release date firms up.

    Matty recalls Marty Cagan spending two days at Apartments.com, and the LivingSocial story of running an experiment, removing it, and having sales say it was already sold. Matty's point is that you can't be agile in only one part of the company, and ties it to Andrew Clay Shafer's observation that everyone wants more reliability, stability and velocity without changing anything.

    Platforms as Products

    Matty says platform engineering is "fundamentally providing Heroku inside your company," and that an internal platform team might think it needn't worry about customers because the CIO mandated it. But people who don't get what they need will go around it, which is "shadow IT all over again," and platforms tend to be thought of as compute and orchestration while event streaming, data pipelines and messaging get left out. Sarah says a SOC audit reveals the rogue platforms, and that you have to think about use cases: at a previous company a backend service didn't handle bulk requests, a front-end engineer looped over a couple thousand rows, and it fell over as soon as two people used it at once.

    MVP Means Prototype

    Matty says Marty Cagan reads MVP as minimum viable prototype, meant to help you learn, even though most people hear minimum viable product and treat it as version 1. Sarah says experiments are supposed to be fast and needn't scale, but then they ship and never get a version 2: "MVPs are not MVPs anymore." Matty compares it to critical production systems under someone's desk, including a machine in a Bank One data center that nobody could identify. Discovery, Matty adds, means asking questions and finding proxies, since people know what they want but don't always communicate it in a form you can build. Sarah says to repeat requirements back and to have conversations, because a PRD or ticket can't hold it all.

    Product Managers as Incident Commanders

    Matty says the two places product people shine are community conference tables, where they spend the day talking to users, and incident command. At PagerDuty a product owner asked to join the incident commander rotation, which had been engineering management. Told that a product owner wasn't an engineer, the answer was that "that's a feature, not a bug." The product owner became the first non-engineering incident commander, and by the time Matty left there were no engineers in the rotation. The reasoning: "never half-ass 2 jobs, whole-ass one job," so an incident commander who isn't a software engineer never gets pulled into fixing. The skills overlap too: prioritizing, communicating and delegating. Matty says it also changes how product folks think about reliability, because the concerns arrive firsthand and not filtered through on-call.

    Matty tells of a sysadmin on the team who couldn't get the product owner to care about a service throwing around 25,000 false errors a minute, because the sysadmin led with the symptom, and the argument that landed was that nobody would notice a real failure. Putting the error rate on the office dashboard got it fixed in about two days. Sarah's version is selling disaster recovery to a general manager by describing the worst outage scenario in dollars, and Matty adds that "money is a lingua franca," not that it's all anyone understands.

    Sidecar Skills and Learning

    Matty calls business cases, writing and speaking "sidecar skills" that set engineers apart, recalling developers who dropped an idea when the CTO asked for a business case, and saying the exercise works like a rubber duck. Sarah says none of the interpersonal, interviewing, listening or public speaking skills came naturally and all were learned. Sarah also warns against underestimating colleagues' ability to learn technical details, from sales to product. Matty extends it to incident commanders, who need to understand a system well enough to know whom to call, without knowing how to rebuild the carburetor.

    Sarah's start and stop: start thinking about who your stakeholders are, and stop building things without validating them.

    • "Everything is a Product" - Matty’s talk
    • “More Buzzwords Won’t Help” - Andrew Clay Shafer
    • Telemetry Hub channel on YouTube
      
      *DevOps World is back for 2023, and you won't want to miss out on this one-of-a-kind event! This year's program is packed with exclusive insights, immersive workshops, and unparalleled networking opportunities taking place across multiple cities in the US, UK, and Asia. Elevate your DevOps game and register using the following links: [NYC area](https://reg.rainfocus.com/flow/cloudbees/devopsnyc/webinar3/page/landing), [Chicago](https://reg.rainfocus.com/flow/cloudbees/devopschicago/webinar3/page/landing), [Silicon Valley](https://reg.rainfocus.com/flow/cloudbees/devopssiliconv/webinar3/page/landing), [Singapore](https://reg.rainfocus.com/flow/cloudbees/devopssingapore/webinar3/page/landing), and [London](https://reg.rainfocus.com/flow/cloudbees/devopslondon/webinar3/page/landing).*

23. Cloud Native Security with Michael Isbitski

    27 de julio de 202356 minEpisodio 189A continuación

    Matty talks with Michael, who began as an enterprise architect at Verizon, moved into assessing application security there, spent about five years in research and advisory at Gartner focused on application …

    Notas del episodio

    Matty talks with Michael, who began as an enterprise architect at Verizon, moved into assessing application security there, spent about five years in research and advisory at Gartner focused on application security, and is now director of cybersecurity strategy at Sysdig. Sysdig sponsors the episode, and the report the conversation starts from is Sysdig's own. The cold open is Matty: "And that's how we did security in the '90s, yo."

    What the Sysdig Report Shows

    Michael says the report is based on anonymized customer data, not a survey, so it covers a slice of the industry: organizations that have acknowledged a security problem and use tools like Sysdig. The number of vulnerabilities is alarming, which says something about the state of open source and the hygiene of components, and scanning usually reveals a worse picture than anyone expected because of nested and transitive dependencies. One result Michael double-checked was the share of non-human identities, which dropped from 88 percent to 58 percent of identities in customers' cloud environments, a shift Michael attributes partly to organizations staffing up after the pandemic.

    On SBOMs, Matty notes that everyone at KubeCon in LA at the end of 2021 wanted to talk about them and the report suggests the industry is mostly still talking. Michael says there are two problems: the SBOM formats aren't settled, and an SBOM has to be dynamic, since a system drifts from its design over time, and has to account for partners and suppliers as well as your own code.

    Build Dependencies, Runtime and Noise

    Matty points to the report's finding that fewer than 1 percent of JavaScript packages are in use at runtime, and guesses it's because build tooling is written in JavaScript. The DevOpsDays site is a static site generator with a long package.json, none of which ships in the built artifact, yet Dependabot calls it "insecure as hell." Matty then catches a mistake in that reasoning live: the site does load Bootstrap on the front end, so there is front-end JavaScript after all.

    Michael adds that a website tends to accumulate JavaScript libraries, then marketing adds tracking and payment processing adds more. A scanner will list every dependency and known vulnerability, but it can't say whether the code is reachable at runtime, which is where Sysdig's runtime insights and "in-use exposure" come in. Without that, organizations are "flying blind" and taking a best guess at what is exploitable. Michael recalls engineering teams suppressing findings in open source libraries they don't own, which gets described as false positives, and Matty calls it normalization of deviance. In cloud native, with microservices, containers and ephemeral resources, the dashboard can be "a sea of red."

    What Shift Left Means

    Matty says "nuance is hard" and the shorter the phrase the more nuance it needs, and recounts writing a talk about shifting left securely out of frustration with a customer's sysadmins, on the idea that a title doesn't imply infallibility. Michael describes the traditional waterfall model with security as questionnaires and compliance, and shift left as pushing security into early design and automating tests in the IDE, at commit, in CI/CD and at runtime. Each stage produces scan results, which creates a correlation problem, and many of the problems of waterfall come back, just earlier.

    Matty's definition: "shifting left to me is not shifting the work to the people on the left. It is actually moving that domain expertise earlier in the conversation." That's why NoOps never happened, since ops turned out to be a domain of expertise, and expecting software engineers to absorb InfoSec is unfair and a bit insulting to security people. Michael says many of the calls at Gartner were about pushing scanning onto other teams because the security team couldn't scale, and notes that dynamic scanning tools are "glorified fuzzers" that need good test automation to reach the functions, and that release decisions and pass or fail builds are still unsolved. Matty adds that monitoring is "just testing with the time dimension," so a check in pre-production and in production should be in parity, and that the old hardening sprint produced a note from the security team saying it was okay, which bad guys on the internet don't care about.

    Matty says the tooling has improved: security scanners once cost around $15,000 a seat, and code instrumentation was limited by cost, so tracing ran on one of 30 servers. Both agree that doing this right means changing how product release and sales think, and not just engineering.

    Security as a Feature, Privacy and Zero Trust

    Matty asks whether organizations treat security as a product feature, in the sense of a secure product and not AuthN. Michael says marketing still wants security features, and that more training material exists through groups like OWASP. Privacy has brought more focus, which Michael traces to GDPR, and so has the US National Cybersecurity Strategy and SEC disclosure mandates.

    Michael builds zero trust from least privilege: zero trust is "least privilege on steroids," assuming the environment is compromised and authorizing continuously. It includes zero trust network access, which Michael says is what people usually think of, along with BeyondCorp, a cloudified VPN, and BeyondProd. Like shift left, people cling to one actionable piece. The report says "90% of granted permissions are not used," which Matty ties to onboarding: a new hire gets cloned from a colleague's access, the same way Chef-era teams copied an existing VM to get a new Apache server, until nobody knows what's on it. At Matty's current company, new hires get almost nothing and get annoyed, which works because the culture accepts 30 to 60 days of ramp up.

    Confessions

    Matty's Netflix password is a variation on the domain admin password from Apartments.com almost ten years earlier, and it was never changed when someone left, because in four or five years only one person who knew it left. "You can have bad password policies if people don't quit." Michael admits to simple passwords that a spouse can remember, since a 3-year-old leaves little time for a password manager. Matty closes with Windows NT 4.0, where passwords expired after 30 days with warnings at 15 and no minimum age, so the team built a Visual Basic app that changed the password 15 times and then back again. Michael will be at a Gartner Security and Risk Management Summit and on LinkedIn, and Matty ends with Gartner's "rogue sessions," where an analyst argues against Gartner's own position.

    • Sysdig 2023 Cloud-Native Security and Usage Report
    • Pushing Left With Tanya Janca (ADO epsiode)
    • Shifting Left Securely (Matt's talk)
      
      *DevOps World is back for 2023, and you won't want to miss out on this one-of-a-kind event! This year's program is packed with exclusive insights, immersive workshops, and unparalleled networking opportunities taking place across multiple cities in the US, UK, and Asia. Elevate your DevOps game and register using the following links: [NYC area](https://reg.rainfocus.com/flow/cloudbees/devopsnyc/webinar3/page/landing), [Chicago](https://reg.rainfocus.com/flow/cloudbees/devopschicago/webinar3/page/landing), [Silicon Valley](https://reg.rainfocus.com/flow/cloudbees/devopssiliconv/webinar3/page/landing), [Singapore](https://reg.rainfocus.com/flow/cloudbees/devopssingapore/webinar3/page/landing), and [London](https://reg.rainfocus.com/flow/cloudbees/devopslondon/webinar3/page/landing).*

24. DevOps With Better Marketing with Pete Cheslock

    29 de junio de 202357 minEpisodio 188A continuación

    Matty calls this the first "rogue session" of Arrested DevOps: a second look at platform engineering, after the episode with Daniel Bryant, with returning guest Pete Cheslock, who skipped that episode and comes …

    Notas del episodio

    Matty calls this the first "rogue session" of Arrested DevOps: a second look at platform engineering, after the episode with Daniel Bryant, with returning guest Pete Cheslock, who skipped that episode and comes in "unadulterated." The first half is Pete's and Matty's take on whether platform engineering is anything new. The second half is about Pete's side project of recording people pronouncing tech words. The cold open is Pete: "If charisma was an open source app or a product, it would be called Riz."

    DevOps With Better Marketing

    Pete's hottest take is "DevOps with better marketing." The immediate reaction was the unoriginal one: isn't that what you've been doing all along, building a platform for your team? Pete cites James Governor's tweet that "we've spent a decade rebuilding Heroku poorly." Matty traces the lineage back through PaaS, with Azure's original web and worker services model, Engine Yard and Heroku, all of which boil down to getting software running somewhere faster without worrying about the bits that run it. Matty also admits to owning and maintaining Pete Chess Bot, which lived only in Heroku, with the code never put on GitHub, until an unpaid Heroku bill got it shut down.

    Matty recalls an agile transformation in 2010 at Apartments.com, where agile coaches said infrastructure was a service that gets consumed and so didn't need to be in the conversation, which is "why the DevOps movement had to happen." Matty pushed for sysadmins embedded in squads, but with too few of them, each ended up on three squads with no time for real work after the ceremonies. When something can't be dedicated to one feature team, Matty says, an abstraction layer is the answer, and that's the evolution into platform engineering in theory.

    A Numbers Problem, and Nobody Owns the Product

    Pete has built this kind of thing three times in a decade: a knife command to spin up servers with Chef on EC2, provisioning bare metal with Chef at a DNS company, and a framework at ThreatStack where developers filled in the application's shape and committed code. The problem is a numbers problem, with about 100 developers and 3 systems people, and self-service is how to stop being the bottleneck. The missing piece at most companies, Pete says, is that nobody acts as product manager for the ops team, whose customers are the developers.

    Matty says that if you offer a platform, it's a product and you have to treat it as one, referencing a talk called Everything's a Product. A platform team may think adoption is guaranteed because the CIO chose the tool, but that attitude is what produced shadow IT. Matty also says a lot of platform engineering conversation is "incredibly Kubernetes-focused" and treats the runtime as the platform, while data and observability get left out, and then the platform team ends up integrating with 15 different ways to do Kafka. Pete says picking Kubernetes because everyone does is the same as "no one gets fired buying IBM."

    Pete ties it to what Pete calls "the DevOps hangover": 10 to 15 years of growth where nobody minded how much Datadog or AWS got consumed, and now someone asks about the Datadog bill and the cost of running Kubernetes, and half the team has been laid off. "I guess we should have just had product managers the whole time." Matty's cynical read is that a product owner on a platform team would be one of the first people laid off. Pete notes that product management is the least defined role Pete has seen, from mini CEO at some companies to owner of one feature at others.

    Rebranding and Silos

    Matty's worry is "the wrong way but faster," the Simpsons line about the max power way, where platform engineering is the SRE team, which was the rebranded ops team, which was the sysadmin team. A coworker once said of 12 years at the same desk and five different companies that the job never changed. Matty adds that "silos are okay" as long as domain experts work together earlier, and describes arguing about shift left and security on Twitter. Pete jokes that if platform engineering pays 24 percent more than SRE, Pete will be a platform engineer. Matty says that's good for the individual, but asks whether a team still doing tickets and requests on Kubernetes instead of vSphere is really treating it like a platform.

    Pete says the step still missing is requirements gathering, which is talking to the other people in the organization about what outcome they want before writing the first YAML: "It's always a people problem." Matty adds that Backstage is a tool for building the portal, not a platform, so you can't "rub some Backstage on it." Platforms are a socio-technical system, and buying one takes buy-in from the people who do the work and the executives who fund it, with the frozen middle in between. Matty says selling OpenShift at Red Hat was hard for that reason, at half a million to $2 million.

    Pete's story is a DNS company where "platform V2" became a trigger word. Pete wrote a ten-page product requirements document, renamed the project Honey Badger, and talked to every team to distill the minimum: provision a server anywhere in the world with a default operating system and a Chef role. It worked only with cross-team buy-in and budget. Matty also plugs a DevOpsDays Chicago talk on organizational politics.

    How Do You Say

    Pete's side project started with a long-standing idea to record friends pronouncing tech words in a game show format, and a Slack teasing about how Pete says a load balancer's name, for which Pete made up an origin story. Pete now works at AppMap, a 10-person company that let Pete record the series, with weekly posts across YouTube Shorts, TikTok, Twitter and LinkedIn. The first iteration is supercuts of about 20 words, about 30 seconds each, from 31 interviewees. Pete's favorites were SQL, epoch, fsck, which has about 20 alternative pronunciations on Wikipedia, and JWT, where the docs say it's pronounced "jot" and nobody Pete recorded said it, until some had implemented it.

    Matty says that's bad marketing, since nobody would search for jot. Pete says some pronunciations exist to help a listener type the word, like /etc and /lib, and that "all pronunciations are valid," since the goal was never to make fun of people. Pete wants more non-native English speakers for a season two, and a form for words and volunteers is linked below.

    Matty's advice on names is to ask everyone how they want their name pronounced, even names you think you know, which is what DevOps Party Games did, and offers "which one do you prefer" as a better framing than asking which is right. The episode closes on words that still trip people up: Pete's words like through and throw while reading aloud to a child, and Matty's arugula, which Matty replaces with rocket. Pete's daughter, after a show says a word from a book they read together, looks over and says, "wow, you weren't even close on that one."

    • Pete's Video Project
    • Pete's TikTok
    • Want to join a future version of Pete's videos?
    • Platform Engineering With Daniel Bryant (ADO Episode)

25. It's Rough Out There with Sidney Miller

    16 de junio de 202352 minEpisodio 187A continuación

    Matty talks with Sidney Miller, who has worked in talent acquisition for 24 years on the back-end infrastructure side, with a focus on candidate experience and on getting marginalized populations into tech. The …

    Notas del episodio

    Matty talks with Sidney Miller, who has worked in talent acquisition for 24 years on the back-end infrastructure side, with a focus on candidate experience and on getting marginalized populations into tech. The conversation is about layoffs, how to protect yourself mentally while out of work, pay transparency and how to negotiate. Matty opens with a disclaimer: the stories are "so cleverly disguised" that nobody should try to work out whether they're about Matty or someone else. The cold open is that same line.

    Who Gets Cut

    Matty describes hanging out a shingle as open to work about a year earlier and watching layoff waves roll through from the end of 2022, while the same companies kept hiring, which could be a left-hand, right-hand problem or "eventual consistency." Sidney says early layoffs hit functional roles like engineering, product and data science, and since March Sidney has seen many sales, tech talent acquisition and HR people on the market, which Sidney reads as a sign of overhiring and a course correction. Even experienced engineering managers are struggling, because the open roles are so specific that the ask is for a "purple unicorn with a T-Rex on its back shooting off machine guns."

    Sidney was laid off in September of the previous year from a Series B fintech when interest rates went up, was picked up at a Series A startup after leaning on Twitter, and was hit by a second reduction after that. The common denominator across all of these is that "when someone loses their job, they lose their, they feel like their value is gone."

    Not Taking It Personally

    Matty calls this "Rule 7: don't take it personally," a phrase from a former therapist, and says it's nearly impossible because so much of identity gets tied to the place you work. Matty recalls bleeding "Chef orange" at Chef, feeling betrayed when the company did things Matty disliked, and needing two jobs to learn the distance. Now Matty loves the company, but "the company is not me," and colleagues are coworkers: "we're not a family."

    On layoff decisions, Matty says the best performers sometimes get let go, since the best performer may be the most expensive, and describes an organization where people couldn't volunteer to be laid off in place of a colleague because the decision was about a function and not about performance. Matty adds that managers who had no part in the decision are the ones who have to make the calls, and that CEOs crying on YouTube about it are tone deaf.

    Sidney's advice for the mental side is to "give yourself space to process" before firing off resumes that aren't ready, to build a support system, and to find a cheerleader, because people in fight or flight end up on a hamster wheel. Sidney adds that a therapist can help. Sidney also notes that even with 24 years and a history of startups that were bought for billions, leaving still felt awful because the identity was wrapped up in "I built that."

    The Cost of Settling

    Matty describes the early 2000s, when a company acquired by GE laid the team off, the job search dragged on, and Matty took a cut of around 30 percent that took years to recover, since the next employer asks what you make now. Matty now takes recruiter calls for the information: asking for ranges, seeing what people are wanting to pay for a role, and watching people who need the work accept it. Matty suspects some companies see a chance to reset pay and get people cheaper. Sidney passes on advice from a woman CMO that it's okay to take a step back in uncertain times and cover what you need to, and that you can always keep looking while working.

    Pay Transparency

    Sidney says employers in California, Colorado, Connecticut, Maryland, Nevada, New York, Rhode Island and Washington are not allowed to ask what you made before, and that roles posted as remote that could be hired in those states tend to disclose a range. Matty says some ranges are games, like a posting spanning over $200,000 where nobody's paying the top number. Sidney says recruiters ask about threshold or expectation, which has nothing to do with last year's W-2.

    Matty walks through Talk Pay: Lauren Voswinkel started the #TalkPay hashtag in 2015, and Paul proposed an in-person version as an open space at DevOpsDays Austin that year. Matty has helped run them at many DevOpsDays and did two at Austin in 2023, anonymous but with people free to volunteer their number. Matty says people fear that their number is too low or too high, and that in practice nobody thinks the lowest number is the person's fault: "They think your employer is bad." The links below cover Paul's write-up and the Corey Quinn and Sonia Gupta talk on salary negotiation.

    Asking for the Range

    Sidney's tactic is to ask what the level is for the role and what salary corresponds to that level, instead of naming a number, because "the first person to name a number loses." If the answer is higher than expected, keep a poker face and celebrate privately. Hiring managers should not hand out different salaries to people doing the same job at the same level, which Sidney calls out where it happens.

    Sidney tells of a woman engineer whose offer came in $40,000 below what a same-level person in Portland, Oregon was getting. Sidney pushed to match the salary, and the engineer cried and said thank you. "It's not rocket science. It's influence."

    Matty adds that most of the audience can't yell at talent acquisition, so the lever is transparency from peers. Matty describes telling a friend exactly what the offer was, and a case where a friend applied to the team without asking. Matty's advice to people who aren't cis white men is to find someone who looks like Matty to coach them on having audacity: before pushing back on a wrong title in an offer letter, Matty asked a friend to "remind me to be a white guy in tech." The recruiter's reply was that it was a mistake and the DocuSign needed regenerating. And if an offer is withdrawn because you asked for more, Matty says, "good."

    Start and Stop

    Sidney's last advice is to stop telling yourself you aren't worthy of the next opportunity, to process what happened through a support system, to ask for the level and the band, and to be as confident as possible. "All they can say is no. You got a 50/50 shot." Matty says a follow-up episode is likely.

    • Sonia Gupta and Corey Quinn - Embarrassingly Large Numbers: Salary Negotiation for Human
    • Talking pay in the public square

26. Data! Data! Data! with Francesco Tisiot

    1 de junio de 202347 minEpisodio 186A continuación

    Matty talks with Francesco Tisiot, a data professional with 15 years in the field, including 12 years of consultancy across a wide range of sectors and about ten of those with big enterprises. The conversation …

    Notas del episodio

    Matty talks with Francesco Tisiot, a data professional with 15 years in the field, including 12 years of consultancy across a wide range of sectors and about ten of those with big enterprises. The conversation covers how data moves through a company, what event streaming and change data capture are, why data sprawl is a technical, financial and security problem, and four directions for judging whether a data platform is robust. The cold open is Matty, after Francesco explains replaying events from Kafka: "I'm still stuck back in 1999."

    Data Is a Journey

    Matty asks how the field has changed from the days when the DBA was the person in the closet. Francesco picked data because the work sits between computers and people: analysts, scientists and engineers translate business expectations into models and documents. "Data is data, but how you think about it, how you make sense of the data, changes every time you interact with different people."

    Francesco frames data as a journey, using a company that sells shoes online. A purchase lands as a record in a transactional database. A startup then runs analytics queries against that same database to compare today's sales with yesterday's, which works at small scale. As the company grows, those queries make the transactional database suffer, so the data gets moved into an analytical database, and one piece of technology becomes a chain of them. Francesco notes the chain follows the growth of the company, including change data capture, event-driven architecture and analytical databases.

    Batch, Streaming and Change Data Capture

    Francesco explains the old batch approach: wait for the night when the website is offline, extract everything from the transactional database, and load it into the analytical database to feed the data marts and warehouses. That still suits reporting on last month's data. It doesn't suit cases that need an immediate reaction, such as reordering inventory when ten pairs of shoes sell. There the work moves from batch to real-time or near real-time, per event. With Kafka, a streaming technology, a change in the database can be propagated to downstream systems, for example an application that compares current stock against a data scientist's prediction.

    Change data capture tracks changes in the database by reading its logs, without continuously querying it. Francesco says it lets a company evolve from batch to event-driven without touching the transactional database or the application in front of it, so the business keeps running while the backend changes.

    Losing the Map

    Matty asks how large organizations keep their data interactions aligned. Francesco describes enterprises where you don't know about a data mart sitting there, or who purchased a tool, because "people come and go and company remains and data remains and, you know, bills remain." With non-technical users building analytics through point and click, there is no longer a single view of reality. Francesco admits losing a ten-year battle against people exporting transactional data into Excel, and says what matters is a global view of which data assets and pipelines exist and how they connect. "If you don't have the map, you are lost."

    Without the map, a company can pay a consulting firm six months later to solve an inventory problem it already solved, or let two teams solve the same problem and end up with different answers because of a small detail in a KPI definition. Francesco adds the financial and security side: GDPR aside, loose control over where data lands invites someone to dump a database export into a bucket by mistake.

    Matty asks whether anyone does this well. Francesco has seen two approaches work. One is documentation, which is risky because it's an afterthought, though Francesco wonders if ChatGPT could help with parsing it. The other is a single tool for all transformations, such as Informatica, which can give column-level data lineage, but data now spans many technologies and teams. Matty says relying on documentation and training is the worst way and prefers guardrails that make the right way the easy way, and notes that Kubernetes is great until you want state.

    Francesco sees a way forward through metadata. Within one database like Postgres, catalog views list tables, users and permissions. Between systems, the gap can be closed by parsing the configuration that connects them, for example the JSON payload of Kafka Connect, which says where data comes from and where it goes. Automated tooling can describe how a pipeline was built but never why, so the reasoning behind a KPI, such as using the last six months of sales instead of three, still needs documentation.

    Four Directions for a Robust Data Platform

    Francesco's framework has four directions, and the episode's existing links include the blog post on it.

    • Scalability. It covers technical scale, such as going from one node to a cluster of 50, and whether the technology can segment all the business cases. It also covers whether the humans can scale, since a technology with no talent pool becomes a problem in five years, and financial scale, since a perfect solution that costs twice what each user brings in is not affordable. Francesco adds that data platforms are sticky and few backend migrations succeeded.
    • Observability. Metrics, alerting and notifications matter because the data team is offering a service within the company. A bird's-eye view also helps when GDPR queries arrive, since you know who to ask. Francesco asks why versioning shouldn't apply to data too, with a way to replay yesterday's data after a mistake.
    • Fast. Speed covers time to develop, time to deliver and time to recover from errors. A dashboard built in a day is not worth it if you always wait 23 hours for the right data.
    • Trustworthy. With one analyst, that person's Excel file is the source of truth, but with 55 analysts there is no number anymore. The company needs a single golden truth, delivered on time, because a KPI that is right at 9 AM one day and 1 PM the next erodes trust. Data also needs to be secure, so nobody who shouldn't see or alter it can.

    Francesco says some of these will bite immediately and some in the long term, and looking at all four lets you compare technologies and make a 360 degree decision.

    Replaying Data With Kafka

    Matty finds versioning and replay hard to imagine, since even restoring batch data to a point in time was nearly impossible. Francesco describes Kafka as a log: events are written one after the other, and reading doesn't delete them, so another application can read the same messages. Add schemas, and Kafka refuses messages in the wrong format. Schema evolution lets you change the schema while existing consumers can still parse messages. In the example, a boolean for putting initials on the shoes is added, billing ignores it and the printing team uses it. If the flag later needs to be a color, the log can be kept for days or years and a new set of consumers can be fed the data generated since yesterday, five hours ago or an hour ago.

    Misconceptions

    Asked about what developers and infrastructure folks get wrong about data, Francesco says the biggest misconception is that data systems are old and boring. The data world is hot, and ChatGPT and AI are data. Innovation with data depends on quality data, which is the biggest problem to solve everywhere, and "Data pipelines are cool."

    Francesco is on Twitter as @ftiziot, which is consistent almost everywhere apart from LinkedIn, and the DMs are mostly open. Francesco also says the main mission in life is telling people how to properly eat Italian dishes, which for example excludes carbonara with cream, pineapple on pizza and cappuccino with lunch or dinner.

    • Kafka Connect tool
    • Metadata parser
    • Francesco’s SOFT blog post

27. Platform Engineering with Daniel Bryant

    18 de mayo de 202349 minEpisodio 185A continuación

    Matty talks with Daniel Bryant, who runs the DevRel team at Ambassador Labs, about platform engineering: what it means, whether it killed DevOps, and how to approach building a platform. Daniel started as a …

    Notas del episodio

    Matty talks with Daniel Bryant, who runs the DevRel team at Ambassador Labs, about platform engineering: what it means, whether it killed DevOps, and how to approach building a platform. Daniel started as a Java engineer, did software architecture and then operations, and built platforms on Mesos and Kubernetes. Matty refers back to an episode recorded in January 2016 with Kelsey Hightower and Andrew Clay Shafer that talked about platforms before the buzzword existed. The cold open is Matty's opening to a topic that might be on the tips of listeners' tongues.

    What Platform Engineering Is

    Daniel's formal definition is the discipline of building toolchains, workflows and platforms to support the team going from idea to observable business value in production. Continuous delivery doesn't end when the app reaches production, since feedback from a business or operational point of view has to come back. Matty says that means Kubernetes alone isn't a platform, and cites James Governor's line that everyone is trying to build their own Heroku.

    Is DevOps Dead?

    Matty asks about "DevOps is dead" marketing at KubeCon. Daniel, with 20 years in Java where "Java is dead" recurs, says a shock headline draws attention, and that engineers want to bury the previous generation to look like they're innovating, while history rhymes. Matty says the question is what you mean by DevOps: if a DevOps team is an automation team that builds infrastructure, platform engineering replaces that, but the principles have been the same throughout. Matty adds that DevOps takes research seriously, and the research behind some of that marketing was talking to three companies.

    Platforms Versus Portals

    Daniel says some people say IDP for internal developer platform and others for internal developer portal, and the difference matters: Backstage is a great jumping-off point, but Daniel sees it as the UI, CLI, SDK and API on top of a platform. So Daniel asks whether people mean a platform soup to nuts or just a service catalog. Matty likes the idea of a service catalog that catches the 80 percent case and lets you do other things at a cost, and raises Charity Majors's maxim that the best tool is the one you don't need and the second best is a SaaS, asking whether there's any SaaS for this, and whether Heroku was the closest.

    Daniel says Heroku and Cloud Foundry were built for web monoliths, but now the 80 percent includes machine learning apps, front ends and systems of record. Spotify talks about golden paths, such as one for machine learning apps, one for microservices and one for front ends, so there's no longer one true way and a one-size-fits-all platform hits at most half the apps. Matty says that makes it a big ask to build an internal SaaS, treated as a product.

    Thinnest Viable Platform

    Daniel borrows a phrase from Team Topologies, the thinnest viable platform. Large organizations like Intuit, which has a global team supporting its platform, can justify it, but for a startup of three people without product-market fit, "please don't build a platform," and something like Cloud Run or Knative with GitHub Actions is enough. Matty says the tech is the easy part, the hard part is how people communicate, and cites a line that source control is a communication tool for developers. Matty also cites Adam Jacob's point that tools influence culture and culture influences tools, a variant of Conway's Law. Matty says large platforms like OpenShift or Tanzu are hard to start small with and get decided at the CIO level, and Daniel says the cognitive load is high, while bottom-up requirements plus mid-management buy-in and then exec support works better than the old golf-course selling.

    Matty says to begin with the end in mind, without solving everything, but avoiding decisions that make it impossible later. Daniel says this is the role of a platform product owner, who starts small and thinks big.

    Who Builds It and Where It Sits on the Curve

    Daniel says platform teams are often drawn from infrastructure or developer experience and enablement teams, and often a pain triggers them, such as an exec discovering 10 versions of a platform and nobody who knows how to maintain version 9. Matty says this is the early adopter part of crossing the chasm, and that typical enterprises haven't arrived, recalling arguing with PagerDuty's founder that enterprises still have production support teams and calling it survivor bias. Matty worries the late majority skips the pre-work and just renames teams: tech ops became cloud ops, then DevOps, then SRE, then platform, with the same remit, and says "get your bag" to anyone getting a better title.

    Backstage

    Daniel describes Backstage as a jumping-off point for an internal developer portal, with a service catalog, automation hooks, search, who's on call and ownership, and TechDocs for living documentation. Spotify sells extensions, and Roadie offers Backstage as a service since it's hard to install, and many people use it as a facade or for inspiration, for templates that spin up a new service with observability and security baked in and links to dashboards.

    Do and Don't

    Daniel's three keys are treating the platform as a product, since you can't have good developer experience without good user experience, focusing on workflows and tool interoperability, and making it composable. The thing not to do is buy a platform. Matty adds "you can't buy DevOps, but I can sell it to you." Daniel says late adopters worry about being left behind and throw money at it, while it's better to read the many blog posts from companies that have built platforms and spend time understanding the problem space first.

    • ADO Episode - Platforms with Kelsey Hightower and Andrew Clay Shafer
    • Daniel’s Kubecon talk
    • Daniel’s blog/tweet thread the kicked off his interest in Platform Eng
    • Spotify golden paths
    • Crossing the Chasm - Technology Adoption Lifecycle
    • Backstage project
    • Backstage as a service

28. Into the VOID Report with Casey Rosenthal and Courtney Nash

    13 de abril de 202360 minEpisodio 184A continuación

    Matty, coming back for a new season in 2023, talks with Casey Rosenthal and Courtney Nash of Verica about the VOID, the Verica Open Incident Database, and what its research says about incident metrics. Casey is …

    Notas del episodio

    Matty, coming back for a new season in 2023, talks with Casey Rosenthal and Courtney Nash of Verica about the VOID, the Verica Open Incident Database, and what its research says about incident metrics. Casey is best known for chaos engineering: Casey wrote the definition with a team at Netflix, started the conferences and the community broadcast, wrote the book on it, and is now CEO of Verica. Courtney is a former chair of the Velocity Conference who worked at O'Reilly, Amazon and Microsoft before taking a research role at Verica, and calls themself "the world's only internet incident librarian." The cold open is Courtney: "We've got a bunch of data that have allowed us to bust a few myths."

    What the VOID Is

    Courtney started collecting incident reports while doing product research on Kubernetes and Kafka, looking for non-marketing information on how they fail in the wild. Having gathered about 1,000, people said thank you and "we have more of that," and the database and its metadata emerged in late 2020, with a first report in 2021 and a second more recently. The database takes a broad view of incidents: postmortems, status page updates, tweets and media articles, anything where someone talked about a website or service falling over. It also collects metadata such as how long the organization said the incident lasted, severity and methodology. Courtney wants data, not abstraction, to address myths about software incidents.

    The VOID focuses on availability incidents since security breach databases already exist. Courtney says the DevOps mentality of sharing failure is a long way off in security. The analogy Courtney uses is the US airline industry in the 1990s, where the push to share incidents came from pilots, not regulation, and they set competitive concerns aside. Courtney and Casey both worry about regulation of software, with Courtney citing the call to regulate software after the Southwest Airlines meltdown, which was an organizational and cultural problem, and Casey calling it a personal nightmare. Casey says the VOID can help redefine the value of availability work, be it incident analysis, response or chaos engineering, and that availability and security are two sides of the same coin for system safety.

    Why MTTR Is Junk

    Courtney explains that MTTR comes from physical manufacturing, where widgets wear in predictable ways with a normal distribution of repair times. Software incidents don't look like that: the duration data are heavily skewed, with a big bump under an hour or two and a long tail, so "you can't take averages of that. It's just garbage." An engineer from Google wrote an O'Reilly report that ran Monte Carlo simulations on incident durations, shortening some and taking averages, and the results were a mess. The VOID team did the same with its own data and got the same results. Courtney says people react either with "oh shit" or "oh shit, but I'm going to fight you on it." Casey adds layers: the statistics are wrong, the data coming in is garbage because methods for determining how long an incident lasts are fraught, and you'd need more data points than Google has.

    Matty says the metric is used to report on the performance of people within a quarter and is about the closest thing to putting a Nagios counter on your people, and recalls incident command workshops where people said they spent their time on mean time to innocence. Courtney asks what decision you would make based on it, since either way you'd have to go look at what's happening in the system and the people operating it. Courtney says the industry's data-driven obsession overlooks data about human behavior, which is still data.

    Bureaucracy and Taylorism

    Casey says organizational researchers call software the bureaucratic profession, bought wholesale from manufacturing and scientific management. Every role, such as people manager, project manager and architect, takes responsibility for expertise away from engineers, which makes sense on an assembly line and is the wrong model for knowledge work. The business must change if it wants availability, resilience and security. Courtney's line: "Taylorism is a corporate disease that we haven't developed a vaccine for yet." Matty recalls the Upton Sinclair quote about salary and understanding, and the frozen middle, where people's jobs exist because of metrics like MTTR.

    Courtney says reliability and security are now central to business concerns, as Southwest showed. Casey describes how every Netflix engineer knew how to look up stream starts per second, one metric correlated with value, and Netflix found that going from four nines to five was pointless because users' Wi-Fi and ISPs wash it out, so they invested in regional failover. Courtney says to find the core metric according to your business, and Matty notes public sector people know their mission better than private sector people know how their company makes money.

    What to Use Instead

    Courtney says what replaces MTTR is the recognition that no single metric captures reliability, and that software systems are sociotechnical, so you need people skilled in social science methods: interviewing, collecting stories, building narratives and finding patterns. Organizations are hiring incident analysts, and Courtney believes the ones who invest will gain a competitive advantage, though the data for that is further off. Courtney's favorite example is an organization in the CIO's office at IBM, described in a DevOps Enterprise Summit talk, where a skunkworks team did quality incident analysis, a bad incident gave the opening to try something different, and now monthly learning-from-incident reviews draw hundreds of people. Courtney also notes that the Microsoft Azure team changed its public reports from RCAs to post-incident reviews and the quality and depth improved dramatically.

    Matty tells how a PagerDuty SRE quietly changed the word "root cause" in the postmortem template without asking permission, nobody objected, and suggests the homework for PagerDuty users: change the template to contributing factors. Casey "fully supports small acts of vandalism." Courtney's last finding is that the VOID data shows zero relationship between an incident's duration and its severity, and Matty and Courtney note severity is negotiable and changes during an incident. Casey says the whole model of severity is wrong at scale, since some group of users is always unable to reach your system.

    • VOID
    • The Verica Open Incident Database (VOID) makes public software-related incident reports available to everyone, increasing understanding of software-based failures in order to make the internet a more resilient and safe place. After scrutinizing nearly 10,000 incidents, one thing is crystal clear: Resilience saves time. Taking the time to understand how to better respond when something green turns red—learning from the people, the processes, and the systems—will make your next incident smoother.
    • 2022 VOID Report
    • “Taylorism is a corporate disease that we haven’t developed a vaccine for yet” - Courtney

29. Continuous Feedback with Roni Dover

    20 de septiembre de 202251 minEpisodio 183A continuación

    Jessica Kerr talks with Roni Dover, a developer who has also worked as a product manager and describes oscillating between the two, about continuous feedback as the missing loop in DevOps. Roni is a board game …

    Notas del episodio

    Jessica Kerr talks with Roni Dover, a developer who has also worked as a product manager and describes oscillating between the two, about continuous feedback as the missing loop in DevOps. Roni is a board game fan and a self-described skeptic, and started an open source project called Digma to put the idea into practice. The cold open is Roni on code: "the tales that this code could tell, if only it could tell what happened back when it was, you know, used or abused."

    Optimizing for Speed Only

    Roni says development processes try to optimize for speed of deployment, cadence and time to release. If you only optimize for speed, "you're just creating a system where you're hurling features over the fence faster," 24 times a day instead of once a month, without improving the learning or the feedback. Jessica points out DevOps is supposed to keep caring after production, and Roni says the tools for that look for problems, which is reactive and generic. As a product manager Roni had tools like Google Analytics showing the impact of a decision, such as whether a navigation change increased adds to cart, and didn't feel like they were running blind, while developers had CI, CD and testing tools that say nothing about impact or performance in production. Jessica calls the missing piece observability, and adds it must be more than monitoring.

    The Inverse Pipeline

    Roni describes continuous feedback as the inverse of continuous deployment: the DevOps loop takes code from source to production, while continuous feedback starts with information from production, goes through stages to work out what's relevant, and ends back in the developer's tools, including the IDE. Roni stresses it's not actually linear, since feedback also exists before you start coding. Code ownership has grown from "done when I sent it to QA" to owning tests and deployment, which Jessica compares to moving from owning a car to parenting, where you want to know how your kids did in kindergarten.

    Roni's examples of what code could tell you: how heavily the code is used and whether it's a bottleneck in a high-concurrency environment, which tells you what to optimize, whether it runs in production at all, which Roni has seen surprise teams who invested three years in a feature whose code path was never reached, and how it scales with concurrency, database size or payload. It could also report runtime errors, such as whether a "should never happen" branch does. Roni adds that developers misuse logging for this and forget to check.

    Biases and Why It Doesn't Happen

    Roni says continuous feedback makes the organization a learning process instead of a shipping process, and stretches the definition of done. Without it you accumulate technical debt and end up "running around, putting out fires." Roni cites biases: estimation anchoring and optimism bias, and confirmation bias in tests, which codify expectations and miss things nobody thought of. Observability injects relatively objective data, and "if you know about a bias, it seldom helps you actually overcome it." Jessica adds that combinations of features, data and ordering go well beyond edge cases.

    Roni says very few engineering organizations actually practice continuous feedback, for three reasons: engineers are busy and can't keep looking for trouble in logs and dashboards, not all have the expertise, and they get data, not insights, and context switching is costly. Roni says an insight would be that this is a bottleneck and why, with a way to double-click for more.

    Digma and OpenTelemetry

    Roni started Digma, which is open source and entering beta, to tackle those three issues by codifying the tribal knowledge about how to measure latency and read time series, bringing it into the IDE so there's no context switch, and making it proactive, so the code sends "life signs" after it ships. Roni also wants to celebrate wins and not make observability all about blame. Readers can sign up at digma.ai, and mentioning the podcast gets a bump up the beta list.

    Roni says OpenTelemetry was pivotal because everyone agrees on it, with vendors aligning around it and a spec that lets new open source tools make the data more useful. The libraries auto-instrument code, so getting from no telemetry to useful data is quick. Digma works as a pipeline and not an APM, ingesting OpenTelemetry data, scanning the code to correlate data to locations, and, if you add the commit ID via an environment variable in CI, relating insights to the code change that precipitated them, which Roni describes as a matryoshka design. Roni would like shorter loops, so adding a trace gives feedback in testing, CI and staging quickly. Jessica relays a friend's wish for something that says there's an N+1 query right here, and Roni says that's exactly Digma's point.

    Roni says a developer wants control, not a 2 AM call three days after a push, and had written a blog post called Breaking the Fourth Wall, about code that talks back. The biggest risk is being spammy or sending people on a wild goose chase, so the feedback has to be accurate and pertinent. Jessica says to test in production as well as before it.

    Roni is @DoppleWare on Twitter and writes on Medium, and the favorite board game Roni mentions is New Angeles, though Roni rarely plays the same game twice, since repeat plays become rule hacking.

    Jess and Roni talk about what continous feedback: where it came from, what it looks like in the context of a dev proces, and the benefits it can bring to engineers and developers. They also discuss Roni's observability project, Digma.ai... and his other passion, complicated board games.

30. Engineers are People with Dagna Bieda

    21 de marzo de 202249 minEpisodio 182A continuación

    Matty talks with Dagna Bieda, a software engineer turned career coach with over ten years of coding and more than three years of coaching, about communication, empathy and the people side of engineering. Dagna …

    Notas del episodio

    Matty talks with Dagna Bieda, a software engineer turned career coach with over ten years of coding and more than three years of coaching, about communication, empathy and the people side of engineering. Dagna has coached clients from large companies and small ones, with two to twenty years of experience and backgrounds from self-taught to bootcamp to college. The cold open is Dagna: "It's important to realize that at the end of the day, we're people working with other people to create products for other people."

    Clarify Assumptions

    Dagna says everyone comes to a conversation with assumptions from family and culture, and clarifying them is a key communication skill. There are two kinds: the ones you hold yourself, such as how many people and how much time a feature has, which you can state, and the ones you make about what others assume, where it's better to ask and overcommunicate, because "there are no stupid questions." Dagna recommends starting with an "I sentence" to create a safe space: "I didn't get it. Why don't you give it another go?" so the other person doesn't get defensive. Matty says that when you keep saying people just don't get it, that's on you, and Dagna says you can't control other people's actions but have an impact on how they behave, and if you think you work with idiots, people sense it.

    Empathy as Understanding Priorities

    Matty asks how to operationalize empathy, recalling the 2014 devopsdays talks that were all about it and a talk title, "You've Convinced Me We Have to Collaborate Now. How the Hell Do I Deal With People?" Dagna says empathy is misunderstood as having to feel what others feel. What works in the workplace is understanding other people's priorities: an engineer wants to fix bugs, implement features, learn and have fun, while a project manager cares about business value delivered on time with no disruptions and isn't moved by refactoring without a tangible change. Dagna gives clients a worksheet to map stakeholders' priorities, and says pitching an idea by showing how it fits their priorities is "how brilliant engineering ideas get actually implemented." Matty says it's like a talk called The 5 Love Languages of DevOps: a change that makes sense to me might land differently for you, like 30 milliseconds faster matters because of an SLA.

    Assertiveness

    Dagna's own stuck point was assertive communication. Dagna is Polish and direct, and recalls a company-wide meeting of around 300 people after layoffs where Dagna tried to raise concerns with leadership with good intentions, and the boss asked why Dagna had called the entire leadership team idiots, which Dagna says was not what was said but how it came across. The lesson is that intent and how you come across can differ completely. Assertive communication means talking about your needs and wants respectfully so others feel heard. Dagna says probably every senior engineer sounds like an asshole now and then by being too direct, and that humans aren't rational beings who only look at facts.

    Step Outside the Forest

    Matty says systems are complex and have ramifications beyond your microservice. Dagna says engineers are so deep in the woods that they see one tree, while a manager sees the health of the project, a director sees groups of trees and the C-level sees the whole forest, and stepping outside the forest is critical for career advancement: "what got you to that senior position is not going to get you past that position." Once your technical foundation is solid, soft skills matter more. Matty points to earlier episodes on principal engineers and says even engineers who became engineers to avoid people deal with people.

    Limiting Beliefs and Marketing Yourself

    Dagna says the way you think impacts how you act, and compares the subconscious to background programs running on an operating system: people around you growing up "install beliefs" with a sudo command. A common engineer belief is that their work speaks for itself, and it doesn't, since nobody sees it unless they pull the repo. Dagna says marketing your work is a communication skill, and changing a belief is possible because people keep changing. Matty adds that ops roles have the same visibility problem as a corporate lawyer, and recommends knowing how your company makes money. Dagna tells of two features: a refactor that cut a mobile build from about 4 minutes to 20 seconds, interesting but affecting two engineers, and a boring build for a huge client that got praise from the boss's boss, the sales rep and the client. A conversation with the manager showed where the impact was, so ask how features affect the business. Dagna adds that if you can't ask your manager that, run. Matty recalls not knowing for years what the bank's treasury services line of business did until reading its intranet page.

    Values and Interviews

    Dagna's first coaching step is figuring out a client's values and the last is finding companies that share them, so you aren't reacting to job postings. Matty notes that interviews are full duplex: the candidate is also evaluating the company, and interviewers are usually untrained. Dagna says candidates who ask questions back are people Dagna wants to hire, and that confidence in your skills lets you negotiate, like the six-month sabbatical Dagna negotiated four months into a startup job, which was spent traveling across Southeast Asia.

    Dagna's one recommendation is to ask more questions and put assumptions on the table. Dagna's coaching site is themindfuldev.com, with a case study video, and the two agree to do a follow-up on confidence.

    • linkedin.com/in/dagnabieda
    • www.themindfuldev.com
    • How Dagna works with her clients
    • https://www.arresteddevops.com/principal-engineer/

Aquí están los 40 episodios más recientes de 211. Los anteriores están en el feed del podcast.