Your AI Is Live. But Is It Actually Working?

In this episode of Data Analytics Chat, we welcome Ryan Stevens, Senior Director of Applied Science at Ramp, the financial technology company helping businesses save time and money across cards, bill pay, expense management, travel and more. Ryan leads a team of machine learning engineers and product data scientists, and before Ramp he worked on ads ranking at Meta. He also holds a PhD in economics.

Ryan shares how Ramp evaluates AI once it is live, from structured outputs like invoice OCR to unstructured chat agents like Stack, Ramp's AI accounting agent. The discussion examines why labels sit at the heart of every evaluation problem, how synthetic data can stand in before launch, and why so many teams reach for an agent before they actually need one.

The episode explores the three layers of production monitoring (errors and latency, business outcomes, and adoption), how much evaluation can be handed to LLM-as-a-judge versus human reviewers, and how leaders can tie evaluation back to business value by starting with the task and weighing its value against the cost of solving it.

00:00 Cold Open
01:02 Introduction to Data Analytics Chat
01:37 Meet Ryan Stevens and Ramp
02:59 How Do You Know Your AI Is Actually Working?
08:29 Accuracy, Quality, Safety or Business Outcomes?
11:26 Why Is AI Evaluation So Hard?
14:29 Has Generative AI Made Measurement Harder?
16:42 Good Evaluation Before Going Into Production
19:13 You Probably Don't Need an AI Agent Yet
20:01 Evaluation at Enterprises vs Startups
23:00 How Evaluation Changes Once a System Is Live
25:39 How Often Should Teams Check Performance?
27:05 Early Warning Signs Your AI Is Failing
28:11 Can AI Really Evaluate AI?
30:12 Using AI Agents to Test and Monitor Other AI
32:11 Your AI Works Technically. But Does It Create Value?
34:39 What a Strong Evaluation Process Looks Like
37:20 The First Thing to Put in Place

Connect with our guests:
Ryan Stevens: https://www.linkedin.com/in/ryanstevens3/

Thank you for listening!