LLM-as-a-Judge, Quotation Fidelity, and A/B Testing Models: AI Publishing at Scale
Shownotes
What happens when a data scientist builds a generative AI proof of concept — and it scales to 700,000 articles and 4 billion page views? Recorded live at AWS Summit London, Romain is joined by Lewis James, Senior Data Scientist at Reach PLC — the UK's largest commercial publisher with over 120 brands including the Mirror, the Express, and OK Magazine. Lewis shares the full journey from GPT-2 experiments to a production AI publishing platform called Launchpad that now assists with 20–30% of the portfolio's daily article output. We explore how the team earned journalist trust by focusing on mundane tasks first, how they built multi-model pipelines with quotation fidelity checks to avoid misquoting, and why working backwards from users — not pushing technology — drove adoption where others failed. The conversation covers the technical evolution from prompt engineering to fine-tuning, model distillation, and agentic workflows built with the Strands Agents SDK running on Amazon Bedrock AgentCore. Lewis also introduces the concept of 'vibe publishing' — giving journalists a chatbot interface with more creative freedom — and discusses how evaluation strategies differ when you're measuring editorial tonality versus factual accuracy. Whether you are building AI-assisted content pipelines, navigating enterprise AI adoption, or thinking about how to earn user trust for generative AI tools, this episode offers a rare look at what three years of production generative AI looks like at massive scale.With Lewis James, Senior Data Scientist at Reach PLC