Generating Opportunity Solution Trees with AI: How Vistaly Rebuilt Its Product Around Interview Synthesis, Evals, and Re

Guests

• Matt O'Connell, Co-Founder and CEO, Vistaly
• CP Dehli, Co-Founder, Vistaly
• Steve Klein, Co-Founder, Vistaly
• Teresa Torres, host of Just Now Possible, building the AI synthesis services for Vistaly V2

Key Takeaways

• The competition isn't good interview synthesis done faster—it's shallow synthesis that never really happened. That changes what "good enough" means.
• Layered analysis matters more than whether a step is a pipeline or an agent. If the interview snapshot is wrong, every layer above it inherits the error.
• Prompt changes run out. Balancing two opposing error modes took an orchestration change—moving the check into an agentic repair loop—not a better prompt.
• Change sets can't be diffed at the end. The same input/output tree pair has multiple valid change sets, and only the semantically meaningful one makes sense to a user.
• Users don't want to collaborate with the AI step by step. They want the answer, then the ability to correct it.

In this episode

• Why Vistaly rebuilt from scratch instead of layering AI features onto V1—and shut off V1 signups to protect the rewrite
• Rebuilding nearly all of V1's functionality in two and a half months, after three years of building it the first time
• The V1 chat interface that walked users through insights one at a time, and why "still faster than before" wasn't fast enough
• How V2 works: upload three interviews, generate an interview snapshot for each, then synthesize a first-draft opportunity solution tree
• The "house of cards" problem: garbage in at the snapshot layer corrupts the tree above it
• Using a code assertion (too many children on a node) as a cheap pre-filter before spending money on an LLM as a judge
• Four new evals and 16 experiment variants to break a seesaw between missing subgroupings and badly framed parents
• Why the fix was a repair loop in the orchestration, not a better prompt—and how that eval became a production guardrail
• Teaching the agent "the rules of the game" so it logs semantic moves (merge, move, reframe) and produces reliable change sets
• Stable IDs, data provenance, and the "basic computer science" Teresa swept under the rug in the prototype
• What beta data revealed about interview quality: sales demos, stakeholder meetings, and LLM-generated fake transcripts uploaded as "interviews"
• Building a 40-point system for generating realistic synthetic transcripts as test data
• Driving tree edits through MCP, including mapping a full Jira epic backlog into the opportunity space
• Adding an agent back to the canvas, and the versioning and rollback problem that comes with it
• Data residency, SOC 2, GDPR, and moving to European Bedrock inference
• Why model upgrades aren't drop-in: prompts are model- and version-specific, and Bedrock constrains tokens independently of the model

Resources & Links

• Vistaly — Opportunity solution tree software with AI-generated interview snapshots and tree synthesis
• Product Talk — Teresa's AI evals guide and her blog posts on building the Vistaly synthesis services
• Model Context Protocol — How Vistaly users drive tree edits from outside the product
• AWS Bedrock — Where Vistaly runs inference, including EU-hosted inference for European customers
• Just Now Possible episode with Banani — the canvas-plus-agent design UX Teresa references
• Just Now Possible episode with Zencity — the layered data pyramid Teresa references
• All Things Product episode on the quality of evidence, with Petra Wille

Chapters

00:00 Meet the Founders
01:05 How Vistaly Began
03:10 Community First Partnership
04:56 Why Build OST Software
07:18 OST Explained Fast
10:28 Evidence Inside the Tree
12:56 AI Changes Discovery
16:58 Chat UI Experiment
23:00 Should AI Do Synthesis
27:10 Rebuilding as V2
32:08 Turning Off V1 Signups
34:43 V2 Workflow and Tensions
39:35 Trusting AI Outputs
39:56 Layered Analysis Pyramid
41:25 Building Better Trees
43:34 Bad Inputs and Coaching
47:20 Teaching Through Software
50:12 Evals and Guardrails
55:23 Deterministic Change Sets
01:00:54 UX for Tree Updates
01:07:12 Chat Agents and Versioning
01:11:17 Privacy Compliance Tradeoffs
01:15:06 Model Selection Reality
01:17:32 What Comes Next
01:19:08 Closing and Links