Saltar al contenido

How a Voice Agent Learns the Rhythm of Conversation — Shawn Wen

1 de octubre de 2026 · 1 h 10 min

Notas del episodio

Tsung-Hsien (Shawn) Wen, CTO of PolyAI, tells Tim Scarfe why voice agents are harder than text agents. Voice adds time, and a good conversation depends on adapting to the person on the line, not just on reasoning to the best answer. Shawn describes an audio-native model (Dialog-RSN-1) that first predicts a turn-taking signal, then replies in text with citations, and writes the transcript last so enterprises can audit it.Along the way: training on real, noisy calls with synthetic noise added, and why over-cleaned audio made the new model worse. Latency, and what a voice agent should do while it thinks. Why a voice with a hint of regional accent beats a generic one. Why public benchmarks fall short for voice, why enterprises want to own their agent harness, and whether behaviour belongs in the harness or in the weights.The last stretch is about working with agents: cognitive debt, the shift from producing content to checking it, Wispr Flow, building tools that agents can use, and whether slop is in the eye of the reader.This episode was produced in partnership with PolyAI.https://poly.ai CHAPTERS0:00 Why voice agents are harder than text4:27 What enterprises want, and why PolyAI built its own model9:04 How an audio-native voice model works15:21 Training data, spectrograms and synthetic noise20:36 The cocktail party problem and the future of turn-taking25:21 Latency, adaptive reasoning and keeping callers' trust31:33 Voices, personality and the uncanny valley36:32 How do you benchmark a voice agent?42:00 Harness engineering and owning the intelligence45:09 Well-specified problems and auditable agents49:46 Weight adaptation and cognitive debt56:31 Agents at work: Wispr Flow, voice and tool building1:02:50 The next decade of voice, and what counts as slopREFERENCESThe Bitter Lesson: http://www.incompleteideas.net/IncIdeas/BitterLesson.html [9:05]Retrieval-augmented generation: https://arxiv.org/abs/2005.11401 [13:23]Mel scale: https://en.wikipedia.org/wiki/Mel_scale [17:16]Victor Zue: https://en.wikipedia.org/wiki/Victor_Zue [17:45]Cocktail party effect: https://en.wikipedia.org/wiki/Cocktail_party_effect [20:39]Speaker diarisation: https://en.wikipedia.org/wiki/Speaker_diarisation [21:16]

Todos los episodios de Machine Learning Street Talk (MLST)

Ajustes

Visualización

Idioma

Determina qué emisoras ves primero en la página de inicio.

Historial

Las emisoras y episodios escuchados recientemente y tus últimas búsquedas solo se guardan en este navegador.

Avisos

Un breve aviso cuando un pódcast que sigues tiene un episodio nuevo. Puedes desactivar pódcast concretos con la campana de su página.

Cuenta

Guarda tus favoritos: en la app y en la web.

Atajos de teclado

/
Abrir búsqueda
Espacio
Reproducir o pausar
← →
En el reproductor: 15 s atrás, 30 s adelante (episodios) · en una fila: siguiente tarjeta
↑ ↓
En una fila: destino dentro de la tarjeta
Inicio / Fin
En una fila: primera o última tarjeta
Esc
Cerrar ventana o menú
?
Este resumen
Reproduciendo ahora

Ir a la emisora

A continuación

    Añade episodios con «A continuación»: sonarán después del episodio actual.