Saltar al contenido

#136: vLLM, LMD, and the Quest to Build the Linux of AI Inference

9 de junio de 2026 · 33 min

Notas del episodio

In this episode, hosts Ronald and Jan are joined at KubeCon by two guests from Red Hat: Brian Stevens, AI CTO and one of the original architects behind the creation of Kubernetes and the CNCF, and Rob Shaw, co-lead of the vLLM project and maintainer of LMD.

Brian shares the remarkable backstory of how Kubernetes came to be open source, including how Red Hat negotiated a single committer seat before agreeing to be a launch partner, and how he later pushed Google to contribute Kubernetes to the newly formed CNCF rather than keeping it proprietary like TensorFlow.

Rob explains what an inference runtime actually is: the critical piece of software that takes an abstract AI model and runs it as efficiently as possible on a GPU or other accelerator — handling everything from CUDA-level kernel optimization to memory management and concurrent request scheduling. vLLM serves as a "Rosetta Stone" between the ever-growing zoo of models (Llama, DeepSeek, Mistral, Qwen, Nvidia Nemotron) and accelerators (Nvidia, AMD, Intel, Google TPUs).

The conversation covers model compression and quantization how techniques like 4-bit precision can deliver 2x hardware efficiency gains while preserving 99%+ model accuracy. Brian and Rob also address the "big model vs. many small models" debate, recommending to always start with the largest capable model to validate a use case before optimizing down.

Looking ahead, both guests see inference as potentially the single largest workload ever run on Kubernetes, and position LMD (now contributed to the CNCF) as the distributed inference layer that will make this possible across heterogeneous accelerator environments preventing enterprises from ending up with 42 incompatible AI stacks.
The episode closes with a discussion on AI slop, human-in-the-loop thinking, and the future of Kubernetes as the universal platform for running AI agents at scale.

Powered by @acc-ict ​

Stuur ons een bericht.

ACC ICT Specialist in IT-CONTINUÏTEIT
Bedrijfskritische applicaties én data veilig beschikbaar, onafhankelijk van derden, altijd en overal

Dutch Cloud Native Day 2026

Two days of cloud native talks, workshops and community in Utrecht, exploring how AI is changing the way we build, run and scale modern platforms.

Like and subscribe! It helps out a lot.

You can also find us on:
De Nederlandse Kubernetes Podcast - YouTube
Nederlandse Kubernetes Podcast (@k8spodcast.nl) | TikTok
De Nederlandse Kubernetes Podcast

Where can you meet us:
Events

This Podcast is powered by:
ACC ICT - IT-Continuïteit voor Bedrijfskritische Applicaties | ACC ICT

Todos los episodios de De Nederlandse Kubernetes Podcast

Ajustes

Visualización

Idioma

Determina qué emisoras ves primero en la página de inicio.

Historial

Las emisoras y episodios escuchados recientemente y tus últimas búsquedas solo se guardan en este navegador.

Avisos

Un breve aviso cuando un pódcast que sigues tiene un episodio nuevo. Puedes desactivar pódcast concretos con la campana de su página.

Cuenta

Guarda tus favoritos: en la app y en la web.

Atajos de teclado

/
Abrir búsqueda
Espacio
Reproducir o pausar
← →
En el reproductor: 15 s atrás, 30 s adelante (episodios) · en una fila: siguiente tarjeta
↑ ↓
En una fila: destino dentro de la tarjeta
Inicio / Fin
En una fila: primera o última tarjeta
Esc
Cerrar ventana o menú
?
Este resumen
Reproduciendo ahora

Ir a la emisora

A continuación

    Añade episodios con «A continuación»: sonarán después del episodio actual.