Distributed LLM Inference on Kubernetes with llm-d, Wasm, & TornadoVM
Show notes
In this episode, Edoardo Vacchi discusses the evolving landscape of WebAssembly on the backend, running Java workloads on GPUs using TornadoVM, and optimizing distributed AI inference using llm-d and vLLM on Kubernetes. Edoardo explains how llm-d manages cache locality (KV and prefix caches), request routing, and batch inference across Kubernetes clusters to maximize expensive GPU utilization. Read a transcript of this interview: https://bit.ly/4eLRHnE Newsletter: Subscribe to the Software Architects' Newsletter, a monthly roundup of the patterns and technologies senior practitioners are working through, with the news and lessons from people doing the work: https://www.infoq.com/software-architects-newsletter InfoQ Online Certification Programs: InfoQ’s online cohorts put you in a small group of senior engineers and architects from other companies who face the same decisions. You bring a real decision from your job and work through it using approaches from QCon talks. You write up what you learned as an article, which makes you put the trade-offs into words before your next proposal at work. You leave with new approaches, or confirmation that the calls you’re already making are the right ones. See dates: https://certification.qconferences.com/ Upcoming Events: QCon San Francisco 2026 (November 16-20, 2026) https://qconsf.com/ QCon AI New York 2026 - AI Engineering Conference (December 15-16, 2026) https://newyork.qcon.ai/ QCon London 2027 (April 13-16, 2027) https://qconlondon.com/ QCon AI Boston 2027 - AI Engineering Conference (July 8-9, 2027) https://boston.qcon.ai/ The InfoQ Podcasts: Weekly conversations with senior software leaders about how they build systems and teams, including what they'd do differently. Listen to all our podcasts and read interview transcripts: The InfoQ Podcast: https://www.infoq.com/podcasts/ Engineering Culture Podcast by InfoQ: https://www.infoq.com/podcasts/#engineering_culture Generally AI: https://www.infoq.com/generally-ai-podcast/ Follow InfoQ: Mastodon: https://techhub.social/@infoq X: https://x.com/InfoQ LinkedIn: https://www.linkedin.com/company/infoq/ Facebook: https://www.facebook.com/InfoQdotcom Instagram: https://www.instagram.com/infoqdotcom/ YouTube: https://www.youtube.com/infoq Bluesky: https://bsky.app/profile/infoq.com Write for InfoQ: Share what you've learned building software with a community of senior practitioners, and get your work in front of the people who read InfoQ. https://www.infoq.com/write-for-infoq