Data Engineering Podcast
NeuEntity Resolution at Scale: Graphs, LLMs, and Real-World Data Messiness
- Folgen
- 521
- Erscheint
- alle zwei Wochen
- Letzte Folge
- 2. Nov. 2026
Über den Podcast
This show goes behind the scenes for the tools, techniques, and difficulties associated with the discipline of data engineering. Databases, workflows, automation, and data manipulation are just some of the topics that you will find here.
Mitwirkende
Folgen 521
Entity Resolution at Scale: Graphs, LLMs, and Real-World Data Messiness
Mit Steven Renwick
SummaryIn this episode, I spoke with Steven Renwick, CEO and co-founder of Tilores, about the realities of entity-oriented data processing and the practical role of LLMs in entity resolution. We covered the …
Shownotes
Summary
In this episode, I spoke with Steven Renwick, CEO and co-founder of Tilores, about the realities of entity-oriented data processing and the practical role of LLMs in entity resolution. We covered the confusing terminology around entity resolution, identity resolution, record linkage, named entity recognition, and disambiguation, and why these problems show up almost everywhere once organizations reach a certain scale. Steven explained why entity resolution is especially critical in domains like financial services, public sector systems, logistics, and any environment where messy, duplicated, or inconsistent records can create compliance, operational, or analytical risk.
We also dug into the question of whether LLMs can replace traditional entity resolution systems. Steven shared why the answer is only partly yes: LLMs can be very useful for discovery, entity extraction, weak-match review, onboarding, and configuration, but they struggle as the core production matching engine because of cost, scale, consistency, explainability, and state management requirements. We explored why graph structures are so useful for representing entities, how TeloRes approaches scale without relying on graph databases directly, and why teams should think carefully about build-versus-buy when tackling entity resolution. The conversation closed with lessons from real-world data quality, the growing importance of trustworthy identity layers for AI and analytics, and how clean entity data can strengthen everything from knowledge graphs to AI applications.
Announcements- Hello and welcome to the Data Engineering Podcast, the show about modern data management
- Today’s episode is sponsored by Parallel - where agents find answers. Most engineers today closely follow new model releases, but don’t pay attention to their agent’s most important tool: web search. Parallel develops enterprise-grade infrastructure for agents to retrieve high-quality context from the web. Their core products are a suite of APIs for retrieving high quality information from the web with Pareto-optimal quality, cost, and speed. Whether you work on voice agents that need 200 millisecond latency, chat bots that balance speed, depth, and quality, or long-horizon agents to do thorough, overnight research for you, Parallel is a single platform for all your agentic research. Get started for free at dataengineeringpodcast.com/parallel
- Your host is Tobias Macey and today I'm interviewing Steven Renwick about whether and where to use LLMs in entity-oriented data processing
- Introduction
- How did you get involved in the area of data management?
- Can you start by giving your definition of what constitutes an "entity"?
- What are some examples of the types of data work that rely heavily on entity resolution/disambiguation?
- How do LLMs alter or augment the functional capabilities of entity-oriented data processing?
- As with many areas of knowledge work, LLMs can easily create a compelling demo, but quickly degrade when trying to go from demo to production. What are the traps and pitfalls that become evident when trying to do entity resolution in production?
- What are your heuristics for what constitutes the threshold beyond which naive approaches become untenable?
- Can you describe what a production grade pipeline for entity resolution looks like?
- What role can/should LLMs play in that process?
- As LLMs become more powerful and faster, how/does that change the equation of their role in entity-oriented data processing? (particularly with diffusion-based models)
- What are the most interesting, innovative, or unexpected ways that you have seen LLMs used in entity extraction/resolution/disambiguation?
- What are the most interesting, unexpected, or challenging lessons that you have learned while working on entity resolution at scale?
- When is entity resolution overkill or unwarranted?
- What are some predictions or wishes that you have for the future directions/applications of entity resolution?
Contact Info
Parting Question- From your perspective, what is the biggest gap in the tooling or technology for data management today?
- Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The AI Engineering Podcast is your guide to the fast-moving world of building AI systems.
- Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
- If you've learned something or tried out a project from the show then tell us about it! Email [email protected] with your story.
- Tilores
- Named Entity Recognition (NER)
- Entity Resolution
- Neo4J
- AWS Neptune
- Apache Spark
- Graph Traversal Algorithms
- Entity Resolution Blocking
- GraphQL
- Clique-Based Graph Compression
- Splink
- Levenshtein Distance
- Cosine Similarity
- GraphRAG
- Entity-Resolved Knowledge Graph
- JEV Decision Model
- Laya
Kapitel (12)
How Amazon Uses Telemetry and AI to Automate Physical Logistics
Mit Keerti Kulkarni
Summary In this episode, I spoke with Keerti Kulkarni about the data and AI systems behind physical logistics automation at Amazon. She explained how her team supports dock-door operations across more than 100 …
Shownotes
Summary
In this episode, I spoke with Keerti Kulkarni about the data and AI systems behind physical logistics automation at Amazon. She explained how her team supports dock-door operations across more than 100 North American facilities by turning telemetry from physical workflows into actionable operational intelligence. We discussed the end-to-end data path from CloudWatch, Firehose, and S3 into Redshift and AI-powered analysis, along with the realities of handling petabyte-scale event streams, late and duplicate data, reconciliation, cost controls, and the need to treat physical telemetry as a first-class signal rather than noisy exhaust. Kirti also shared how these systems are designed to protect worker privacy while still surfacing the right information for safety and operational decisions.
We also dug into the role of LLMs and agentic workflows in helping operations teams diagnose failures quickly through conversational interfaces. Kirti described how AI-generated summaries help identify root causes such as authorization problems, hardware issues, or connectivity failures, while human feedback loops, structured outputs, and strict vocabulary controls help manage hallucination risk. Beyond troubleshooting, we explored how the platform is evolving toward proactive maintenance, training insights, per-site cost attribution, and broader modular agent architectures that could be shared across logistics teams. It’s a detailed look at what it takes to combine data engineering, real-time operations, and AI in a large-scale physical environment.
Announcements- Hello and welcome to the Data Engineering Podcast, the show about modern data management
- Today’s episode is sponsored by Parallel - where agents find answers. Most engineers today closely follow new model releases, but don’t pay attention to their agent’s most important tool: web search. Parallel develops enterprise-grade infrastructure for agents to retrieve high-quality context from the web. Their core products are a suite of APIs for retrieving high quality information from the web with Pareto-optimal quality, cost, and speed. Whether you work on voice agents that need 200 millisecond latency, chat bots that balance speed, depth, and quality, or long-horizon agents to do thorough, overnight research for you, Parallel is a single platform for all your agentic research. Get started for free at dataengineeringpodcast.com/parallel
- Your host is Tobias Macey and today I'm interviewing Keerti Kulkarni about scaling insights for physical logistics automation with AI-enabled intelligence
Interview
- Introduction
- How did you get involved in the area of data management?
- Can you start by unpacking what "physical logistics automation is"?
- What are the technical and organizational systems involved in the production and consumption of that data?
- What are some of the ways that inaccuracies or increased latency manifest in real-world impact?
- You recently worked on a substantial system for managing this data. Can you describe the challenges and limitations of the system that you started from?
- What was your process for identifying the weak points of the system and how to evolve it?
- What is the design of the system that you have running today?
- What are some of the design choices that had the biggest impact on the final result?
- What are the most complicated aspects of managing cost attribution to the different physical locations, data flows, and model usage?
- What is the role of AI in providing insights to the organizational users?
- How do you measure and manage accuracy of the models as the system evolves?
- What are the most interesting, innovative, or unexpected ways that you have seen your system used?
- What are the most interesting, unexpected, or challenging lessons that you have learned while working on physical logistics automation?
- What do you have planned for the future of your platform?
Contact Info
Parting Question- From your perspective, what is the biggest gap in the tooling or technology for data management today?
- Amazon Key
- AWS Cloudwatch Analytics
- AWS Firehose
- AWS Kinesis
- AWS Quick
- OSHA == Occupational Safety and Health Administration
- OmniGraph
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SAKapitel
Kapitel werden geladen …
Why Metadata Still Matters: From Data Catalogs to AI-Ready Semantic Hubs
Mit Christian Bremeau und Simon Dynin
SummaryIn this episode, I spoke with Christian Bremeau and Simon Dynin about their decades-long work in metadata management and how that experience shaped the evolution of Meta Integration Technology and its …
Shownotes
Summary
In this episode, I spoke with Christian Bremeau and Simon Dynin about their decades-long work in metadata management and how that experience shaped the evolution of Meta Integration Technology and its direct-to-customer MetaKarta platform. We explored the recurring challenges of fragmented data ecosystems, why every tool inevitably creates its own metadata silo, and why organizations still need a vendor-neutral system to stitch lineage, semantics, and change history across databases, ETL/ELT pipelines, BI tools, and cloud platforms. Christian and Simon explained why deep lineage, versioning, configuration management, and deterministic understanding of transformations matter far more than a polished catalog UI when teams need to trust numbers, assess impact, or satisfy audits.
We also dug into the growing importance of metadata in the age of AI. Christian and Simon shared their view that metadata management is becoming foundational for both BI and agentic systems, especially as organizations struggle with semantic drift, tool sprawl, and non-deterministic AI behavior. They introduced Metacarta’s new Semantic Hub as a way to reconcile business logic across legacy and modern tools, compile shared semantics into downstream systems, and provide consistent answers for both human analysts and AI agents. Along the way, we discussed metadata ops, the organizational processes needed to make metadata programs succeed, and the practical situations where a cross-platform metadata system is—or isn’t—the right fit.
Announcements- Hello and welcome to the Data Engineering Podcast, the show about modern data management
- Today’s episode is sponsored by Parallel - where agents find answers. Most engineers today closely follow new model releases, but don’t pay attention to their agent’s most important tool: web search. Parallel develops enterprise-grade infrastructure for agents to retrieve high-quality context from the web. Their core products are a suite of APIs for retrieving high quality information from the web with Pareto-optimal quality, cost, and speed. Whether you work on voice agents that need 200 millisecond latency, chat bots that balance speed, depth, and quality, or long-horizon agents to do thorough, overnight research for you, Parallel is a single platform for all your agentic research. Get started for free at dataengineeringpodcast.com/parallel
- Your host is Tobias Macey and today I'm interviewing Christian Bremeau and Simon Dynin about his long history of dealing with metadata at Meta Integration Technology, and how they are adapting to the age of AI
Interview
- Introduction
- How did you get involved in the area of data management?
- Can you start by sharing some of the highlights of your career in metadata and how the industry's focus on that subject has changed over the years?
- What are the major sources of friction and fragmentation in metadata management in your experience?
- There are numerous metadata management, metadata integration, data catalog, and other governance/discovery/semantic platforms in the ecosystem. How do you explain the differentiating factors of your system to people who are evaluating their options?
- One of the constant challenges in data systems is the tension between data locality and data consistency. What is your approach to managing a canonical, reliable definition of a data asset, while making it available and accessible to all of the consuming systems?
- There was a surge of interest in metadata systems in the 2018 - 2022 time-frame as the industry invested in cloud data warehouses, ELT, and all of the other "modern data stack" patterns. Now AI is driving a new surge of interest. What are the categorical differences between these periods in your experience?
- What are the new stressors that you anticipate from the continued growth in LLM-mediated and agentic data interactions?
- Can you describe the overall architecture of the MetaKarta platform and how it has evolved in the latest version?
- For a team who is trying to tame their data estate, what does the adoption of MetaKarta look like?
- How does it change their day-to-day work?
- You're promoting the idea of "MetaDataOps" as a dedicated concern for an organization. What are the skills and activities of someone who is focused on that capability?
- What are the most interesting, innovative, or unexpected ways that you have seen MetaKarta used?
- What are the most interesting, unexpected, or challenging lessons that you have learned while working on Metadata and MetaKarta?
- When is MetaKarta the wrong choice?
- What do you have planned for the future of MetaKarta?
Contact Info
Parting Question- From your perspective, what is the biggest gap in the tooling or technology for data management today?
Links- Meta Integration Technology
- MetaKarta)
- Cognos
- Informatica
- ER Studio
- Talend
- Data Catalog
- ETL == Extract, Transform, Load
- ELT == Extract, Load, Transform
- ThoughtSpot
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SAKapitel
Kapitel werden geladen …
Reducing Data Debt with Agile Ledger Architecture
Mit Christopher Doidge
SummaryIn this episode Christopher Doidge talks about his Agile Ledger Architecture (ALA) approach to data warehousing and how it aims to reduce data debt while shortening the path from raw data to trustworthy …
Shownotes
Summary
In this episode Christopher Doidge talks about his Agile Ledger Architecture (ALA) approach to data warehousing and how it aims to reduce data debt while shortening the path from raw data to trustworthy business insight. Christopher explained that ALA is not a replacement for existing warehouse patterns like medallion architecture, star schemas, or other modeling approaches, but a complementary discipline focused on pushing business definitions upstream, enforcing cleaner ledger-style transformations, and producing gold-layer tables that stakeholders can actually use without relying on analysts to repeatedly rebuild the same logic. He also discussed his “15-minute litmus test” for time-to-insight, the importance of durable documentation through a data dictionary, and why undocumented business logic living in analyst scripts is often one of the biggest hidden forms of data debt. Overall, this was a thoughtful conversation about designing warehouse systems that serve not just analysts, but the broader business as well.
Announcements- Hello and welcome to the Data Engineering Podcast, the show about modern data management
- Today’s episode is sponsored by Parallel - where agents find answers. Most engineers today closely follow new model releases, but don’t pay attention to their agent’s most important tool: web search. Parallel develops enterprise-grade infrastructure for agents to retrieve high-quality context from the web. Their core products are a suite of APIs for retrieving high quality information from the web with Pareto-optimal quality, cost, and speed. Whether you work on voice agents that need 200 millisecond latency, chat bots that balance speed, depth, and quality, or long-horizon agents to do thorough, overnight research for you, Parallel is a single platform for all your agentic research. Get started for free at dataengineeringpodcast.com/parallel
- Your host is Tobias Macey and today I'm interviewing Christopher Doidge about a data warehousing approach called Agile Ledger Architecture that is designed to drive down data debt
- Introduction
- How did you get involved in the area of data management?
- Can you describe what the agile ledger architecture is and the story behind it?
- There are numerous patterns and practices that have been developed for data warehousing over the past 40 years. How does the agile ledger architecture fit in that ecosystem? (e.g. is it compatible with Kimball, Inmon, Data Vault, Anchor Modeling, etc.?)
- What are some examples of the types of debt that accumulate in current approaches to warehouse implementation, and the impact that it has on the utility of that asset?
- Digging into the architecture itself, what are the core principles that it is built on?
- What are the technologies or practices that it is best suited to? (e.g. event streams, lakehouse, ELT workflows, etc.)
- For someone who has already invested a substantial amount of effort into building a warehouse, what does adoption of the agile ledger architecture look like?
- Once you have started that adoption, what are the technical controls that you can put in place to ensure that this architecture is maintained and doesn't regress or get sidestepped in another portion of the warehouse?
- As LLMs and agents grow to become the predominant consumers (and often producers) of a warehouse, what are the benefits that the agile ledger architecture provides to ensure appropriate context and grounding to produce useful insights and accurate answers?
- What are the most interesting, innovative, or unexpected ways that you have seen the agile ledger architecture used?
- What are the most interesting, unexpected, or challenging lessons that you have learned while working on data warehouse design and implementation?
- When is the agile ledger architecture the wrong choice?
- What do you have planned for the future of this architecture?
Parting Question- From your perspective, what is the biggest gap in the tooling or technology for data management today?
- Agile Ledger Architecture Book (affiliate link)
- Ralph Kimball
- Bill Inmon
- Data Vault
- Star Schema
- Anchor Modeling
- Surrogate Key
- Data Lakehouse
Kapitel
Kapitel werden geladen …
What Context Really Means in Data Engineering and AI
Mit Soham Mazumdar
SummaryIn this episode Soham Mazumdar, co-founder and CEO of Wisdom.ai, talks about what “context” really means in data engineering and AI systems. He explores why context has become such an overloaded term …
Shownotes
Summary
In this episode Soham Mazumdar, co-founder and CEO of Wisdom.ai, talks about what “context” really means in data engineering and AI systems. He explores why context has become such an overloaded term, spanning everything from semantic layers and data catalogs to tribal knowledge, query logs, dashboards, and even agent memory. Soham explained that the big shift is that context is no longer being prepared primarily for human analysts, but for LLMs and agents that can’t reliably fill in missing gaps on their own. That change raises the bar for how context is represented, validated, benchmarked, and maintained so that AI systems can produce trustworthy outcomes.
Announcements- Hello and welcome to the Data Engineering Podcast, the show about modern data management
- Your host is Tobias Macey and today I'm interviewing Soham Mazumdar about what "context" actually means in data engineering
- Introduction
- How did you get involved in the area of data management?
- One of the perennial challenges of engineering in all forms is building a shared understanding of what a given word means. "Context" is one that is being used for an increasing number of purposes with the introduction of AI agents. Can you start by sharing some of the ways that this terminology overload has caused problems in your own experience?
- Data engineering has arguably always been about context engineering, but at the scale of human consumers. What are the substantive changes that AI/agentic consumers bring to the discipline?
- While we all understand the notion of "context", turning it into a useful and re-usable component is a different matter entirely. What are some of the ways that "business context" or "technical context" manifests as a tangible artifact?
- This also brings up the question of data modeling. What are some of the key attributes that are necessary when storing, enriching, evolving, and joining into that context?
- One could argue that the entire history of data warehousing is about building organizational context. What are the real differences in approach for today's work of capturing and activating that context?
- How does your work at Wisdom AI address the technical and operational burdens of capturing, modeling, and exposing context at the speed necessary to keep up with organizational demands?
- What are the most interesting, innovative, or unexpected ways that you have seen Wisdom AI used?
- What are the most interesting, unexpected, or challenging lessons that you have learned while working on Wisdom AI/context engineering?
- When is Wisdom AI the wrong choice?
- What do you have planned for the future of Wisdom AI?
Parting Question- From your perspective, what is the biggest gap in the tooling or technology for data management today?
- Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The AI Engineering Podcast is your guide to the fast-moving world of building AI systems.
- Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
- If you've learned something or tried out a project from the show then tell us about it! Email [email protected] with your story.
- Wisdom AI
- Context Engineering
- Knowledge Graph
- Ontology
- Snowflake Open Semantic Interchange (OSI)
- Semantic Layer
- Palantir Foundry
Kapitel
Kapitel werden geladen …
Specialized AI for Data Engineers: Inside Astronomer’s Otto
Mit Yetunde Dada
Summary In this episode Yetunde Dada discusses Otto, Astronomer’s AI agent for Airflow, and the broader challenge of making agentic tooling actually useful for data engineers. She explored why generic coding …
Shownotes
Summary
In this episode Yetunde Dada discusses Otto, Astronomer’s AI agent for Airflow, and the broader challenge of making agentic tooling actually useful for data engineers. She explored why generic coding assistants often fall short in data workflows, how Otto adds the missing context around Airflow, Astro, upgrades, and troubleshooting, and why Astronomer focused first on high-leverage use cases such as DAG authoring, investigation of pipeline failures, version migrations, and legacy scheduler modernization. She also discussed the practical realities of introducing agents into engineering teams: model choice, security boundaries, vendor lock-in concerns, validation of generated code, and the need for agents to fit into existing workflows rather than forcing users into new ones. Overall, this conversation offers a detailed look at how specialized AI agents can support data engineers today, and where Astronomer is headed next with a vision for self-healing pipelines that keep humans in control while automating more of the operational burden.
Announcements- Hello and welcome to the Data Engineering Podcast, the show about modern data management
- Your host is Tobias Macey and today I'm interviewing Yetunde Dada about Otto, Astronomer's expert Airflow agent
Interview
- Introduction
- How did you get involved in the area of data management?
- Can you describe what Otto is and the story behind it?
- What are the core problems that you are trying to solve with Otto and for whom?
- What was your process for identifying the scope of activities that Otto should be incorporated into?
- Orchestration engines are a rich source of information. What are the aspects of Airflow that lend themselves to extending with this agentic context?
- What are the other supporting systems that are necessary to enable Otto to work effectively, especially in mixed orchestration environments? (e.g. metadata platforms)
- One of the explicit capabilities that you invested in is code review for Airflow DAGs. What are the pain points that you are trying to solve with a specialized review agent?
- Can you describe the architecture of the Otto system and how you're managing the complex task of context curation?
- In a production context accuracy and latency are both critical, and often in tension with each other. How do you monitor and optimize for each of those objectives?
- What are the options for tuning Otto's behavior to bias more toward one direction or another?
- What are some examples of the type of work that Otto can help automate?
- How is it measurably different from a generic coding agent that has MCP connections to something like an Open Metadata or DataHub for platform and data context, Airflow documentation, etc.?
- There are numerous general purpose and specialized agent systems available. What are some of the ways that Otto can work collaboratively with those other products?
- What are the most interesting, innovative, or unexpected ways that you have seen Otto used?
- What are the most interesting, unexpected, or challenging lessons that you have learned while working on Otto?
- What do you have planned for the future of Otto?
Contact Info
Parting Question
- From your perspective, what is the biggest gap in the tooling or technology for data management today?
Links
- Astronomer
- Otto
- Announcement Post
- Astronomer Cosmo dbt automation
- Hadoop
- Spark
- Otto Automatic Pipeline Failure Investigation
- Otto Code Review
- Astro CLI
- Astro IDE
- Airflow MCP
- Kedro
- Quantum Black
- Django
- React
- Airflow Providers
- Pi Framework
- Agent Skills
- AGENTS.md
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SAKapitel
Kapitel werden geladen …
Why Multi-Agent Systems Need Shared State, Graph Semantics, and Governance
Mit Ragnor Comerford
Summary In this episode Ragnor Comerford talks about OmniGraph, a lakehouse-native graph storage layer designed around the needs of agentic systems. He explores how graphs are primarily a semantic model for …
Shownotes
Summary
In this episode Ragnor Comerford talks about OmniGraph, a lakehouse-native graph storage layer designed around the needs of agentic systems. He explores how graphs are primarily a semantic model for representing the world, rather than just a specialized engine for traversal workloads, and how that perspective shaped OmniGraph’s design on top of object storage, Lance, Arrow, and DataFusion. Ragnor explained the motivation for combining graph semantics with Git-style branching and merging so that teams can manage probabilistic writers such as AI agents with stronger governance, shared context, and safer collaboration patterns. He also dug into the practical tradeoffs of building a graph engine for multi-agent coordination instead of traditional graph analytics use cases. He closed with a look at emerging use cases such as company “brain” systems, software development lifecycle graphs, research workflows, and event-driven agent orchestration, along with a broader conversation about composability, and sovereign AI infrastructure.
Announcements- Hello and welcome to the Data Engineering Podcast, the show about modern data management
- Your host is Tobias Macey and today I'm interviewing Ragnor Comerford about OmniGraph, a lakehouse-native graph storage layer with git semantics
Interview
- Introduction
- How did you get involved in the area of data management?
- Can you describe what OmniGraph is and the story behind it?
- What was the original problem that you were trying to solve by creating it?
- There are numerous graph engines available, what are the properties of OmniGraph that differentiate it from the competition?
- Cypher (GQL) and Gremlin are all established languages with years of examples to work from. What are the benefits of developing a new and more constrained query interface for an agentic audience?
- How does that change the potential applications of OmniGraph? (e.g. general knowledge graph, fraud detection, SIEM, etc.)
- Can you describe the architecture of OmniGraph?
- You have built the system on top of several well-established open source components. What was your process for deciding what to use and how to compose it?
- What are some examples of systems that can be built with OmniGraph?
- What are other components/integration points that compose well with OmniGraph?
- Given the technologies that you are building on top of, what are the automatic benefits/integrations that you benefit from?
- Given that the underlying storage is Lance, and Lance's interoperability with Parquet/Iceberg, what are the opportunities for modeling graphs on top of existing lakehouse data?
- What are the most interesting, innovative, or unexpected ways that you have seen OmniGraph used?
- What are the most interesting, unexpected, or challenging lessons that you have learned while working on OmniGraph?
- When is OmniGraph the wrong choice?
- What do you have planned for the future of OmniGraph?
Contact Info
Parting Question
- From your perspective, what is the biggest gap in the tooling or technology for data management today?
Closing Announcements
- Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The AI Engineering Podcast is your guide to the fast-moving world of building AI systems.
- Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
- If you've learned something or tried out a project from the show then tell us about it! Email [email protected] with your story.
Links
- OmniGraph
- ModernRelay
- Information Theory
- Lance
- Git
- Neo4J
- GraphRAG
- TigerGraph
- LakeHouse
- Iceberg
- Dolt
- PuppyGraph
- Podcast Episode
- Terraform
- Gremlin
- Cypher
- GQL
- SPARQL
- In-context Learning
- BM25 Indexing
- Data Fusion
- Adjacency Matrix
- Predicate Pushdown
- Agentic Mesh book (affiliate link)
- witan-council
- witan-code
- Web Assembly
- Clickhouse
- DSPy
- MCP-UI
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SAKapitel
Kapitel werden geladen …
Building the Context Flywheel for AI Data Agents
Summary In this episode Prukalpa Sankar, co-founder of Atlan, talks about what it takes to build a “context flywheel” for AI agents in data-intensive organizations. She explained why model intelligence alone …
Shownotes
Summary
In this episode Prukalpa Sankar, co-founder of Atlan, talks about what it takes to build a “context flywheel” for AI agents in data-intensive organizations. She explained why model intelligence alone isn’t enough to make AI useful in production, and how real performance depends on contextual intelligence: institutional knowledge, semantic meaning, procedural know-how, and access to the right tools. She also dug into how metadata catalogs are evolving into broader context layers that serve both humans and agents, and why agentic systems are changing the economics of metadata and governance work. Prakulpa shared Atlan’s perspective on bootstrapping context from existing systems such as warehouses, BI tools, query logs, and SaaS applications, then using simulation, traces, and human governance loops to improve agent accuracy over time.
Announcements- Hello and welcome to the Data Engineering Podcast, the show about modern data management
- Your host is Tobias Macey and today I'm interviewing Prukalpa Sankar about strategies for building a context flywheel for your data agents
- Introduction
- How did you get involved in the area of data management?
- You have spent several years working in the metadata catalog space with Atlan. What are the notable changes in scope, adoption, and application that you have seen since we last spoke (June 2022)?
- The recurring theme since the start of 2026 has been agentic augmentation of all engineering workflows, including data. How do you differentiate between data catalogs, semantic layers, agent memory, context layers, etc. when architecting an AI-powered data-oriented system?
- One of the perennial problems with data catalogs, business glossaries, master data management, etc. is the up-front investment required to get a real-world impact. How can agents help reduce the activation energy needed to get to that return on effort?
- One of the perennial problems in data engineering is fragmentation and siloing of data. This is exacerbated by AI systems due to the introduction of vector data as a new specialization. What are the forces that you are seeing play into the current set of tensions and the architectural primitives that we need to bring to bear to keep things maintainable?
- Since the introduction of transformer-based generative models we have been combating hallucinations. While we have made progress, it is still critical to ensure accuracy and trustworthiness when working with business data. What are the policy elements of governance and technical controls to ensure a high degree of confidence in agent-generated context and business semantics?
- What are the most interesting, innovative, or unexpected ways that you have seen teams build context layers for their agentic data workloads?
- What are the most interesting, unexpected, or challenging lessons that you have learned while working on business context engineering?
- When is agent-managed context the wrong choice?
- What are your predictions for the next set of architectural shifts that will be driven by the pressures of AI-powered systems?
Contact Info
Parting Question- From your perspective, what is the biggest gap in the tooling or technology for data management today?
- Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The AI Engineering Podcast is your guide to the fast-moving world of building AI systems.
- Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
- If you've learned something or tried out a project from the show then tell us about it! Email [email protected] with your story.
- Atlan
- Atlan Context Lakehouse
- Iceberg
- Business Glossary
- Master Data Management
- Semantic Layer
- Cube.dev
- MCP == Model Context Protocol
- A2A == Agent to Agent Protocol
- Decision Traces
- Apache Doris
- StarRocks
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SAKapitel
Kapitel werden geladen …
Holding Kafka Right: Product-Friendly Streaming with TypeStream
Summary In this episode Jevin Maltais talks about the practical realities of building reliable, product-focused streaming systems with Kafka. Jevin shares lessons from roles at Zapier, Humi, and Clio, where …
Shownotes
Summary
In this episode Jevin Maltais talks about the practical realities of building reliable, product-focused streaming systems with Kafka. Jevin shares lessons from roles at Zapier, Humi, and Clio, where real-time synchronization, customer data unification, and document sync at scale highlighted both the strengths and common misuses of Kafka. He digs into using events as the source of truth, materialized views with KTables, and how schema registries and type safety prevent downstream breakage. Jevin explains why teams often reach for heavyweight Kafka clusters without leveraging Streams, Connect, or interactive queries—and how his project, TypeStream, aims to make those capabilities accessible via config-as-code while keeping a thin abstraction and clear escape hatches. He also explore trade-offs across Kafka-compatible alternatives, CDC with Debezium in the real world, and where abstractions should stop so teams can scale responsibility as complexity grows.
Announcements- Hello and welcome to the Data Engineering Podcast, the show about modern data management
- This episode is sponsored by DataDriven.io, the free data engineering interview prep platform built by data engineers for data engineers. Ever walked into a data engineering interview and gotten a question that has nothing to do with real data engineering work? Interviewing is its own skill, separate from the job. Watch your code execute live, inspect Spark internals, and whiteboard your data models and pipelines and defend your decisions. Unlike SQL-only or Python-only practice, DataDriven.io covers the full interview loop: star schemas, slowly changing dimensions, grain and fact table design, idempotency, watermarks, dead letter queues, change data capture, and backpressure. Every question comes from real Data Engineer interview loops at Google, Amazon, Meta, Stripe, Databricks, Netflix, and Airbnb. Go to dataengineeringpodcast.com/datadriven today to start practicing.
- Your host is Tobias Macey and today I'm interviewing Jevin Maltais about the challenges of building a reliable streaming
Interview
- Introduction
- How did you get involved in the area of data management?
- Can you describe what Typestream is and the story behind it?
- What are the common challenges that teams encounter when trying to build on top of Kafka?
- How do those challenges/misconfigurations impact the team's ability to deliver on product goals?
- What are the fundamental design aspects of Kafka that contribute to the difficulties that teams encounter when using it as an element of their architecture?
- There have been numerous projects taking aim at Kafka, with varying approaches and degrees of effectiveness (e.g. RedPanda, AutoMQ, Pulsar, etc.). What are the tradeoffs that each of those approaches requires?
- What makes the original Kafka project so resilient in the face of all of that competition?
- Can you describe the architecture of Typestream and how each of the core elements contribute to a better user experience?
- For teams who want to take advantage of streaming capabilities, but don't want to invest in becoming Kafka experts, what does the Typestream workflow look like?
- If they don't want to manage the operational overhead of a Kafka cluster, how tightly coupled is Typestream to the original Kafka? (can someone use RedPanda or AutoMQ instead?)
- What are the most interesting, innovative, or unexpected ways that you have seen Typestream used?
- What are the most interesting, unexpected, or challenging lessons that you have learned while working on Typestream?
- When is Typestream the wrong choice?
- What do you have planned for the future of Typestream?
Contact Info
Parting Question
- From your perspective, what is the biggest gap in the tooling or technology for data management today?
Closing Announcements
- Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The AI Engineering Podcast is your guide to the fast-moving world of building AI systems.
- Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
- If you've learned something or tried out a project from the show then tell us about it! Email [email protected] with your story.
Links
- Typestream
- Zapier
- Airflow
- Kafka
- KTables
- KSQL
- RedPanda
- Pulsar
- AutoMQ
- Kafka Schema Registry
- Debezium
- Change Data Capture
- Kafka Connect
- Terraform
- Kafka Compacted Topic
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SAKapitel
Kapitel werden geladen …
Text to Data Products: Kaarvi’s End-to-End AI for Ingestion, Quality, and Dashboards
Mit Shravan Gunda
Summary In this episode Shravan Gunda, founder and CEO of Kaarvi AI, talks about building an AI-native, agent-driven data platform designed to eliminate the janitorial work that consumes most data teams. He …
Shownotes
Summary
In this episode Shravan Gunda, founder and CEO of Kaarvi AI, talks about building an AI-native, agent-driven data platform designed to eliminate the janitorial work that consumes most data teams. He explores Kaarvi’s multi-agent architecture that runs queries across seven LLMs in parallel for reliability, its synthetic data generator that mirrors source schemas for quick testing, and “Hey Kaarvi” chat for text-to-SQL, text-to-transformations, and text-to-dashboard workflows. He also digs into on-prem versus SaaS deployments, domain-specialized agents for privacy and accuracy, code blocks for custom Python/SQL, and the roadmap for a marketplace and desktop assistant. Shravan highlights how Kaarvi compresses weeks of work into hours and bridges the gap between business users and data engineers by turning AI into a dependable force multiplier.
Announcements- Hello and welcome to the Data Engineering Podcast, the show about modern data management
- This episode is sponsored by DataDriven.io, the free data engineering interview prep platform built by data engineers for data engineers. Ever walked into a data engineering interview and gotten a question that has nothing to do with real data engineering work? Interviewing is its own skill, separate from the job. Watch your code execute live, inspect Spark internals, and whiteboard your data models and pipelines and defend your decisions. Unlike SQL-only or Python-only practice, DataDriven.io covers the full interview loop: star schemas, slowly changing dimensions, grain and fact table design, idempotency, watermarks, dead letter queues, change data capture, and backpressure. Every question comes from real Data Engineer interview loops at Google, Amazon, Meta, Stripe, Databricks, Netflix, and Airbnb. Go to dataengineeringpodcast.com/datadriven today to start practicing.
- Your host is Tobias Macey and today I'm interviewing Shravan Gunda about building an agent-driven data platform at Kaarvi
- Introduction
- How did you get involved in the area of data management?
- Can you describe what Kaarvi is and the story behind it?
- "AI" is a very broad term that encompasses numerous possible implementations. Can you give some more detail about the different types and applications of AI in Kaarvi's architecture?
- What are some of the core assumptions of data workflows that need to be reconsidered when AI is embedded in the execution path?
- What are the most interesting, innovative, or unexpected ways that you have seen Kaarvi used?
- What are the most interesting, unexpected, or challenging lessons that you have learned while working on Kaarvi?
- When is Kaarvi the wrong choice?
- What do you have planned for the future of Kaarvi?
Contact Info
Parting Question
- From your perspective, what is the biggest gap in the tooling or technology for data management today?
Closing Announcements
- Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The AI Engineering Podcast is your guide to the fast-moving world of building AI systems.
- Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
- If you've learned something or tried out a project from the show then tell us about it! Email [email protected] with your story.
Links
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SAKapitel
Kapitel werden geladen …
Scaling Graph Analytics Without ETL: Inside PuppyGraph’s Architecture
Mit Weimo Liu
SummaryIn this episode Weimo Liu, co‑founder of PuppyGraph, talks about the engineering behind their “zero-copy” graph querying engine for lakehouse and database sources. He explores how PuppyGraph lets you run …
Shownotes
Summary
In this episode Weimo Liu, co‑founder of PuppyGraph, talks about the engineering behind their “zero-copy” graph querying engine for lakehouse and database sources. He explores how PuppyGraph lets you run Cypher and Gremlin traversals and graph algorithms directly on data in Iceberg, Delta, Hudi, Hive, and even MongoDB—without loading into a separate graph store. Weimo explains their edge-sharded, vectorized, MPP architecture that tackles hub nodes, multi-hop traversals, and shuffle at scale, targeting sub-second to single-digit-second workloads. He digs into practical graph data modeling on top of normalized and denormalized tables, logical views, and flexible mappings; strategies for caching, adaptive reads, and leveraging Iceberg metadata; and how PuppyGraph’s operator-based engine unifies query and algorithms. He also covers real-world applications—from cybersecurity log analysis to entity resolution and agentic workflows—when to choose embedded or transactional graph databases instead, and what’s next for enterprise features and broader warehouse integrations.
Announcements- Hello and welcome to the Data Engineering Podcast, the show about modern data management
- This episode is sponsored by DataDriven.io, the free data engineering interview prep platform built by data engineers for data engineers. Ever walked into a data engineering interview and gotten a question that has nothing to do with real data engineering work? Interviewing is its own skill, separate from the job. Watch your code execute live, inspect Spark internals, and whiteboard your data models and pipelines and defend your decisions. Unlike SQL-only or Python-only practice, DataDriven.io covers the full interview loop: star schemas, slowly changing dimensions, grain and fact table design, idempotency, watermarks, dead letter queues, change data capture, and backpressure. Every question comes from real Data Engineer interview loops at Google, Amazon, Meta, Stripe, Databricks, Netflix, and Airbnb. Go to dataengineeringpodcast.com/datadriven today to start practicing.
- Your host is Tobias Macey and today I'm interviewing Weimo Liu about the engineering behind PuppyGraph's zero-copy ETL for querying your lakehouse as a graph
- Introduction
- How did you get involved in the area of data management?
- Can you start by describing what PuppyGraph is and the story behind it?
- What are some of the key use cases that people are turning to PuppyGraph and graph data models for?
- Graph engines have struggled to take off for several years, not least of which is due to the difficulty of scaling them to large data volumes as a result of the topological nature of the data. Can you describe the architecture of PuppyGraph and some of the ways that you are addressing that challenge of data volume for graphs?
- latency/data exploration
- types of traversals and limitations
- lakehouse architecture pros/cons for graphs
- data modeling/translation
- shortcomings of zero-ETL and how transforming the underlying representation could provide benefits
- For someone who is looking for a graph engine to support a connected data use case, what are the guiding questions that you would ask to lead them toward PuppyGraph vs. a dedicated graph database like Memgraph/Neo4J/etc.?
- What are the most interesting, innovative, or unexpected ways that you have seen PuppyGraph used?
- What are the most interesting, unexpected, or challenging lessons that you have learned while working on PuppyGraph?
- When is PuppyGraph the wrong choice?
- What do you have planned for the future of PuppyGraph and graph data exploration on large data volumes?
Parting Question- From your perspective, what is the biggest gap in the tooling or technology for data management today?
- Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The AI Engineering Podcast is your guide to the fast-moving world of building AI systems.
- Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
- If you've learned something or tried out a project from the show then tell us about it! Email [email protected] with your story.
- PuppyGraph
- TigerGraph
- Google F1
- Graph Database
- Google Pregel
- Iceberg
- Graph Supernode
- MPP == Massively Parallel Processing
- Spark GraphX
- Trino
- Ladybug DB
- lance-graph
- KuzuDB
- MemGraph
- Labelled Property Graph
- RDF Triples
- Cypher Query Language
- Gremlin
- CDC == Change Data Capture
- Neo4J
- JanusGraph
- NetworkX
- PyTorch
- DuckDB
- Iceberg Array
- LanceDB
- Palo Alto Networks
- Columnar ADBC
%Kapitel
Kapitel werden geladen …
Maximizing GPU Utilization: Heterogeneous Pipelines with Ray and Kubernetes
Mit Robert Nishihara
SummaryIn this episode Robert Nishihara, co-founder of Anyscale and co-creator of Ray, talks about maximizing hardware utilization for AI and data-intensive workloads. He explores Ray’s evolution alongside …
Shownotes
Summary
In this episode Robert Nishihara, co-founder of Anyscale and co-creator of Ray, talks about maximizing hardware utilization for AI and data-intensive workloads. He explores Ray’s evolution alongside Kubernetes and PyTorch, and why consolidation at these layers has enabled a new generation of complex, heterogeneous workloads. Robert explains how data preparation has shifted to GPU- and inference-heavy, multimodal pipelines; where Ray fits compared to Spark and workflow orchestrators; and why Ray excels at composing heterogeneous pools of compute, handling failures, and scaling complex systems like multi-node LLM inference and reinforcement learning. He digs into practical strategies for boosting GPU utilization across training and inference, elasticity and prioritization of workloads, topology-aware scheduling, and the importance of fast failure recovery as hardware scales from nodes to racks. If you’re wrestling with expensive GPUs, multimodal data curation, or cross-node LLM inference, this conversation offers concrete mental models and architectural guidance.
Announcements- Hello and welcome to the Data Engineering Podcast, the show about modern data management
- Your host is Tobias Macey and today I'm interviewing Robert Nishihara about the challenges of maximizing the utility of your available hardware for AI applications
- Introduction
- How did you get involved in the area of data management?
- Can you start by giving an overview of the major contributors to wasted or idle compute?
- Why does it matter if the available compute isn't being maximized?
- What are some of the typical ad-hoc methods that teams might use to try to get the most out of their available hardware (especially GPUs)?
- What are the most interesting, innovative, or unexpected ways that you have seen Ray used?
- What are the most interesting, unexpected, or challenging lessons that you have learned while working on Ray and distributed compute for data and AI?
- When is Ray the wrong choice?
- What do you have planned for the future of Ray?
Parting Question- From your perspective, what is the biggest gap in the tooling or technology for data management today?
- Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The AI Engineering Podcast is your guide to the fast-moving world of building AI systems.
- Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
- If you've learned something or tried out a project from the show then tell us about it! Email [email protected] with your story.
- AnyScale
- Ray
- Deep Learning
- Computer Vision
- Kubernetes
- Cursor
- Claude Code
- Kube-Ray
- PyTorch
- Tensorflow
- Theano
- Caffe
- vLLM
- SGLang
- Ray Tune
- Neural Network
- Learning Rates
- Reinforcement Learning
- AlphaGo
- Cursor Composer 2
- ImageNet
- Transformer Architecture
- Stochastic Gradient Descent
- Airflow
- Dagster
- Flyte
- Mixture of Experts
- Prefill
- Temporal
- Actor Framework
- RDMA == Remote Direct Memory Access
- Neoclouds
- AI Engineering Podcast Episode
Kapitel
Kapitel werden geladen …
The AI-First Data Engineer: 10–50x Productivity and What Changes Next
Mit Gleb Mezhanskiy
Summary In this episode, I sit down with Gleb Mezhanskiy, CEO and co-founder of Datafold, to explore how agentic AI is reshaping data engineering. We unpack the leap from chat-assisted coding to truly agentic …
Shownotes
Summary
In this episode, I sit down with Gleb Mezhanskiy, CEO and co-founder of Datafold, to explore how agentic AI is reshaping data engineering. We unpack the leap from chat-assisted coding to truly agentic workflows where AI not only writes SQL and dbt models but also executes queries, debugs, runs tests, and ships production-ready outcomes. Gleb explains why teams that master this AI-first loop can see 10–50x gains, how security/compliance concerns can be addressed with platform-native LLM endpoints, and why the role of data engineers is shifting from code authors to operators of autonomous agents. We dig into the consolidation of the modern data stack, the economics driving more data products (Jevons paradox), and why product thinking, domain knowledge, and cross-functional skills will define the next wave of standout data professionals. We also cover practical steps for leaders and ICs: modernizing off legacy platforms, establishing safe AI adoption paths, codifying reusable “skills” and context for agents, and building validation utilities that keep the inner loop fast and trustworthy. Finally, Gleb shares how Datafold moved to fully AI-driven software delivery and why “outcomes over tools” is the emerging model for complex initiatives like data platform migrations—and how this reframes data quality for the AI era, emphasizing broad data access plus rich context over brittle human-centric tests.
Announcements- Hello and welcome to the Data Engineering Podcast, the show about modern data management
- If you lead a data team, you know this pain: Every department needs dashboards, reports, custom views, and they all come to you. So you're either the bottleneck slowing everyone down, or you're spending all your time building one-off tools instead of doing actual data work. Retool gives you a way to break that cycle. Their platform lets people build custom apps on your company data—while keeping it all secure. Type a prompt like 'Build me a self-service reporting tool that lets teams query customer metrics from Databricks—and they get a production-ready app with the permissions and governance built in. They can self-serve, and you get your time back. It's data democratization without the chaos. Check out Retool at dataengineeringpodcast.com/retool today and see how other data teams are scaling self-service. Because let's be honest—we all need to Retool how we handle data requests.
- Your host is Tobias Macey and today I'm bringing back Gleb Mezhanskiy to talk about our predictions for the impact of AI on data engineering for 2026
Interview- Introduction
- How did you get involved in the area of data management?
- What are the concrete steps that teams need to be taking today to take advantage of agentic AI capabilities?
- What are the new guardrails/constraints/workflows that need to be in place before you let AI loose on your data systems?
- How do you balance the potential cost savings and productivity increases with the up-front investment and variability in inference spend?
Contact Info
Parting Question
- From your perspective, what is the biggest gap in the tooling or technology for data management today?
Closing Announcements
- Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The AI Engineering Podcast is your guide to the fast-moving world of building AI systems.
- Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
- If you've learned something or tried out a project from the show then tell us about it! Email [email protected] with your story.
Links
- Blog Post
- Datafold
- Claude Opus 4.5
- Harry Potter - Muggles
- Jevon's Paradox
- Modern Data Stack
- Dagster Compass
- Gravity Orion
- MCP == Model Context Protocol
- Qwen
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SAKapitel
Kapitel werden geladen …
Treat Metering Like Finance: Building Data Platforms for Consumption Economics
Mit Himant Goyal
Summary In this episode Himant Goyal, Senior Product Manager at Salesforce, talks about how data platform investments enable reliable, accurate metering for consumption-based business models. Himant explains …
Shownotes
Summary
In this episode Himant Goyal, Senior Product Manager at Salesforce, talks about how data platform investments enable reliable, accurate metering for consumption-based business models. Himant explains why consumption turns operations into a real-time optimization problem spanning metering, cost attribution, billing, governance, and cross-functional ownership. He explores the richness required in usage data to support sophisticated pricing, the importance of treating metering like a financial system, and the architectural foundations - event schemas, durable ingestion, normalization/validation, a usage ledger, and clear serving layers - needed to power near-real-time visibility with fine-grained drilldowns. He also digs into anti-patterns and reliability concerns such as late or duplicate data, time zone pitfalls, SLAs, and automated policy decisions for pipeline failures. Himant shares practical guidance for capturing usage events from products and logs, balancing push vs. pull and real-time vs. batch processing to manage costs. He highlights configurable metering and rate-card versioning for rapid onboarding of new products, and the cultural shift required for finance, product, and engineering to co-own metering.
Announcements- Hello and welcome to the Data Engineering Podcast, the show about modern data management
- If you lead a data team, you know this pain: Every department needs dashboards, reports, custom views, and they all come to you. So you're either the bottleneck slowing everyone down, or you're spending all your time building one-off tools instead of doing actual data work. Retool gives you a way to break that cycle. Their platform lets people build custom apps on your company data—while keeping it all secure. Type a prompt like 'Build me a self-service reporting tool that lets teams query customer metrics from Databricks—and they get a production-ready app with the permissions and governance built in. They can self-serve, and you get your time back. It's data democratization without the chaos. Check out Retool at dataengineeringpodcast.com/retool today and see how other data teams are scaling self-service. Because let's be honest—we all need to Retool how we handle data requests.
- Your host is Tobias Macey and today I'm interviewing Himant Goyal about how data platform investments support consumption based business models
- Introduction
- How did you get involved in managing the data products or data management?
- Can you start by outlining the types of businesses and products that are "consumption based" and the impact that it has on the economics of the company?
- What are the unique operational challenges that are presented by having consumption as the unit of cost?
- How does the availability and accessibility of metering data impact the level of detail/nuance that the business can employ in their pricing strategies?
- When we talk about the infrastructure for usage tracking, it often feels like a high-stakes stream processing problem. What are the core architectural components required to build a reliable metering pipeline?
- How do you think about the trade-offs between "push" models (application emits events) vs. "pull" models (the platform scrapes resource usage)?
- Accuracy is non-negotiable when data is tied directly to revenue. What are the strategies for ensuring idempotency and handling deduplication in the ingestion layer?
- How do you address the "late-arriving data" problem in a usage-based world, especially when dealing with monthly billing cycles or credit exhaustion?
- From an uptime and reliability perspective, should the metering system be in the critical path of the service itself?
- If the metering service is down, do you "fail open" and provide free service, or "fail closed" and impact availability? How do you build for that kind of resilience?
- One of the common pitfalls is treating metering like logging or observability. How do you ensure that usage metering is treated as a first-class product priority rather than an afterthought for the platform team?
- What does the interface look like for product engineers to "register" a new billable event without breaking the downstream data contract?
- Once you have this data, there is often a requirement for real-time visibility for the end user. What are the data modeling requirements to support both "high-volume ingestion" and "low-latency querying" for customer-facing billing dashboards?
- How do you bridge the gap between the raw event stream and the aggregated "billable unit" in the data warehouse or lakehouse?
- What are the most interesting, innovative, or unexpected ways that you have seen usage-based metering used?
- What are the most interesting, unexpected, or challenging lessons that you have learned while working on building consumption-based data platforms?
- When is usage-based metering the wrong choice? (e.g., When does the complexity of the data platform outweigh the economic benefits?)
- What are your predictions for the future of consumption-based data architectures?
Contact Info
Parting Question
- From your perspective, what is the biggest gap in the tooling or technology for data management today?
Links
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SAKapitel
Kapitel werden geladen …
Beyond the PDF: Rowan Cockett on Reproducible, Composable Science
Mit Rowan Cockett
Summary In this episode Rowan Cockett, co-founder and CEO of CurveNote and co-founder of the Continuous Science Foundation, talks about building data systems that make scientific research reproducible …
Shownotes
Summary
In this episode Rowan Cockett, co-founder and CEO of CurveNote and co-founder of the Continuous Science Foundation, talks about building data systems that make scientific research reproducible, reusable, and easier to communicate. He digs into the sociotechnical roots of the reproducibility crisis - from data integrity and access to entrenched publishing incentives and PDF-bound workflows. He explores open standards and tools like Jupyter, Jupyter Book, and the push toward cloud-optimized formats (e.g., Zarr), along with graceful degradation strategies that keep interactive research usable over time. Rowan details how CurveNote enables interactive, reproducible articles that spin up compute on demand while delegating large dataset storage to specialized partners, and how community efforts like the Continuous Science Foundation and initiatives with Creative Commons aim to fix credit, licensing, and attribution. He also discusses the Open Exchange Architecture (OXA) initiative to establish a modular, computational standard for sharing science, the momentum in computational biosciences and neuroscience, and why true progress hinges on interoperability and composability across data, code, and narrative.
Announcements- Hello and welcome to the Data Engineering Podcast, the show about modern data management
- If you lead a data team, you know this pain: Every department needs dashboards, reports, custom views, and they all come to you. So you're either the bottleneck slowing everyone down, or you're spending all your time building one-off tools instead of doing actual data work. Retool gives you a way to break that cycle. Their platform lets people build custom apps on your company data—while keeping it all secure. Type a prompt like 'Build me a self-service reporting tool that lets teams query customer metrics from Databricks—and they get a production-ready app with the permissions and governance built in. They can self-serve, and you get your time back. It's data democratization without the chaos. Check out Retool at dataengineeringpodcast.com/retool today and see how other data teams are scaling self-service. Because let's be honest—we all need to Retool how we handle data requests.
- Your host is Tobias Macey and today I'm interviewing Rowan Cockett about building data systems that make scientific research easier to reproduce
Interview
- Introduction
- How did you get involved in the area of data management?
- Can you describe what your interest is in reproducibility of scientific research?
- What role does data play in the set of challenges that plague reproducibility of published research?
- What are some of the notable changes in the areas of scientific process, and data systems that have contributed to the current crisis of reproducibility?
- Beyond technological shortcomings, what are the processes that lead to problematic experiment/research design, and how does that complicate the work of other teams trying to build on the experimental findings?
- How does a monolithic approach change the types of research that would be possible with more modular/composable experimentation and research?
- Focusing now on the data-oriented aspects of research, what are the habits of research teams that lead to friction and waste in storing, processing, publishing, and ultimately consuming the information that supports the research findings?
- What are the elements of the work that you are doing at the Continous Science Foundation and Curvenote to break the status quo?
- Are there any areas of study that you are more susceptible to friction and siloing of their data?
- What does a typical engagement with a research group look like as you try to improve the accessibility of their work?
- What are the most interesting, innovative, or unexpected ways that you have seen research data (re-)used?
- What are the most interesting, unexpected, or challenging lessons that you have learned while working on reproducibility of scientific research?
- What are the next set of challenges that you are focused on addressing in the research/reproducibility space?
Contact Info
Parting Question
- From your perspective, what is the biggest gap in the tooling or technology for data management today?
Closing Announcements
- Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The AI Engineering Podcast is your guide to the fast-moving world of building AI systems.
- Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
- If you've learned something or tried out a project from the show then tell us about it! Email [email protected] with your story.
Links
- Continuous Science Foundation
- Curvenote
- Zenodo
- Dryad
- HDF5
- Iceberg
- Zarr
- Myst Markdown
- Jupyter Notebook
- ArXiv
- Journal of Open Source Software (JOSS)
- Data Carpentry
- Software Carpentry
- Open Rxiv
- Bio Rxiv
- Med Rxiv
- Force 11
- JupyterBook
- Open Exchange Architecture (OXA)
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SAKapitel
Kapitel werden geladen …
Beyond Prompts: Practical Paths to Self‑Improving AI
Mit Raj Shukla
Summary In this episode Raj Shukla, CTO of SymphonyAI, explores what it really takes to build self‑improving AI systems that work in production. Raj unpacks how agentic systems interact with real-world …
Shownotes
Summary
In this episode Raj Shukla, CTO of SymphonyAI, explores what it really takes to build self‑improving AI systems that work in production. Raj unpacks how agentic systems interact with real-world environments, the feedback loops that enable continuous learning, and why intelligent memory layers often provide the most practical middle ground between prompt tweaks and full Reinforcement Learning. He discusses the architecture needed around models - data ingestion, sensors, action layers, sandboxes, RBAC, and agent lifecycle management - to reach enterprise-grade reliability, as well as the policy alignment steps required for regulated domains like financial crime. Raj shares hard-won lessons on tool use evolution (from bespoke tools to filesystem and Unix primitives), dynamic code-writing subagents, model version brittleness, and how organizations can standardize process and entity graphs to accelerate time-to-value. He also dives into pitfalls such as policy gaps and tribal knowledge, strategies for staged rollouts and monitoring, and where small models and cost optimization make sense. Raj closes with a vision for bringing RL-style improvement to enterprises without requiring a research team - letting businesses own the reasoning and memory layers that truly differentiate their AI systems.
Announcements- Hello and welcome to the Data Engineering Podcast, the show about modern data management
- If you lead a data team, you know this pain: Every department needs dashboards, reports, custom views, and they all come to you. So you're either the bottleneck slowing everyone down, or you're spending all your time building one-off tools instead of doing actual data work. Retool gives you a way to break that cycle. Their platform lets people build custom apps on your company data—while keeping it all secure. Type a prompt like 'Build me a self-service reporting tool that lets teams query customer metrics from Databricks—and they get a production-ready app with the permissions and governance built in. They can self-serve, and you get your time back. It's data democratization without the chaos. Check out Retool at dataengineeringpodcast.com/retool today and see how other data teams are scaling self-service. Because let's be honest—we all need to Retool how we handle data requests.
- Your host is Tobias Macey, and today I’m interviewing Raj Shukla about building self-improving AI systems — and how they enable AI scalability in real production environments.
Interview
- Introduction
- How did you get involved in AI/ML?
- Can you start by outlining what actually improves over time in a self-improving AI system?
- How is that different from simply improving a model or an agent?
- How would you differentiate between an agent/agentic system vs. a self-improving system?
- One of the components that are becoming common in agentic architectures is a "memory" layer. What are some of the ways that contributes to a self-improvement feedback loop?
- In what ways are memory layers insufficient for a generalized self-improvement capability?
- For engineering and technology leaders, what are the key architectural and operational steps you recommend to build AI that can move from pilots into scalable, production systems?
- One of the perennial challenges for technology leaders is how to build AI systems that scale over time.
- How has AI changed the way you think about long-term advantage?
- How do self-improvement feedback loops contribute to AI scalability in real systems?
- What are some of the other key elements necessary to build a truly evolutionary AI system?
- What are the hidden costs of building these AI systems that teams should know before starting? I’m talking about enterprise who are deploying AI into their internal mission-critical workflows.
- What are the most interesting, innovative, or unexpected ways that you have seen self-improving AI systems implemented?
- What are the most interesting, unexpected, or challenging lessons that you have learned while working on evolutionary AI systems?
- What are some of the ways that you anticipate agentic architectures and frameworks evolving to be more capable of self-improvement?
Contact Info
Closing Announcements
- Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The AI Engineering Podcast is your guide to the fast-moving world of building AI systems.
- Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
- If you've learned something or tried out a project from the show then tell us about it! Email [email protected] with your story.
Parting Question
- From your perspective, what are the biggest gaps in tooling, technology, or training for AI systems today?
Links
- Symphony AI
- Reinforcement Learning
- Agentic Memory
- In-Context Learning
- Context Engineering
- Few-Shot Learning
- OpenClaw
- Deep Research Agent
- RAG == Retrieval Augmented Generation
- Agentic Search
- Google Gemma Models
- Ollama
The intro and outro music is from Hitman's Lovesong feat. Paola Graziano by The Freak Fandango Orchestra/CC BY-SA 3.0Kapitel
Kapitel werden geladen …
Orion at Gravity: Trustworthy AI Analysts for the Enterprise
Mit Lucas Thelosen und Drew Gilson
Summary In this episode of the Data Engineering Podcast, Lucas Thelosen and Drew Gilson, co-founders of Gravity, discuss their vision for agentic analytics in the enterprise, enabled by semantic layers and …
Shownotes
Summary
In this episode of the Data Engineering Podcast, Lucas Thelosen and Drew Gilson, co-founders of Gravity, discuss their vision for agentic analytics in the enterprise, enabled by semantic layers and broader context engineering. They share their journey from Looker and Google to building Orion, an AI analyst that combines data semantics with rich business context to deliver trustworthy and actionable insights. Lucas and Drew explain how Orion uses governed, role-specific "custom agents" to drive analysis, recommendations, and proactive preparation for meetings, while maintaining accuracy, lineage transparency, and human-in-the-loop feedback. The conversation covers evolving views on semantic layers, agent memory, retrieval, and operating across messy data, multiple warehouses, and external context like documents and weather. They emphasize the importance of trust, governance, and the path to AI coworkers that act as reliable colleagues. Lucas and Drew also share field stories from public companies where Orion has surfaced board-level issues, accelerated executive prep with last-minute research, and revealed how BI investments are actually used, highlighting a shift from static dashboards to dynamic, dialog-driven decisions. They stress the need for accessible (non-proprietary) models, managing context and technical debt over time, and focusing on business actions - not just metrics - to unlock real ROI.
Announcements- Hello and welcome to the Data Engineering Podcast, the show about modern data management
- If you lead a data team, you know this pain: Every department needs dashboards, reports, custom views, and they all come to you. So you're either the bottleneck slowing everyone down, or you're spending all your time building one-off tools instead of doing actual data work. Retool gives you a way to break that cycle. Their platform lets people build custom apps on your company data—while keeping it all secure. Type a prompt like 'Build me a self-service reporting tool that lets teams query customer metrics from Databricks—and they get a production-ready app with the permissions and governance built in. They can self-serve, and you get your time back. It's data democratization without the chaos. Check out Retool at dataengineeringpodcast.com/retool today and see how other data teams are scaling self-service. Because let's be honest—we all need to Retool how we handle data requests.
- Your host is Tobias Macey and today I'm interviewing Lucas Thelosen and Drew Gilson about the application of semantic layers to context engineering for agentic analytics
Interview
- Introduction
- How did you get involved in the area of data management?
- Can you start by digging into the practical elements of what is involved in the creation and maintenance of a "semantic layer"?
- How does the semantic layer relate to and differ from the physical schema of a data warehouse?
- In generative AI and agentic systems the latest term of art is "context engineering". How does a semantic layer factor into the context management for an agentic analyst?
- What are some of the ways that LLMs/agents can help to populate the semantic layer?
- What are the cases where you want to guard against hallucinations by keeping a human in the loop?
- Beyond a physical semantic layer, what are the other elements of context that you rely on for guiding the activities of your agents?
- What are some utilities that you have found helpful for bootstrapping the structural guidelines for an existing warehouse environment?
- What are the most interesting, innovative, or unexpected ways that you have seen Orion used?
- What are the most interesting, unexpected, or challenging lessons that you have learned while working on Orion?
- When is Orion the wrong choice?
- What do you have planned for the future of Orion?
Contact Info
Parting Question
- From your perspective, what is the biggest gap in the tooling or technology for data management today?
Closing Announcements
- Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The AI Engineering Podcast is your guide to the fast-moving world of building AI systems.
- Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
- If you've learned something or tried out a project from the show then tell us about it! Email [email protected] with your story.
Links
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SAKapitel
Kapitel werden geladen …
From Models to Momentum: Uniting Architects and Engineers with ER/Studio
Mit Jamie Knowles und Ryan Hirsch
Summary In this episode of the Data Engineering Podcast, Jamie Knowles (Product Director) and Ryan Hirsch (Product Marketing Manager) discuss the importance of enterprise data modeling with ER/Studio. They …
Shownotes
Summary
In this episode of the Data Engineering Podcast, Jamie Knowles (Product Director) and Ryan Hirsch (Product Marketing Manager) discuss the importance of enterprise data modeling with ER/Studio. They highlight how clear, shared semantic models are a foundational discipline for modern data engineering, preventing semantic drift, speeding up delivery, and reducing rework. Jamie explains that ER/Studio helps teams define logical models that translate into physical designs and code across warehouses and analytics platforms, while maintaining traceability and governance. The conversation also touches on how AI increases the tolerance for ambiguity, but doesn't fix unclear definitions - it amplifies them. Jamie and Ryan describe ER/Studio's integrations with governance tools, collaboration features like TeamServer, reverse engineering, and metadata bridges, as well as new AI-assisted modeling capabilities. They emphasize that most data problems are meaning problems, and investing in architecture and a semantic backbone can make engineering faster, governance simpler, and analytics more reliable.
Announcements- Hello and welcome to the Data Engineering Podcast, the show about modern data management
- If you lead a data team, you know this pain: Every department needs dashboards, reports, custom views, and they all come to you. So you're either the bottleneck slowing everyone down, or you're spending all your time building one-off tools instead of doing actual data work. Retool gives you a way to break that cycle. Their platform lets people build custom apps on your company data—while keeping it all secure. Type a prompt like 'Build me a self-service reporting tool that lets teams query customer metrics from Databricks—and they get a production-ready app with the permissions and governance built in. They can self-serve, and you get your time back. It's data democratization without the chaos. Check out Retool at dataengineeringpodcast.com/retool today and see how other data teams are scaling self-service. Because let's be honest—we all need to Retool how we handle data requests.
- Your host is Tobias Macey and today I'm interviewing Jamie Knowles and Ryan Hirsch about ER/Studio and the foundational role of enterprise data modeling in modern data engineering.
Interview
- Introduction
- How did you get involved in the area of data management?
- Can you describe what ER/Studio is and the story behind it?
- How has it evolved to handle the shift from traditional on-prem databases to modern, complex, and highly regulated enterprise environments?
- How do you define "Enterprise Data Architecture" today, and how does it differ from just managing a collection of pipelines in a modern data stack?
- In your view, what are the distinct responsibilities of a Data Architect versus a Data Engineer, and where is the critical overlap where they typically succeed or fail together?
- From what you see in the field, how often are the technical struggles of data engineering teams—like tool sprawl or "broken" pipelines—actually just "data meaning" problems in disguise?
- What is a logical data model, and why do you advocate for framing these as "knowledge models" rather than just technical diagrams?
- What are the long-term consequences, such as "semantic drift" or the erosion of trust, when organizations skip logical modeling to go straight to physical implementation and pipelines?
- What is the intersection of data modeling and data governance?
- What are the elements of integration between ER/Studio and governance platforms that reduce friction and time to delivery?
- For the engineers who worry that architecture and modeling slow down development, how does having a central design authority actually help teams scale and reduce downstream rework?
- What does a typical workflow look like across data architecture and data engineering for individuals and teams who are using ER/Studio as a core part of their modeling?
- What are the most interesting, innovative, or unexpected ways that you have seen ER/Studio used? * Context: Specifically regarding grounding AI initiatives or defining enterprise ontologies.
- What are the most interesting, unexpected, or challenging lessons that you have learned while working on ER/Studio?
- When is ER/Studio the wrong choice for a data team or a specific project?
- What do you have planned for the future of ER/Studio, particularly regarding AI and the "design-time" foundation of the data stack?
Contact Info
Parting Question
- From your perspective, what is the biggest gap in the tooling or technology for data management today?
Closing Announcements
- Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The AI Engineering Podcast is your guide to the fast-moving world of building AI systems.
- Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
- If you've learned something or tried out a project from the show then tell us about it! Email [email protected] with your story.
Links
- Idera
- Wherescape
- ER/Studio
- Entity-Relation Diagram (ERD)
- Business Keys
- Medallion Architecture
- RDF == Resource Description Framework
- Collibra
- Martin Fowler
- DB2
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SAKapitel
Kapitel werden geladen …
From Data Models to Mind Models: Designing AI Memory at Scale
Mit Vasilije Markovich
Summary In this episode of the Data Engineering Podcast, Vasilije "Vas" Markovich, founder of Cognee, discusses building agentic memory, a crucial aspect of artificial intelligence that enables systems to …
Shownotes
Summary
In this episode of the Data Engineering Podcast, Vasilije "Vas" Markovich, founder of Cognee, discusses building agentic memory, a crucial aspect of artificial intelligence that enables systems to learn, adapt, and retain knowledge over time. He explains the concept of agentic memory, highlighting the importance of distinguishing between permanent and session memory, graph+vector layers, latency trade-offs, and multi-tenant isolation to ensure safe knowledge sharing or protection. The conversation covers practical considerations such as storage choices (Redis, Qdrant, LanceDB, Neo4j), metadata design, temporal relevance and decay, and emerging research areas like trace-based scoring and reinforcement learning for improving retrieval. Vas shares real-world examples of agentic memory in action, including applications in pharma hypothesis discovery, logistics control towers, and cybersecurity feeds, as well as scenarios where simpler approaches may suffice. He also offers guidance on when to add memory, pitfalls to avoid (naive summarization, uncontrolled fine-tuning), human-in-the-loop realities, and Cognee's future plans: revamped session/long-term stores, decision-trace research, and richer time and transformation mechanisms. Additionally, Vas touches on policy guardrails for agent actions and the potential for more efficient "pseudo-languages" for multi-agent collaboration.
Announcements- Hello and welcome to the Data Engineering Podcast, the show about modern data management
- If you lead a data team, you know this pain: Every department needs dashboards, reports, custom views, and they all come to you. So you're either the bottleneck slowing everyone down, or you're spending all your time building one-off tools instead of doing actual data work. Retool gives you a way to break that cycle. Their platform lets people build custom apps on your company data—while keeping it all secure. Type a prompt like 'Build me a self-service reporting tool that lets teams query customer metrics from Databricks—and they get a production-ready app with the permissions and governance built in. They can self-serve, and you get your time back. It's data democratization without the chaos. Check out Retool at dataengineeringpodcast.com/retool today and see how other data teams are scaling self-service. Because let's be honest—we all need to Retool how we handle data requests.
- Your host is Tobias Macey and today I'm interviewing Vasilije Markovic about agentic memory architectures and applications
Interview
- Introduction
- How did you get involved in the area of data management?
- Can you start by giving an overview of the different elements of "memory" in an agentic context?
- storage and retrieval mechanisms
- how to model memories
- how does that change as you go from short-term to long-term?
- managing scope and retrieval triggers
- What are some of the useful triggers in an agent architecture to identify whether/when/what to create a new memory?
- How do things change as you try to build a shared corpus of memory across agents?
- What are the most interesting, innovative, or unexpected ways that you have seen agentic memory used?
- What are the most interesting, unexpected, or challenging lessons that you have learned while working on Cognee?
- When is a dedicated memory layer the wrong choice?
- What do you have planned for the future of Cognee?
Contact Info
Parting Question
- From your perspective, what is the biggest gap in the tooling or technology for data management today?
Closing Announcements
- Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The AI Engineering Podcast is your guide to the fast-moving world of building AI systems.
- Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
- If you've learned something or tried out a project from the show then tell us about it! Email [email protected] with your story.
Links
- Cognee
- AI Engineering Podcast Episode
- [Kimball Memory](
- Cognitive Science
- Context Window
- RAG == Retrieval Augmented Generation
- Memory Types
- Redis Vector Store
- Qdrant
- Vector on Edge
- Milvus
- LanceDB
- KuzuDB
- Neo4J
- Mem0
- Zepp Graphiti
- A2A (Agent-to-Agent) Protocol
- Snowplow
- Reinforcement Learning
- Model Finetuning
- OpenClaw
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SAKapitel
Kapitel werden geladen …
Prompt Management, Tracing, and Evals: The New Table Stakes for GenAI Ops
Mit Aman Agarwal
Summary In this episode of the Data Engineering Podcast, Aman Agarwal, creator of OpenLit, discusses the operational groundwork required to run LLM-powered applications reliably and cost-effectively. He …
Shownotes
Summary
In this episode of the Data Engineering Podcast, Aman Agarwal, creator of OpenLit, discusses the operational groundwork required to run LLM-powered applications reliably and cost-effectively. He highlights common blind spots that teams face, including opaque model behavior, runaway token costs, and brittle prompt management, and explains how OpenTelemetry-native observability can turn these black-box interactions into stepwise, debuggable traces across models, tools, and data stores. Aman showcases OpenLit's approach to open standards, vendor-neutral integrations, and practical features such as fleet-managed OTEL collectors, zero-code Kubernetes instrumentation, prompt and secret management, and evaluation workflows. They also explore experimentation patterns, routing across models, and closing the loop from evals to prompt/dataset improvements, demonstrating how better visibility reshapes design choices from prototype to production. Aman shares lessons learned building in the open, where OpenLit fits and doesn't, and what's next in context management, security, and ecosystem integrations, providing resources and examples of multi-database observability deployments for listeners.
Announcements- Hello and welcome to the Data Engineering Podcast, the show about modern data management
- If you lead a data team, you know this pain: Every department needs dashboards, reports, custom views, and they all come to you. So you're either the bottleneck slowing everyone down, or you're spending all your time building one-off tools instead of doing actual data work. Retool gives you a way to break that cycle. Their platform lets people build custom apps on your company data—while keeping it all secure. Type a prompt like 'Build me a self-service reporting tool that lets teams query customer metrics from Databricks—and they get a production-ready app with the permissions and governance built in. They can self-serve, and you get your time back. It's data democratization without the chaos. Check out Retool at dataengineeringpodcast.com/retool today and see how other data teams are scaling self-service. Because let's be honest—we all need to Retool how we handle data requests.
- Your host is Tobias Macey and today I'm interviewing Aman Agarwal about the operational investments that are necessary to ensure you get the most out of your AI models
Interview
- Introduction
- How did you get involved in the area of AI/data management?
- Can you start by giving your assessment of the main blind spots that are common in the existing AI application patterns?
- As teams adopt agentic architectures, how common is it to fall prey to those same blind spots?
- There are numerous tools/services available now focused on various elements of "LLMOps". What are the major components necessary for a minimum viable operational platform for LLMs?
- There are several areas of overlap, as well as disjoint features, in the ecosystem of tools (both open source and commercial). How do you advise teams to navigate the selection process? (point solutions vs. integrated tools, and handling frameworks with only partial overlap)
- Can you describe what OpenLit is and the story behind it?
- How would you characterize the feature set and focus of OpenLit compared to what you view as the "major players"?
- Once you have invested in a platform like OpenLit, how does that change the overall development workflow for the lifecycle of AI/agentic applications?
- What are the most complex/challenging elements of change management for LLM-powered systems? (e.g. prompt tuning, model changes, data changes, etc.)
- How can the information collected in OpenLit be used to develop a self-improvement flywheel for agentic systems?
- Can you describe the architecture and implementation of OpenLit?
- How have the scope and goals of the project changed since you started working on it?
- Given the foundational aspects of the project that you have built, what are some of the adjacent capabilities that OpenLit is situated to expand into?
- What are the sharp edges and blind spots that are still challenging even when you have OpenLit or similar integrated?
- What are the most interesting, innovative, or unexpected ways that you have seen OpenLit used?
- What are the most interesting, unexpected, or challenging lessons that you have learned while working on OpenLit?
- When is OpenLit the wrong choice?
- What do you have planned for the future of OpenLit?
Contact Info
Parting Question
- From your perspective, what is the biggest gap in the tooling or technology for data/AI management today?
Closing Announcements
- Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The AI Engineering Podcast is your guide to the fast-moving world of building AI systems.
- Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
- If you've learned something or tried out a project from the show then tell us about it! Email [email protected] with your story.
Links
- OpenLit
- Fleet Hub
- OpenTelemetry
- LangFuse
- LangSmith
- TensorZero
- AI Engineering Podcast Episode
- Traceloop
- Helicone
- Clickhouse
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SAKapitel
Kapitel werden geladen …
From Legacy to AI-Ready: How MongoDB AMP Accelerates Modernization
Mit Shilpa Kolhar
SummaryIn this episode, Shilpa Kolhar, SVP of Product and Engineering at MongoDB, discusses using MongoDB as a unified foundation for AI-driven and agentic applications. She explains how the Application …
Shownotes
Summary
In this episode, Shilpa Kolhar, SVP of Product and Engineering at MongoDB, discusses using MongoDB as a unified foundation for AI-driven and agentic applications. She explains how the Application Modernization Platform (AMP) accelerates the transition from legacy relational systems to a document-first architecture, driven by the need for AI-readiness and speed of change. Shilpa highlights MongoDB's features, such as its native JSON document model, Atlas Vector Search, auto-embeddings, and integrated search, which help eliminate drift and latency across operational data, indexing, and vectors, emphasizing the importance of keeping context, transactions, and embeddings together for real-time AI use cases. She shares best practices for re-architecting legacy systems, including schema validation and versioning patterns to tame schema drift, aggregation pipelines for consistent reads, and pragmatic standardization across services, while also detailing AMP's approach to scoping large estates and the balance of LLM-powered automation with human-in-the-loop governance.
Announcements- Hello and welcome to the Data Engineering Podcast, the show about modern data management
- If you lead a data team, you know this pain: Every department needs dashboards, reports, custom views, and they all come to you. So you're either the bottleneck slowing everyone down, or you're spending all your time building one-off tools instead of doing actual data work. Retool gives you a way to break that cycle. Their platform lets people build custom apps on your company data—while keeping it all secure. Type a prompt like 'Build me a self-service reporting tool that lets teams query customer metrics from Databricks—and they get a production-ready app with the permissions and governance built in. They can self-serve, and you get your time back. It's data democratization without the chaos. Check out Retool at dataengineeringpodcast.com/retool today and see how other data teams are scaling self-service. Because let's be honest—we all need to Retool how we handle data requests.
- Your host is Tobias Macey and today I'm interviewing Shilpa Kolhar about using MongoDB as the foundation for AI-driven applications
- Introduction
- How did you get involved in the area of data management?
- Can you describe what MongoDB is and the core primitives that it offers?
- The MongoDB engine has gone through substantial evolution since it was first introduced over 20 years ago. What are some of the most notable features that have been added in recent years?
- You recently launched the MongoDB Application Modernization Platform (AMP). What are the key elements of modernization that it is focused on?
- How do the core primitives of the MongoDB engine align with modernization objectives?
- There is a lot of attention being paid now to AI applications where data is the most critical element for success. What are the features of MongoDB that lend itself to being the context store for generative AI services?
- Besides the data used for context and grounding, AI applications also want to track user interactions and form short and long term memory to improve the system over time. How can MongoDB assist in that work as well?
- While the lack of schema enforcement on write can be beneficial to rapid evolution of software, it can also be a detriment if not managed well. How can MongoDB help in avoiding schema drift over time that leads to old data being incompatible with current code?
- What are the most interesting, innovative, or unexpected ways that you have seen MongoDB used?
- What are the most interesting, unexpected, or challenging lessons that you have learned while working on MongoDB and application modernization?
- When is MongoDB/AMP the wrong choice?
- What do you have planned for the future of AMP?
Parting Question- From your perspective, what is the biggest gap in the tooling or technology for data management today?
- Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The AI Engineering Podcast is your guide to the fast-moving world of building AI systems.
- Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
- If you've learned something or tried out a project from the show then tell us about it! Email [email protected] with your story.
- MongoDB
- MongoDB AMP
- Google Gemini
- Voyage AI
- Qdrant
- ChromaDB
- Weaviate
- Pinecone
- MongoDB Autoembedding
- Retool
- ODM == Object Document Mapper
- RAG == Retrieval Augmented Generation
- Agentic Memory
Kapitel
Kapitel werden geladen …
Branches, Diffs, and SQL: How Dolt Powers Agentic Workflows
Mit Tim Sehn
Summary In this episode Tim Sehn, founder and CEO of DoltHub, talks about Dolt - the world’s first version‑controlled SQL database - and why Git‑style semantics belong at the heart of data systems and AI …
Shownotes
Summary
In this episode Tim Sehn, founder and CEO of DoltHub, talks about Dolt - the world’s first version‑controlled SQL database - and why Git‑style semantics belong at the heart of data systems and AI workflows. Tim explains how Dolt combines a MySQL/Postgres‑compatible interface with a novel storage engine built on a “Prollytree” to enable fast, row‑level branching, merging, and diffs of both schema and data. He digs into real production use cases: powering applications that expose version control to end users, reproducible ML feature stores, managing massive configuration for games, and enabling safe agentic writes via branch‑based review flows. He compares Dolt’s approach to LakeFS, Neon, and PlanetScale, and explores developer workflows unlocked by decentralized clones, full audit logs, and PR‑style data reviews.
Announcements- Hello and welcome to the Data Engineering Podcast, the show about modern data management
- If you lead a data team, you know this pain: Every department needs dashboards, reports, custom views, and they all come to you. So you're either the bottleneck slowing everyone down, or you're spending all your time building one-off tools instead of doing actual data work. Retool gives you a way to break that cycle. Their platform lets people build custom apps on your company data—while keeping it all secure. Type a prompt like 'Build me a self-service reporting tool that lets teams query customer metrics from Databricks—and they get a production-ready app with the permissions and governance built in. They can self-serve, and you get your time back. It's data democratization without the chaos. Check out Retool at dataengineeringpodcast.com/retool today and see how other data teams are scaling self-service. Because let's be honest—we all need to Retool how we handle data requests.
- Your host is Tobias Macey and today I'm interviewing Tim Sehn about Dolt, a version controlled database engine and its applications for agentic workflows
Interview
- Introduction
- How did you get involved in the area of data management?
- Can you describe what Dolt is and the story behind it?
- What are the key use cases that you are focused on solving by adding version control to the database layer?
- There are numerous projects related to different aspects of versioning in different data contexts (e.g. LakeFS, Datomic, etc.). What are the versioning semantics that you are focused on?
- You position Dolt as "the database for AI". How does data versioning relate to AI use cases?
- What types of AI systems are able to make best use of Dolt's versioning capabilities?
- Can you describe how Dolt and Doltgres are implemented?
- How have the design and scope of the project changed since you first started working on it?
- What are some of the architecture and integration patterns around relational databases that change when you introduce version control semantics as a core primitive?
- What are some anti-patterns that you have seen teams develop around Dolt's versioning functionality?
- What are the most interesting, innovative, or unexpected ways that you have seen Dolt used?
- What are the most interesting, unexpected, or challenging lessons that you have learned while working on Dolt?
- When is Dolt the wrong choice?
- What do you have planned for the future of Dolt?
Contact Info
Parting Question
- From your perspective, what is the biggest gap in the tooling or technology for data management today?
Closing Announcements
- Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The AI Engineering Podcast is your guide to the fast-moving world of building AI systems.
- Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
- If you've learned something or tried out a project from the show then tell us about it! Email [email protected] with your story.
Links
- Dolt
- DoltHub
- Stockmarket Data
- LakeFS
- Datomic
- Git
- MySQL
- Prolly Tree
- Neon
- Django
- Feature Store
- MCP Server
- Nessie
- Iceberg
- PlanetScale
- O(NlogN) Big O Complexity
- B-Tree
- Git Merge
- Git Rebase
- AST == Abstract Syntax Tree
- Supabase
- CockroachDB
- Document Database
- MongoDB
- Gastown
- Beads
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SAKapitel
Kapitel werden geladen …
Logical First, Physical Second: A Pragmatic Path to Trusted Data
Mit Jamie Knowles
Summary In this episode of the Data Engineering Podcast Jamie Knowles, Product Director for ER/Studio, talks about data architecture and its importance in driving business meaning. He discusses how data …
Shownotes
Summary
In this episode of the Data Engineering Podcast Jamie Knowles, Product Director for ER/Studio, talks about data architecture and its importance in driving business meaning. He discusses how data architecture should start with business meaning, not just physical schemas, and explores the pitfalls of jumping straight to physical designs. Jamie shares his practical definition of data architecture centered on shared semantic models that anchor transactional, analytical, and event-driven systems. The conversation covers strategies for evolving an architecture in tandem with delivery, including defining core concepts, aligning teams through governance, and treating the model as a living product. He also examines how generative AI can both help and harm data architecture, accelerating first drafts but amplifying risk without a human-approved ontology. Jamie emphasizes the importance of doing the hard work upfront to make meaning explicit, keeping models simple and business-aligned, and using tools and patterns to reuse that meaning everywhere.
Announcements- Hello and welcome to the Data Engineering Podcast, the show about modern data management
- If you lead a data team, you know this pain: Every department needs dashboards, reports, custom views, and they all come to you. So you're either the bottleneck slowing everyone down, or you're spending all your time building one-off tools instead of doing actual data work. Retool gives you a way to break that cycle. Their platform lets people build custom apps on your company data—while keeping it all secure. Type a prompt like 'Build me a self-service reporting tool that lets teams query customer metrics from Databricks—and they get a production-ready app with the permissions and governance built in. They can self-serve, and you get your time back. It's data democratization without the chaos. Check out Retool at dataengineeringpodcast.com/retool today and see how other data teams are scaling self-service. Because let's be honest—we all need to Retool how we handle data requests.
- You’re a developer who wants to innovate—instead, you’re stuck fixing bottlenecks and fighting legacy code. MongoDB can help. It’s a flexible, unified platform that’s built for developers, by developers. MongoDB is ACID compliant, Enterprise-ready, with the capabilities you need to ship AI apps—fast. That’s why so many of the Fortune 500 trust MongoDB with their most critical workloads. Ready to think outside rows and columns? Start building at MongoDB.com/Build
- Composable data infrastructure is great, until you spend all of your time gluing it together. Bruin is an open source framework, driven from the command line, that makes integration a breeze. Write Python and SQL to handle the business logic, and let Bruin handle the heavy lifting of data movement, lineage tracking, data quality monitoring, and governance enforcement. Bruin allows you to build end-to-end data workflows using AI, has connectors for hundreds of platforms, and helps data teams deliver faster. Teams that use Bruin need less engineering effort to process data and benefit from a fully integrated data platform. Go to dataengineeringpodcast.com/bruin today to get started. And for dbt Cloud customers, they'll give you $1,000 credit to migrate to Bruin Cloud.
- Your host is Tobias Macey and today I'm interviewing Jamie Knowles about the impact that a well-developed data architecture (or lack thereof) has on data engineering work
Interview- Introduction
- How did you get involved in the area of data management?
- Can you start by giving your definition of "data architecture" and what it encompasses?
- How does the nuance change depending on the type of system you are designing? (e.g. data warehouse vs. transactional application database vs. event-driven streaming service)
- In application teams that are large enough there is typically a software architect, but that work often ends up happening organically through trial and error. Who is the responsible party for designing and enforcing a proper data architecture?
- There have been several generational shifts in approach to data warehouse projects in particular. What are some of the anti-patterns that crop up when there is no-one forming a strong opinion on the design/architecture of the warehouse?
- The current stage is largely defined by the ELT pattern. What are some of the ways that workflow can encourage shortcuts?
- Often the need for a proper architecture isn't felt until an organic architecture has developed. What are some of the ways that teams can short-circuit that pain and iterate toward a more sustainable design?
- The common theme in all of the data architecture conversations that I've had is the need for business involvement. There is also a strong push for the business to just want the engineers to deliver data. What are some of the ways that AI utilities can help to accelerate delivery while also capturing business context?
- For teams that are already neck deep in a messy architecture, what are the strategies and tactics that they need to start working toward today to get to a better data architecture?
- What are the most interesting, innovative, or unexpected ways that you have seen teams approach the creation and implementation of their data architecture?
- What are the most interesting, unexpected, or challenging lessons that you have learned while working in data architecture?
- How do you see the introduction of AI at each stage of the data lifecycle changing the ways that teams think about their architectural needs?
Contact Info
Parting Question- From your perspective, what is the biggest gap in the tooling or technology for data management today?
Closing Announcements- Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The AI Engineering Podcast is your guide to the fast-moving world of building AI systems.
- Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
- If you've learned something or tried out a project from the show then tell us about it! Email [email protected] with your story.
Links
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SAKapitel
Kapitel werden geladen …
Your Data, Your Lake: How Observe Uses Iceberg and Streaming ETL for Observability
Mit Jacob Leverich
Summary In this episode Jacob Leverich, cofounder and CTO of Observe, talks about applying lakehouse architectures to observability workloads. Jacob discusses Observe’s decision to leverage cloud-native …
Shownotes
Summary
In this episode Jacob Leverich, cofounder and CTO of Observe, talks about applying lakehouse architectures to observability workloads. Jacob discusses Observe’s decision to leverage cloud-native warehousing and open table formats for scale and cost efficiency. He digs into the core pain points teams face with fragmented tools, soaring costs, and data silos, and how a lakehouse approach - paired with streaming ingest via OpenTelemetry, Kafka-backed durability, curated/columnarized tables, and query orchestration - can deliver low-latency, interactive troubleshooting across logs, metrics, and traces at petabyte scale. He also explore the practicalities of loading and organizing telemetry by use case to reduce read amplification, the role of Iceberg (including v3’s JSON shredding) and Snowflake’s implementation, and why open table formats enable “your data in your lake” strategies.
Announcements- Hello and welcome to the Data Engineering Podcast, the show about modern data management
- If you lead a data team, you know this pain: Every department needs dashboards, reports, custom views, and they all come to you. So you're either the bottleneck slowing everyone down, or you're spending all your time building one-off tools instead of doing actual data work. Retool gives you a way to break that cycle. Their platform lets people build custom apps on your company data—while keeping it all secure. Type a prompt like 'Build me a self-service reporting tool that lets teams query customer metrics from Databricks—and they get a production-ready app with the permissions and governance built in. They can self-serve, and you get your time back. It's data democratization without the chaos. Check out Retool at dataengineeringpodcast.com/retool today and see how other data teams are scaling self-service. Because let's be honest—we all need to Retool how we handle data requests.
- You’re a developer who wants to innovate—instead, you’re stuck fixing bottlenecks and fighting legacy code. MongoDB can help. It’s a flexible, unified platform that’s built for developers, by developers. MongoDB is ACID compliant, Enterprise-ready, with the capabilities you need to ship AI apps—fast. That’s why so many of the Fortune 500 trust MongoDB with their most critical workloads. Ready to think outside rows and columns? Start building at MongoDB.com/Build
- Composable data infrastructure is great, until you spend all of your time gluing it together. Bruin is an open source framework, driven from the command line, that makes integration a breeze. Write Python and SQL to handle the business logic, and let Bruin handle the heavy lifting of data movement, lineage tracking, data quality monitoring, and governance enforcement. Bruin allows you to build end-to-end data workflows using AI, has connectors for hundreds of platforms, and helps data teams deliver faster. Teams that use Bruin need less engineering effort to process data and benefit from a fully integrated data platform. Go to dataengineeringpodcast.com/bruin today to get started. And for dbt Cloud customers, they'll give you $1,000 credit to migrate to Bruin Cloud.
- Your host is Tobias Macey and today I'm interviewing Jacob Leverich about how data lakehouse technologies can be applied to observability for unlimited scale and orders of magnitude improvement on economics
Interview
- Introduction
- How did you get involved in the area of data management?
- Can you start by giving an overview of what the major pain points have been in the observability space? (e.g. limited scale/retention, costs, integration fragmentation)
- What are the elements of the ecosystem and tech stacks that led to that state of the world?
- What are you building at Observe that circumvents those pain points?
- What are the major ecosystem evolutions that make this a feasible architecture? (e.g. columnar storage, distributed compute, protocol consolidation)
- Can you describe the architecture of the Observe platform?
- How have the design of the platform evolved/changed direction since you first started working on it?
- What was your process for determining which core technologies to build on top of?
- What were the missing pieces that you had to engineer around to get a cohesive and performant platform?
- The perennial problem with observability systems and data lakes is their tendency to succumb to entropy. What are the guardrails that you are relying on to help customers maintain a well-structured and usable repository of information?
- Data lakehouses are excellent for flexibility and scaling to massive data volumes, but they're not known for being fast. What are the areas of investment in the ecosystem that is changing that narrative?
- As organizations overcome the constraints of limited retention periods and anxiety over cost, what new use cases does that unlock for their observability data?
- How do AI applications/agents change the requirements around observability data? (collection, scale, complexity, applications, etc.)
- What are the most interesting, innovative, or unexpected ways that you have seen Observe/lakehouse technologies used for observability?
- What are the most interesting, unexpected, or challenging lessons that you have learned while working on Observe?
- When is Observe/lakehouse technologies the wrong choice?
- What do you have planned for the future of Observe?
Contact Info
Parting Question
- From your perspective, what is the biggest gap in the tooling or technology for data management today?
Closing Announcements
- Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The AI Engineering Podcast is your guide to the fast-moving world of building AI systems.
- Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
- If you've learned something or tried out a project from the show then tell us about it! Email [email protected] with your story.
Links
- Observe Inc.
- Lakehouse Architecture
- Splunk
- Observability
- RSyslog
- GlusterFS
- Dremel
- Drill
- BigQuery
- Snowflake SIGMOD Paper
- Prometheus
- Datadog
- NewRelic
- AppDynamics
- DynaTrace
- Loki
- Cortex
- Mimir
- Tempo
- Cardinality
- FluentBit
- FluentD
- OpenTelemetry
- OTLP == OpenTelemetry Line Protocol
- Kafka
- VPC Flow Logs
- Read Amplification
- Lance
- Iceberg
- Hudi
- PromQL
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SAKapitel
Kapitel werden geladen …
Semantic Operators Meet Dataframes: Building Context for Agents with FENIC
Mit Kostas Pardalis
Summary In this episode Kostas Pardalis talks about Fenic - an open-source, PySpark-inspired dataframe engine designed to bring LLM-powered semantics into reliable data engineering workflows. Kostas shares why …
Shownotes
Summary
In this episode Kostas Pardalis talks about Fenic - an open-source, PySpark-inspired dataframe engine designed to bring LLM-powered semantics into reliable data engineering workflows. Kostas shares why today’s data infrastructure assumptions (BI-first, expert-operated, CPU-bound) fall short for AI-era tasks that are increasingly inference- and IO-bound. He explores how Fenic introduces semantic operators (e.g., semantic filter, extract, join) as first-class citizens in the logical plan so the optimizer can reason about inference, costs, and constraints. This enables developers to turn unstructured data into explicit schemas, compose transformations lazily, and offload LLM work safely and efficiently. He digs into Fenic’s architecture (lazy dataframe API, logical/physical plans, Polars execution, DuckDB/Arrow SQL path), how it exposes tools via MCP for agent integration, and where it fits in context engineering as a companion for memory/state management in agentic systems.
Announcements- Hello and welcome to the Data Engineering Podcast, the show about modern data management
- You’re a developer who wants to innovate—instead, you’re stuck fixing bottlenecks and fighting legacy code. MongoDB can help. It’s a flexible, unified platform that’s built for developers, by developers. MongoDB is ACID compliant, Enterprise-ready, with the capabilities you need to ship AI apps—fast. That’s why so many of the Fortune 500 trust MongoDB with their most critical workloads. Ready to think outside rows and columns? Start building at MongoDB.com/Build
- Composable data infrastructure is great, until you spend all of your time gluing it together. Bruin is an open source framework, driven from the command line, that makes integration a breeze. Write Python and SQL to handle the business logic, and let Bruin handle the heavy lifting of data movement, lineage tracking, data quality monitoring, and governance enforcement. Bruin allows you to build end-to-end data workflows using AI, has connectors for hundreds of platforms, and helps data teams deliver faster. Teams that use Bruin need less engineering effort to process data and benefit from a fully integrated data platform. Go to dataengineeringpodcast.com/bruin today to get started. And for dbt Cloud customers, they'll give you $1,000 credit to migrate to Bruin Cloud.
- If you lead a data team, you know this pain: Every department needs dashboards, reports, custom views, and they all come to you. So you're either the bottleneck slowing everyone down, or you're spending all your time building one-off tools instead of doing actual data work. Retool gives you a way to break that cycle. Their platform lets people build custom apps on your company data—while keeping it all secure. Type a prompt like 'Build me a self-service reporting tool that lets teams query customer metrics from Databricks—and they get a production-ready app with the permissions and governance built in. They can self-serve, and you get your time back. It's data democratization without the chaos. Check out Retool at dataengineeringpodcast.com/retool today and see how other data teams are scaling self-service. Because let's be honest—we all need to Retool how we handle data requests.
- Your host is Tobias Macey and today I'm interviewing Kostas Pardalis about Fenic, an opinionated, PySpark-inspired DataFrame framework for building AI and agentic applications
Interview
- Introduction
- How did you get involved in the area of data management?
- Can you describe what Fenic is and the story behind it?
- What are the core problems that you are trying to address with Fenic?
- Dataframes have become a popular interface for doing chained transformations on structured data. What are the benefits of using that paradigm for LLM use-cases?
- Can you describe the architecture and implementation of Fenic?
- How have the design and scope of the project changed since you first started working on it?
- You position Fenic as a means of bringing reliability to LLM-powered transformations. What are some of the anti-patterns that teams should be aware of when getting started with Fenic?
- What are some of the most common first steps that teams take when integrating Fenic into their pipelines or applications?
- What are some of the ways that teams should be thinking about using Fenic and semantic operations for data pipelines and transformations?
- How does Fenic help with context engineering for agentic use cases?
- What are some examples of toolchains/workflows that could be replaced with Fenic?
- How does Fenic integrate with the broader ecosystem of data and AI frameworks? (e.g. Polars, Arrow, Qdrant, LangChan/Pydantic AI)
- What are the most interesting, innovative, or unexpected ways that you have seen Fenic used?
- What are the most interesting, unexpected, or challenging lessons that you have learned while working on Fenic?
- When is Fenic the wrong choice?
- What do you have planned for the future of Fenic?
Contact Info
Parting Question
- From your perspective, what is the biggest gap in the tooling or technology for data management today?
Closing Announcements
- Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The AI Engineering Podcast is your guide to the fast-moving world of building AI systems.
- Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
- If you've learned something or tried out a project from the show then tell us about it! Email [email protected] with your story.
Links
- Fenic
- RudderStack
- Podcast Episode
- Trino
- Starburst
- Trino Project Tardigrade
- Typedef AI
- dbt
- PySpark
- UDF == User-Defined Function
- LOTUS
- Pandas
- Polars
- Relational Algebra
- Arrow
- DuckDB
- Markdown
- Pydantic AI
- AI Engineering Podcast Episode
- LangChain
- Ray
- Dask
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SAKapitel
Kapitel werden geladen …
Beyond Dashboards: How Data Teams Earn a Seat at the Table
Mit Goutham Budati
Summary In this episode Goutham Budati about his Data–Perspective–Action framework and how it empowers data teams to become true business partners. Gautham traces his path from automating Excel reports to …
Shownotes
Summary
In this episode Goutham Budati about his Data–Perspective–Action framework and how it empowers data teams to become true business partners. Gautham traces his path from automating Excel reports to leading high‑impact data organizations, then breaks down why technical excellence alone isn’t enough: teams must pair reliable data systems with deliberate storytelling, clear problem framing, and concrete action plans. He digs into tactics for moving from reactive ticket-taking to proactive influence — weekly one‑page narratives, design-first discovery, sampling stakeholders for real pain points, and treating dashboards as living roadmaps. He also explores how to right-size technical scope, preserve trust in core metrics, organize teams as “build” and “storytelling” duos, and translate business macros and micros into resilient system designs.
Announcements- Hello and welcome to the Data Engineering Podcast, the show about modern data management
- Composable data infrastructure is great, until you spend all of your time gluing it together. Bruin is an open source framework, driven from the command line, that makes integration a breeze. Write Python and SQL to handle the business logic, and let Bruin handle the heavy lifting of data movement, lineage tracking, data quality monitoring, and governance enforcement. Bruin allows you to build end-to-end data workflows using AI, has connectors for hundreds of platforms, and helps data teams deliver faster. Teams that use Bruin need less engineering effort to process data and benefit from a fully integrated data platform. Go to dataengineeringpodcast.com/bruin today to get started. And for dbt Cloud customers, they'll give you $1,000 credit to migrate to Bruin Cloud.
- You’re a developer who wants to innovate—instead, you’re stuck fixing bottlenecks and fighting legacy code. MongoDB can help. It’s a flexible, unified platform that’s built for developers, by developers. MongoDB is ACID compliant, Enterprise-ready, with the capabilities you need to ship AI apps—fast. That’s why so many of the Fortune 500 trust MongoDB with their most critical workloads. Ready to think outside rows and columns? Start building at MongoDB.com/Build
- Your host is Tobias Macey and today I'm interviewing Goutham Budati about his data-perspective-action framework for empowering data teams to be more influential in the business
Interview
- Introduction
- How did you get involved in the area of data management?
- Can you describe what the Data-Perspective-Action framework is and the story behind it?
- What does it look like when someone operates at each of those three levels?
- How does that change the day-to-day work of an individual contributor?
- Why does technically excellent data work sometimes fail to drive decisions?
- How do you identify whether a data system or pipeline is actually creating value versus just existing?
- What's the moment when you realized that building reliable systems wasn't the same as enabling better decisions?
- Better decisions still need to be powered by reliable systems. How do you manage the tension of focusing on up-time against focusing on impact?
- What does it mean to add "Perspective" to data? How is that different from analysis or insights?
- How do you know when you're overwhelming stakeholders versus giving them what they need?
- What changes when you start designing systems to surface signal rather than just providing comprehensive data?
- How do you learn what business context matters for turning data into something actionable?
- What does it mean to design for Action from day one? How does that change what you build?
- How do you get stakeholders to actually act on data instead of just consuming it?
- Walk us through how you structure collaboration with business partners when you're trying to drive decisions, not just inform them.
- What's the relationship between iteration and trust when you're building data products?
- What does the transition from order-taker to strategic partner actually look like? What has to change?
- How do you position data work as driving the business rather than supporting it?
- Why does storytelling matter for data professionals? What role does it play that technical communication doesn't cover?
- What organizational structures or team setups help data people gain influence?
- Tell us about a time when you built something technically sound that failed to create impact. What did you learn?
- What are the common patterns in dysfunctional data organizations? What causes the breakdown?
- How do you rebuild credibility when you inherit a data function that's lost trust with the business?
- What's the relationship between technical excellence and stakeholder trust? Can you have one without the other?
- When is this framework the wrong lens? What situations call for a different approach?
- How do you balance the demand for technical depth with the need to develop business and communication skills?
- How should data professionals position themselves as AI and ML tools become more accessible?
- What shifts do you see coming in how businesses think about data work?
- How is your thinking about data impact evolving?
- For someone who recognizes they're focused purely on the technical work and wants to expand their impact—where should they start?
Contact Info
Parting Question
- From your perspective, what is the biggest gap in the tooling or technology for data management today?
Closing Announcements
- Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The AI Engineering Podcast is your guide to the fast-moving world of building AI systems.
- Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
- If you've learned something or tried out a project from the show then tell us about it! Email [email protected] with your story.
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA
Kapitel
Kapitel werden geladen …
Unfreezing The Data Lake: The Future-Proof File Format
Mit Xinyu Zeng
Summary In this episode PhD researcher Xinyu Zeng talks about F3, the “future-proof file format” designed to address today’s hardware realities and evolving workloads. He digs into the limitations of Parquet …
Shownotes
Summary
In this episode PhD researcher Xinyu Zeng talks about F3, the “future-proof file format” designed to address today’s hardware realities and evolving workloads. He digs into the limitations of Parquet and ORC - especially CPU-bound decoding, metadata overhead for wide-table projections, and poor random-access behavior for ML training and serving - and how F3 rethinks layout and encodings to be efficient, interoperable, and extensible. Xinyu explains F3’s two major ideas: a decoupled, flexible layout that separates IO units, dictionary scope, and encoding choices; and self-decoding files that embed WebAssembly kernels so new encodings can be adopted without waiting on every engine to upgrade. He discusses how table formats and file formats should increasingly be decoupled, potential synergies between F3 and table layers (including centralizing and verifying WASM kernels), and future directions such as extending WASM beyond encodings to indexing or filtering.
Announcements- Hello and welcome to the Data Engineering Podcast, the show about modern data management
- You’re a developer who wants to innovate—instead, you’re stuck fixing bottlenecks and fighting legacy code. MongoDB can help. It’s a flexible, unified platform that’s built for developers, by developers. MongoDB is ACID compliant, Enterprise-ready, with the capabilities you need to ship AI apps—fast. That’s why so many of the Fortune 500 trust MongoDB with their most critical workloads. Ready to think outside rows and columns? Start building at MongoDB.com/Build
- Composable data infrastructure is great, until you spend all of your time gluing it together. Bruin is an open source framework, driven from the command line, that makes integration a breeze. Write Python and SQL to handle the business logic, and let Bruin handle the heavy lifting of data movement, lineage tracking, data quality monitoring, and governance enforcement. Bruin allows you to build end-to-end data workflows using AI, has connectors for hundreds of platforms, and helps data teams deliver faster. Teams that use Bruin need less engineering effort to process data and benefit from a fully integrated data platform. Go to dataengineeringpodcast.com/bruin today to get started. And for dbt Cloud customers, they'll give you $1,000 credit to migrate to Bruin Cloud.
- Your host is Tobias Macey and today I'm interviewing Xinyu Zeng about the future-proof file format
Interview
- Introduction
- How did you get involved in the area of data management?
- Can you describe what the F3 project is and the story behind it?
- We have several widely adopted file formats (Parquet, ORC, Avro, etc.). Why do we keep creating new ones?
- Parquet is the format with perhaps the broadest adoption. What are the challenges that such wide use poses when trying to modify or extend the specification?
- The recent focus on vector data is perhaps the most visible change in storage requirements. What are some of the other custom types of data that might need to be supported in the file storage layer?
- Can you describe the key design principles of the F3 format?
- What are the engineering challenges that you faced while developing your implementation of the F3 proof-of-concept?
- The key challenge of introducing a new format is that of adoption. What are the provisions in F3 that might simplify the adoption of the format in the broader ecosystem? (e.g. integration with compute frameworks)
- What are some examples of features in data lake use cases that could be enabled by F3?
- What are some of the other ideas/hypotheses that you developed and discarded in the process of your reasearch?
- What are the most interesting, unexpected, or challenging lessons that you have learned while working on F3?
- What do you have planned for the future of F3?
Contact Info
Parting Question
- From your perspective, what is the biggest gap in the tooling or technology for data management today?
Links
- F3 Paper
- Formats Evaluation Paper
- F3 Github
- SAL Paper
- RisingWave
- Tencent Cloud
- Parquet
- Arrow
- Andy Pavlo
- Wes McKinney
- CMU Public Seminar
- VLDB
- ORC
- Protocol Buffers
- Lance
- PAX == Partition Attributes Across
- WASM == Web Assembly
- DataFusion
- DuckDB
- DuckLake
- Velox
- Vortex File Format
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA
Kapitel
Kapitel werden geladen …
From Context to Semantics: How Metadata Powers Agentic AI
Mit Sriharsha Chintalapani und Suresh Srinivas
Summary In this episode Suresh Srinivas and Sriharsha Chintalapani explore how metadata platforms are evolving from human-centric catalogs into the foundational context layer for AI and agentic systems. They …
Shownotes
Summary
In this episode Suresh Srinivas and Sriharsha Chintalapani explore how metadata platforms are evolving from human-centric catalogs into the foundational context layer for AI and agentic systems. They discuss the origins and growth of OpenMetadata and Collate, why “context” is necessary but “semantics” is critical for precise AI outcomes, and how a schema-first, API-first, unified platform enables discovery, observability, and governance in one workflow. They share how AI agents can now automate documentation, classification, data quality testing, and enforcement of policies, and why aligning governance with user identity and intent is essential as agentic access scales. They also dig into scalability strategies, MCP-based agent workflows, AI governance (including model/agent tracking), and the emerging convergence of big data with ontologies to deliver machine-understandable meaning.
Announcements- Hello and welcome to the Data Engineering Podcast, the show about modern data management
- Data teams everywhere face the same problem: they're forcing ML models, streaming data, and real-time processing through orchestration tools built for simple ETL. The result? Inflexible infrastructure that can't adapt to different workloads. That's why Cash App and Cisco rely on Prefect. Cash App's fraud detection team got what they needed - flexible compute options, isolated environments for custom packages, and seamless data exchange between workflows. Each model runs on the right infrastructure, whether that's high-memory machines or distributed compute. Orchestration is the foundation that determines whether your data team ships or struggles. ETL, ML model training, AI Engineering, Streaming - Prefect runs it all from ingestion to activation in one platform. Whoop and 1Password also trust Prefect for their data operations. If these industry leaders use Prefect for critical workflows, see what it can do for you at dataengineeringpodcast.com/prefect.
- Data migrations are brutal. They drag on for months—sometimes years—burning through resources and crushing team morale. Datafold's AI-powered Migration Agent changes all that. Their unique combination of AI code translation and automated data validation has helped companies complete migrations up to 10 times faster than manual approaches. And they're so confident in their solution, they'll actually guarantee your timeline in writing. Ready to turn your year-long migration into weeks? Visit dataengineeringpodcast.com/datafold today for the details.
- Composable data infrastructure is great, until you spend all of your time gluing it together. Bruin is an open source framework, driven from the command line, that makes integration a breeze. Write Python and SQL to handle the business logic, and let Bruin handle the heavy lifting of data movement, lineage tracking, data quality monitoring, and governance enforcement. Bruin allows you to build end-to-end data workflows using AI, has connectors for hundreds of platforms, and helps data teams deliver faster. Teams that use Bruin need less engineering effort to process data and benefit from a fully integrated data platform. Go to dataengineeringpodcast.com/bruin today to get started. And for dbt Cloud customers, they'll give you $1,000 credit to migrate to Bruin Cloud.
- You’re a developer who wants to innovate—instead, you’re stuck fixing bottlenecks and fighting legacy code. MongoDB can help. It’s a flexible, unified platform that’s built for developers, by developers. MongoDB is ACID compliant, Enterprise-ready, with the capabilities you need to ship AI apps—fast. That’s why so many of the Fortune 500 trust MongoDB with their most critical workloads. Ready to think outside rows and columns? Start building at MongoDB.com/Build
- Your host is Tobias Macey and today I'm interviewing Suresh Srinivas and Sriharsha Chintalapani about how metadata catalogs provide the context clues necessary to give meaning to your data for AI systems
- Introduction
- How did you get involved in the area of data management?
- Can you start by giving an overview of the roles that metadata catalogs are playing in the current state of the ecosystem?
- How has the OpenMetadata platform evolved over the past 4 years?
- How has the focus on LLMs/generative AI changed the trajectory of services like OpenMetadata?
- The initial set of use cases for data catalogs was to facilitate discovery and documentation of data assets for human consumption. What are the structural elements of that effort that have paid dividends for an AI audience?
- How does the AI audience change the requirements around the cataloging and presentation of metadata?
- One of the constant challenges in data infrastructure now is the tension of making data accessible to AI systems (agentic or otherwise) and incorporating AI into the inner loop of the service. What are the opportunities for bringing AI inside the boundaries of a system like OpenMetadata vs. as a client or consumer of the platform?
- The key phrase of the past ~2 years is "context engineering". What role does the metadata catalog play in that undertaking?
- What are the capabilities that the catalog needs to be able to effectively populate and curate that context?
- How much awareness does the LLM or agent need to have to be able to use the catalog effectively?
- What does a typical workflow/agent loop look like when it is using something like OpenMetadata in pursuit of knowledge that it needs to achieve an objective?
- How do agentic use cases strain the existing set of governance frameworks?
- What new considerations (procedural or technical) need to be factored into governance practices to balance velocity with security?
- What are the most interesting, innovative, or unexpected ways that you have seen OpenMetadata/Collate used in AI/agentic contexts?
- What are the most interesting, unexpected, or challenging lessons that you have learned while working on OpenMetadata/Collate?
- When is OpenMetadata/Collate the wrong choice?
- What do you have planned for the future of OpenMetadata?
Parting Question- From your perspective, what is the biggest gap in the tooling or technology for data management today?
Links- OpenMetadata
- Hadoop
- Hortonworks
- Context Engineering
- MCP == Model Context Protocol
- JSON Schema
- dbt
- LangSmith
- OpenMetadata MCP Server
- API Gateway
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA
Kapitel
Kapitel werden geladen …
From Data Engineering to AI Engineering: Where the Lines Blur
Summary In this solo episode of the Data Engineering Podcast, host Tobias Macey reflects on how AI has transformed the practice and pace of data engineering over time. Starting from its origins in the Hadoop …
Shownotes
Summary
In this solo episode of the Data Engineering Podcast, host Tobias Macey reflects on how AI has transformed the practice and pace of data engineering over time. Starting from its origins in the Hadoop and cloud warehouse era, he explores the discipline's evolution through ML engineering and MLOps to today's blended boundaries between data, ML, and AI engineering. The conversation covers how unstructured data is becoming more prominent, vectors and knowledge graphs are emerging as key components, and reliability expectations are changing due to interactive user-facing AI. The host also delves into process changes, including tighter collaboration, faster dataset onboarding, new governance and access controls, and the importance of treating experimentation and evaluation as fundamental testing practices.
Announcements- Hello and welcome to the Data Engineering Podcast, the show about modern data management
- Data teams everywhere face the same problem: they're forcing ML models, streaming data, and real-time processing through orchestration tools built for simple ETL. The result? Inflexible infrastructure that can't adapt to different workloads. That's why Cash App and Cisco rely on Prefect. Cash App's fraud detection team got what they needed - flexible compute options, isolated environments for custom packages, and seamless data exchange between workflows. Each model runs on the right infrastructure, whether that's high-memory machines or distributed compute. Orchestration is the foundation that determines whether your data team ships or struggles. ETL, ML model training, AI Engineering, Streaming - Prefect runs it all from ingestion to activation in one platform. Whoop and 1Password also trust Prefect for their data operations. If these industry leaders use Prefect for critical workflows, see what it can do for you at dataengineeringpodcast.com/prefect.
- Composable data infrastructure is great, until you spend all of your time gluing it together. Bruin is an open source framework, driven from the command line, that makes integration a breeze. Write Python and SQL to handle the business logic, and let Bruin handle the heavy lifting of data movement, lineage tracking, data quality monitoring, and governance enforcement. Bruin allows you to build end-to-end data workflows using AI, has connectors for hundreds of platforms, and helps data teams deliver faster. Teams that use Bruin need less engineering effort to process data and benefit from a fully integrated data platform. Go to dataengineeringpodcast.com/bruin today to get started. And for dbt Cloud customers, they'll give you $1,000 credit to migrate to Bruin Cloud.
- Data migrations are brutal. They drag on for months—sometimes years—burning through resources and crushing team morale. Datafold's AI-powered Migration Agent changes all that. Their unique combination of AI code translation and automated data validation has helped companies complete migrations up to 10 times faster than manual approaches. And they're so confident in their solution, they'll actually guarantee your timeline in writing. Ready to turn your year-long migration into weeks? Visit dataengineeringpodcast.com/datafold today for the details.
- You’re a developer who wants to innovate—instead, you’re stuck fixing bottlenecks and fighting legacy code. MongoDB can help. It’s a flexible, unified platform that’s built for developers, by developers. MongoDB is ACID compliant, Enterprise-ready, with the capabilities you need to ship AI apps—fast. That’s why so many of the Fortune 500 trust MongoDB with their most critical workloads. Ready to think outside rows and columns? Start building at MongoDB.com/Build
- Your host is Tobias Macey and today I'm interviewing reflecting about the increasingly blurry boundaries between data engineering and AI engineering
- Introduction
- I started this podcast in 2017, right when the term "Data Engineer" was becoming widely used for a specific job title with a reasonably well-understood set of responsibilities. This was in response to the massive hype around "data science" and consequent hiring sprees that characterized the mid-2000s to mid-2010s. The introduction of generative AI and AI Engineering to the technical ecosystem is changing the scope of responsibilities for data engineers and other data practitioners. Of note is the fact that:
- AI models can be used to process unstructured data sources into structured data assets
- AI applications require new types of data assets
- The SLAs for data assets related to AI serving are different from BI/warehouse use cases
- The technology stacks for AI applications aren't necessarily the same as for analytical data pipelines
- Because everything is so new there is not a lot of prior art, and the prior art that does exist isn't necessarily easy to find because of differences in terminology
- Experimentation has moved from being just an MLOps capability into being a core need for organizations
- From your perspective, what is the biggest gap in the tooling or technology for data management today?
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA
Kapitel
Kapitel werden geladen …
Malloy: Hierarchical Data, Semantic Models, and the Future of Analytics
Mit Michael Toy
Summary In this episode Michael Toy, co-creator of Malloy, talks about rethinking how we work with data beyond SQL. Michael shares the origins of Malloy from his and Lloyd Tabb’s experience at Looker, why SQL’s …
Shownotes
Summary
In this episode Michael Toy, co-creator of Malloy, talks about rethinking how we work with data beyond SQL. Michael shares the origins of Malloy from his and Lloyd Tabb’s experience at Looker, why SQL’s mental model often fights human problem solving, and how Malloy aims to be a composable, maintainable language that treats SQL as the assembly layer rather than something humans should write. He explores Malloy’s core ideas — semantic modeling tightly coupled with a query language, hierarchical data as the default mental model, and preserving context so analysis stays interactive and open-ended. He also digs into the developer experience and ecosystem: Malloy’s TypeScript implementation, VS Code integration, CLI, emerging notebook support, and how Malloy can sit alongside or replace parts of existing transformation workflows. Michael discusses practical trade-offs in language design, the surprising fit for LLM-generated queries, and near-term roadmap areas like dimensional filtering, better aggregation strategies across levels, and closing gaps that still require escaping to SQL. He closes with an invitation to contribute to the open-source project and help shape its evolution.
Announcements- Hello and welcome to the Data Engineering Podcast, the show about modern data management
- Data teams everywhere face the same problem: they're forcing ML models, streaming data, and real-time processing through orchestration tools built for simple ETL. The result? Inflexible infrastructure that can't adapt to different workloads. That's why Cash App and Cisco rely on Prefect. Cash App's fraud detection team got what they needed - flexible compute options, isolated environments for custom packages, and seamless data exchange between workflows. Each model runs on the right infrastructure, whether that's high-memory machines or distributed compute. Orchestration is the foundation that determines whether your data team ships or struggles. ETL, ML model training, AI Engineering, Streaming - Prefect runs it all from ingestion to activation in one platform. Whoop and 1Password also trust Prefect for their data operations. If these industry leaders use Prefect for critical workflows, see what it can do for you at dataengineeringpodcast.com/prefect.
- Data migrations are brutal. They drag on for months—sometimes years—burning through resources and crushing team morale. Datafold's AI-powered Migration Agent changes all that. Their unique combination of AI code translation and automated data validation has helped companies complete migrations up to 10 times faster than manual approaches. And they're so confident in their solution, they'll actually guarantee your timeline in writing. Ready to turn your year-long migration into weeks? Visit dataengineeringpodcast.com/datafold today for the details.
- Composable data infrastructure is great, until you spend all of your time gluing it together. Bruin is an open source framework, driven from the command line, that makes integration a breeze. Write Python and SQL to handle the business logic, and let Bruin handle the heavy lifting of data movement, lineage tracking, data quality monitoring, and governance enforcement. Bruin allows you to build end-to-end data workflows using AI, has connectors for hundreds of platforms, and helps data teams deliver faster. Teams that use Bruin need less engineering effort to process data and benefit from a fully integrated data platform. Go to dataengineeringpodcast.com/bruin today to get started. And for dbt Cloud customers, they'll give you $1,000 credit to migrate to Bruin Cloud.
- You’re a developer who wants to innovate—instead, you’re stuck fixing bottlenecks and fighting legacy code. MongoDB can help. It’s a flexible, unified platform that’s built for developers, by developers. MongoDB is ACID compliant, Enterprise-ready, with the capabilities you need to ship AI apps—fast. That’s why so many of the Fortune 500 trust MongoDB with their most critical workloads. Ready to think outside rows and columns? Start building at MongoDB.com/Build
- Your host is Tobias Macey and today I'm interviewing Michael Toy about Malloy, a modern language for building composable and maintainable analytics and data models on relational engines
Interview
- Introduction
- How did you get involved in the area of data management?
- Can you describe what Malloy is and the story behind it?
- What is the core problem that you are trying to solve with Malloy?
- There are countless projects that aim to reimagine/reinvent/replace SQL. What are the factors that make Malloy stand out in your mind?
- Who are the target personas for the Malloy language?
- One of the key success factors for any language is the ecosystem around it and the integrations available to it. How does Malloy fit in the toolchains and workflows for data engineers and analysts?
- Can you describe the key design and syntax elements of Malloy?
- How have the scope and focus of the language evolved since you first started working on it?
- How do the structure and semantics of Malloy change the ways that teams think about their data models?
- SQL-focused tools have gained prominence as the means of building the tranfromation stage of data pipelines. How would you characterize the capabilities of Malloy as a tool for building translation pipelines?
- What are the most interesting, innovative, or unexpected ways that you have seen Malloy used?
- What are the most interesting, unexpected, or challenging lessons that you have learned while working on Malloy?
- When is Malloy the wrong choice?
- What do you have planned for the future of Malloy?
Contact Info
Parting Question- From your perspective, what is the biggest gap in the tooling or technology for data management today?
- Malloy
- Lloyd Tabb
- SQL
- Looker
- LookML
- dbt
- Relational Algebra
- Typescript
- Ruby
- [Truffle](
- Malloy VSCode Plugin
- Malloy CLI
- Malloy Pick Statement
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA
Kapitel
Kapitel werden geladen …
Hier stehen die neuesten 40 von 521 Folgen. Ältere Folgen findest du im Feed des Podcasts.
Empfohlen von Data Engineering Podcast
Ähnliche Podcasts
Mehr Bildung-Podcasts
Ist das dein Podcast?
Bestätige es mit einem Eintrag in deinem Feed: Du bekommst hier einen Code, trägst ihn als podcast:txt in den Kanal deines Feeds ein, und wir prüfen ihn. Danach steht dein Podcast als „verifiziert“ bei SoundNode.