Exploring Complex Systems & Maintainable Data Science Pipelines
Show notes
What are the key characteristics of complex systems, and what are practical patterns for tackling complex coding problems? Christopher Trudeau is back on the show this week with another batch of PyCoder’s Weekly articles and projects.
Christopher covers an article about working on complex systems. The piece explores the distinction between complicated and complex problems. Each requires a fundamentally different approach to solve.
We dig into a couple of tutorials about creating repeatable data science pipelines. One covers validating DataFrames and tables with the pointblank library to build into a continuous integration workflow. The other explores how to use configuration files to build modular pipelines that avoid the pitfalls of hardcoded paths and parameters.
We also share other articles and projects from the Python community, including recent releases, three Python Enhancement Proposals (PEPs), the difficulties with breaking up lines of text, a static site generator for marimo notebooks, and a static analyzer for malicious Python code.
Course Spotlight: Sending Emails Using Python
Learn how to send emails with Python using SMTP and smtplib, attach files, format HTML messages, and personalize bulk emails.
Topics:
- 00:00:00 – Introduction
- 00:02:24 – Python 3.12.14, 3.11.16 and 3.10.21 Released
- 00:03:02 – Django Is Moving to an Annual Release Cycle
- 00:04:17 – PEP 828: Supporting ‘Yield From’ in Asynchronous Generators
- 00:04:56 – PEP 841: Adding Frozen Syntax to Optimize Immutable Types
- 00:07:04 – PEP 844: public and private Builtins
- 00:10:09 – Validating Data With Pointblank in Python
- 00:18:22 – Working on Complex Systems: Patterns I Learned at Google
- 00:22:14 – Video Course Spotlight
- 00:23:58 – Hydra for Python Configuration: Build Modular and Maintainable Pipelines
- 00:29:29 – Breaking Up (Lines) Is Hard to Do
- 00:35:00 – marimo-book: Static Site Generator for marimo Notebooks
- 00:38:06 – marimo-lsp: A language server and VS Code extension for marimo
- 00:38:31 – hexora: Static analysis of malicious Python code
- 00:40:01 – Thanks and goodbye
News:
- Python 3.12.14, 3.11.16 and 3.10.21 Released
- Django Is Moving to an Annual Release Cycle
- PEP 828: Supporting ‘Yield From’ in Asynchronous Generators (Accepted)
- PEP 841: Adding Frozen Syntax to Optimize Immutable Types (Draft)
- PEP 844: public and private Builtins (Draft)
Show Links:
- Validating Data With Pointblank in Python – Learn how to validate data in Python with Pointblank: declare quality checks, split clean from failing rows, and rerun validation plans from YAML.
- Working on Complex Systems: Patterns I Learned at Google – Explore the key characteristics of complex systems and learn practical patterns for tackling complex problems
- The Coder Cafe - Teiva Harsanyi – The Coder Cafe introduces vital software engineering concepts that will upgrade your day-to-day practice, regardless of your skill level. Discount Code (CODERCAFE40) Expires November 21st 2026.
- Hydra for Python Configuration: Build Modular and Maintainable Pipelines – Hardcoded paths and parameters can quickly drift out of sync across a data science project’s scripts. This article shows how to move them into Hydra configuration files, access values with dot notation, override settings from the command line, swap entire configuration groups, and run experiment sweeps with a single multi-run flag.
- Breaking Up (Lines) Is Hard to Do – Here’s a seemingly simple question: given a chunk of multi-line text, how do you split it and return an array? Unicode makes everything harder than it might first seem.
Projects:
- marimo-book: Static Site Generator for marimo Notebooks
- marimo-lsp: A language server and VS Code extension for marimo
- hexora: Static analysis of malicious Python code
Additional Links:
- pointblank - PyPI
- Episode #274: Preparing Data Science Projects for Production With Khuyen Tran
- marimo-book
- DartBrains - How can we understand how the brain works?
- marimo - Visual Studio Marketplace
Level up your Python skills with our expert-led courses: