Machine Learning Guide

The aggregate job market held, the entry-level door narrowed, and software postings sit a quarter below pre-pandemic. Why cheap implementation made specification, verification and domain scarce, how ML roles split five ways, and how to position.

• Try a walking desk - stay healthy & sharp while you learn & code
• More OCDevel shows - this one has siblings, each on its own subject and produced the same way

What coding agents did to programming and machine learning jobs by late 2026: the labor data, the mechanism behind it, how the ML career splintered into five roles, and a concrete positioning plan. Companion to the vibe coding trio (Vibe Coding in 2026, Inside a Coding Agent, Agentic Software Engineering) and the agents pair (AI Agents in 2026, OpenClaw and the Personal Agent).

Two different claims hide inside "AI is taking programming jobs": displacement (the role disappears, nobody is rehired) and task change (the role stays, the work shifts). They appear in different data. Displacement shows in unemployment and layoff reports; task change shows in what postings ask for and how teams are shaped. The aggregate evidence is mostly task change with one real pocket of displacement at the entry level, which sets up offensive advice for most listeners and defensive advice for new entrants.

• Aggregate. The BLS Employment Situation has unemployment at 4.1% with payrolls beating forecasts, far from the 10-20% Dario Amodei floated in the Axios "white-collar bloodbath" interview. Both he and Sam Altman have since softened the timeline; Altman said he was "delighted to be wrong" (Fortune, Time). The Yale Budget Lab tracker finds no discernible disruption. Goldman Sachs Research estimates a net drag of about 16,000 jobs a month across 800+ occupations, with a long-run baseline of 6-7% of workers displaced.
• Entry level. Stanford's Canaries in the Coal Mine (August 2026 paper, dashboard) puts 22-25 year olds in AI-exposed occupations 19% behind less-exposed peers, up from 15% a year earlier, driven by reduced hiring rather than separations and concentrated in automation-style exposure. The authors call these descriptive indicators, not causal estimates. Their software-developer case study finds young-developer pay grew somewhat faster than older developers' after ChatGPT, consistent with firms hiring fewer but better-paid juniors; the CPS sample is too small for a software-specific employment percentage. The EIG entry-level working paper is the main counterweight.
• Software demand. Indeed's software development postings index (Feb 2020 = 100) sits near 75, roughly a quarter below pre-pandemic and still drifting down, against a much smaller decline in total postings. Confounds stacked on top of AI: rate hikes, the Section 174 expensing change, and the 2021 overhire. SignalFire's State of Talent has new grads at 7% of Big Tech hires, down 25% from 2023 and over 50% from 2019, so half the collapse predates ChatGPT.
• Layoffs. Challenger, Gray & Christmas counts 116,175 of 529,914 announced 2026 cuts through August as AI-attributed (about 22%, already more than double all of 2025); AI led every month from March to July, then fell to 3,462 in August, while year-to-date cuts are down 41%.
• Grads and incumbents. The NY Fed college labor market data (2026:Q2) has computer science at 7.0% unemployment and 19.1% underemployment and computer engineering at 7.8% and 15.8%, against 5.6% and 42% for all recent graduates: worst on getting a job, among the best on getting a good one. CompTIA's tech jobs report has tech occupation unemployment at 2.8% and over 320,000 active postings asking for AI-related capabilities. The reversal wave: CNBC and Forbes on employers rehiring after AI cuts, Forrester's 55% regret figure, Robert Half's one-in-three refill figure, and the Klarna and IBM cases.
• Projections. BLS 2025-2035: software developers +10% from 1.72 million, data scientists +35%, computer programmers -7%.
• Measurement. METR's randomized trial of 16 experienced open-source developers on 246 real issues found AI made them 19% slower while they believed it sped them up 20%, so self-reported productivity is unreliable in both directions.

When implementation cost falls toward zero, value moves to specification, verification and domain knowledge. The BLS programmer-vs-developer split is that thesis in two rows. Andrew Ng's AI Rewards Generalists Who Can Build New Skills and his five-part AI Engineering Skills Map argue the bottleneck moved from how to build to what to build; David Autor calls AI a supplement to workers with judgment and domain knowledge. Juniors are hit because the traditional junior role was the commoditized part, and it was also the tuition for learning the other two skills. The ceiling on the mechanism shows in two benchmarks from the same year: OpenAI's GDPval, where the newest models win or tie against experts on most one-shot deliverables (with caveats about automated grading), against Scale's Remote Labor Index, where the best agent completed about 4% of real multi-day projects (via Carnegie). Agents produce artifacts; humans still run projects.

Data scientist demand is projected to grow three times as fast as developer demand, and Levels.fyi puts ML/AI-focused engineers in the US around $248k average total compensation. The title splintered into five roles, roughly by headcount: AI engineer (application layer: retrieval, tool use, agent loops, evals, context design); forward-deployed engineer (the Palantir-origin role the labs adopted, where domain is the constraint; see Anthropic's FDE posting); evals and AI quality (titles like Research Engineer, Model Evaluations on Anthropic's jobs board); inference, serving and platform infrastructure; and research engineer or scientist, the smallest and most competitive tier. "AI engineer" now means treating the model as a component with a failure distribution and designing the system around it. Prompt engineering as a standalone title, fine-tuning as a default move, and train-from-scratch generalist ML roles lost ground.

• Accelerationists (Amodei, Altman, Mustafa Suleyman): disruption within one to five years, entry-level first; strongest evidence is the benchmark curve. The aggregate prediction has failed so far and both leading voices softened it; Suleyman's 12-18 month clock has not expired.
• Skeptics (Yann LeCun, who left Meta to found AMI Labs on a world-model thesis; Gary Marcus; Daron Acemoglu, whose macro estimate is under 1% TFP gain over a decade): strongest evidence is the Remote Labor Index; weakest point is the cheap-but-imperfect case that reshapes jobs without replacing them.
• Pragmatists (Andrew Ng, Brynjolfsson, Autor): technology real, effects uneven, the question is which tasks move. Best track record so far because they predicted least. The Carnegie Endowment's three views cuts the map differently and is worth reading alongside. The Anthropic Economic Index (January, March) shows augmentation edging up on consumer chat while API and coding-agent usage stays automation-dominant.

Own a domain where correct answers require knowledge not on the internet. Own verification: reading diffs fast, writing the test before the bug, building the eval harness, catching reward-hacked tests. Run agents fluently and measure yourself rather than trusting the feeling (the METR gap). Ship agentic work in public with specs, tests, evals and review trail visible, the new portfolio. New entrants: don't look like the traditional junior; compete for the well-paid junior seats that remain, at companies with real domains, in the roles that are hiring.

Fundamentals first, because you can't verify what you don't understand: the Machine Learning Guide core episodes. Then the applied layer, which changes every few months: the vibe coding trio, the agents pair, the media trio. Then a domain and a project, which no course provides.

• MLA 22: Vibe Coding in 2026
• MLA 23: Inside a Coding Agent
• MLA 24: Agentic Software Engineering
• MLA 28: AI Agents in 2026
• MLA 29: OpenClaw and the Personal Agent

Every show Gnothi has produced, on AI, coding agents, video generation and agentic business, is at ocdevel.com/moremlg.

OpenClaw as the worked example of the always-on personal agent: gateway, markdown memory, heartbeats, skills, coding agents from your phone, hosted vs local models, and the 2026 security record (exposed instances, two critical CVEs, ClawHavoc) with the posture that makes it survivable.

• Try a walking desk - stay healthy & sharp while you learn & code
• More OCDevel shows - this one has siblings, each on its own subject and produced the same way

Second and last episode of the agents pair. AI Agents in 2026 covered the theory: loops, tools, memory, protocols, SDKs, evaluation. This one takes a single category, the always-on personal agent, and its most-cloned instance, OpenClaw, all the way down to the security posture required before it touches your inbox.

A personal agent is the agent loop with three additions: it runs continuously on a machine you control and can wake itself; its interface is messaging (WhatsApp, Telegram, Signal, iMessage, Slack) rather than a chat tab; and it holds your files, shell, browser, calendar and, if you allow it, email. Messaging removes the gap between having a thought and delegating it, and a chat thread is a natural home for asynchronous work that reports back later. The access that makes it useful is also the entire security problem.

OpenClaw is an MIT-licensed, self-hosted TypeScript agent created by Peter Steinberger in late 2025. It was released as Warelay, passed through claw-themed names, hit an Anthropic trademark complaint, and settled on OpenClaw at the end of January 2026 per Wikipedia; the org remains openclaw/openclaw. Steinberger joined OpenAI in February 2026 and stewardship moved to the OpenClaw Foundation, a US 501(c)(3) chaired by Dave Morin with a small full-time staff and donors including OpenAI, GitHub, Nvidia and Microsoft; the repo's README says OpenAI is a donor, not an owner, and there is no paid tier or hosted service. The project ships near-weekly CalVer releases plus an extended-stable line, and the GitHub blog's maintainer profile called it the fastest-growing project in the site's history. OpenClaw 2.0 (v2026.8.1) landed at the end of August.

Rewrites that keep the idea and shrink the surface, per OSS Insight's fork-wave analysis: nanobot (Python, small auditable core), ZeroClaw (Rust, single static binary), PicoClaw (Go, from Sipeed, embedded targets) and NanoClaw (TypeScript, container-first). Hosted versions are all third-party one-click deploys or small managed services; the foundation runs none.

One long-lived Node gateway per host binds to loopback on port 18789, owns every channel connection, routes inbound messages to sessions, loads context, calls the configured model, executes tools, streams the reply and persists everything under ~/.openclaw. Nodes are paired devices (laptop, phone, headless box) that lend the gateway local screen, camera and shell.

Memory is markdown in the agent workspace: AGENTS.md (operating instructions), SOUL.md (persona and boundaries), IDENTITY.md, USER.md (stable facts about you, with its own character budget), MEMORY.md (curated durable facts), a memory/ directory of daily notes, and a one-time BOOTSTRAP.md interview. The memory docs state there is no hidden state; a hybrid memory_search index covers the memory file and daily notes, a background "dreaming" sweep promotes recurring material into MEMORY.md, and a flush runs before context compaction.

Initiative comes from the heartbeat, a periodic main-session turn (30 minutes by default, 60 on subscription auth) that can stay silent via a no-reply marker, and from the automations scheduler (one-shot, interval or cron, delivered to a channel, a webhook or nowhere). HEARTBEAT.md is legacy; its checklist now lives in DB-backed scratch. Skills follow Anthropic's Agent Skills format (SKILL.md with name and description front matter) and install from ClawHub; the docs say to treat third-party skills as untrusted code and read them first. The browser tool drives a dedicated agent-owned Chrome/Brave/Edge profile through a loopback-only control service with a strict SSRF policy.

OpenClaw is a harness over sixty-plus providers using provider/model references, including Ollama, llama.cpp, LM Studio, vLLM and SGLang for local inference. The tradeoff is the one from the agents episode with higher stakes: everything the agent reads is forwarded to the model, so inbox triage on a hosted model sends your inbox to the provider. Frontier models are better at the judgment calls (is this urgent, is this instruction really from me), local is the only defensible choice for regulated third-party data, and a per-task split (local for reading private content, hosted for writing public content) is a common compromise. The docs make no claim about local-model quality inside OpenClaw. Subscription auth reuses an existing Claude CLI login or an OpenAI OAuth flow per the OAuth docs, which describe the Claude CLI path as sanctioned per Anthropic staff guidance and warn that a community proxy needs a terms check; no published Anthropic term was found either way.

The old Claude Code bridge skill is superseded by agent runtimes: built-in, Codex app-server, Claude CLI and a Copilot plugin, with external harnesses (Claude Code, Gemini CLI, OpenCode, Cursor) driven over the Agent Client Protocol through acpx. Tasks land in managed worktrees: isolated branches with checkpoints in a state DB, filesystem snapshots where supported, a cap around 100 live worktrees, and dirty or unpushed work never auto-cleaned. See Agentic Software Engineering for why worktrees are the right isolation unit.

The official IMAP plugin watches a mailbox, spawns an isolated restricted-reader session per allowed message, ranks trust by DMARC/SPF/DKIM, and does not send mail, which encodes the read-versus-write distinction the security section relies on. Calendar and most SaaS arrive as skills or MCP servers; Cisco's DefenseClaw announcement describes connecting email, calendar and Discord through Zapier-hosted MCP servers so a glue service holds the OAuth tokens.

Exposure: Bitsight, SecurityScorecard and Censys counted between 30,000 and 135,000 internet-facing instances in early 2026, roughly two thirds with no authentication; Censys confirmed 63,070 live instances at the end of March. Bugs: CVE-2026-25253 (CVSS 8.8), a one-click RCE where the control UI auto-connected to a gatewayUrl from the query string and leaked the auth token, worked even against loopback-bound instances and was patched in v2026.1.29 per the GitHub advisory; CVE-2026-32922 (CVSS 9.9) let a pairing token rotate itself into admin, fixed in v2026.3.11. Well over a hundred advisories were logged between February and April.

Supply chain: Koi Security's ClawHavoc report found 341 malicious ClawHub skills (335 from one campaign) disguised as wallets, trading bots and Workspace integrations, delivering the Atomic macOS Stealer and targeting always-on Mac minis; the count later passed 800 as the registry grew past 10,000 (The Hacker News, Unit 42). Cisco's skill research scanned about 31,000 agent skills, found a quarter with at least one vulnerability, and demonstrated exfiltration through an attacker-controlled Telegram bot.

Injection: Giskard exploited a live deployment for exfiltration and account takeover; PromptArmor showed Telegram and Discord link previews exfiltrate data with no click; CrowdStrike called a misconfigured instance "a powerful AI backdoor agent." Ambient: Wiz found Moltbook's database open with about 1.5 million API tokens; Meta banned OpenClaw on work devices and then acquired Moltbook; China restricted state use.

Responses: fast patches, loopback default, DM pairing codes for unknown senders, openclaw security audit, the VirusTotal partnership scanning every ClawHub skill with daily rescans, openclaw skills verify, publisher gating, and the layered access model in the security docs: DM modes, per-agent profiles, control-plane tool restrictions, node exec policies, sandbox and read-only variants, exec approvals, strict browser SSRF, and "one trust boundary per gateway." None of it fixes indirect prompt injection; the maintainers say scanning is not a silver bullet, skills remain arbitrary code, and the agent still holds real credentials. Cisco's open-source DefenseClaw adds pre-execution scanning and runtime allow/block enforcement.

Posture, as eight rules: loopback plus a tunnel (Tailscale or SSH) and an auth token even locally; the agent gets its own OS user, mailbox, calendar, browser profile and capped API keys; every integration starts read-only with human approval on irreversible writes; a dedicated box with nothing else on it; read every skill before enabling; design so a hijacked agent is a nuisance, not a breach, by shrinking the write surface; read the logs and memory files weekly; run the audit after every change and pin a version.

By mid-2026 the consumer frenzy had cooled and what remained was solo founders and small teams running it as infrastructure. Surviving uses share one shape, a scheduled or triggered read delivered to the chat you already use: morning briefings, read-only inbox triage with thresholds, watching deadlines and pipelines and even school lunch menus, voice memos returned as structured notes, lead research and CRM updates, and coding-agent dispatch from a phone. Value compounds through the memory files rather than any single automation, which is why the posture has to precede the setup.

Claude Cowork and OpenAI's ChatGPT Work give the delegate-a-task shape sandboxed and session-based, without messaging or always-on. The rewrites give a smaller, auditable surface. The SDKs from AI Agents in 2026 give one tight automation in an afternoon. The real decision is how much of your life you want in one process.

• AI Agents in 2026: Loops, Tools, Memory, Protocols, and Evaluation
• Inside a Coding Agent: Context, Instructions, Hooks, Skills, MCP
• Agentic Software Engineering: Specs, Verification, and the Review Loop

The Gnothi companion show, OCDevel Agentic Business, follows one business as agents take on research, software, sales and recurring operations.

What an AI agent actually is, why coding agents got good first, how memory really works, what MCP and A2A standardize, which SDKs are alive, how to evaluate on trajectories, and where the products stand after browser agents contracted.

• Try a walking desk - stay healthy & sharp while you learn & code
• More OCDevel shows - this one has siblings, each on its own subject and produced the same way

First of two episodes on AI agents. This one is the architecture: the loop, tools and verifiable feedback, memory, the protocols (MCP, A2A, computer use), the SDK landscape, evaluation and observability, the product map, and when multiple agents help. The next episode, OpenClaw and the Personal Agent, applies it to one always-on assistant with security as the centerpiece. Coding-agent products and mechanics live in the vibe coding sequence starting at MLA 22.

A chat model returns a message; a workflow is your code calling a model at fixed steps; an agent is a model that owns the control flow, choosing its next action from what it observes. That puts systems on a spectrum (chat, chat plus tools, workflows, agents) rather than in a binary, the framing Anthropic's Building Effective Agents uses. The loop itself is ReAct (Yao et al.): thought, action, observation, repeat, with the reasoning trace letting the model track and update a plan. What changed by 2026 is not the loop but the infrastructure around it, and every part of that infrastructure is an attack on per-step error compounding.

Function calling: you describe tools as schemas, the model emits a structured call, your code executes it and returns the observation. The model never runs anything itself, which is the security model. Writing effective tools for agents gives the practical rules: few high-impact tools, clear namespaces, meaningful identifiers, token-efficient responses, descriptions treated as prompt engineering. Effective context engineering for AI agents adds the overlap test: if a human cannot say which tool applies, neither can the agent. The central principle: coding agents got good first because tests and compilers give verifiable feedback that catches a bad step inside the same loop that made it. Find or manufacture the verifier before writing the prompt.

"Memory" means four things: the context window (the only memory the model has), retrieval from an external store, files on disk, and episodic records of prior sessions. Most agent memory is files. Anthropic's memory tool is a client-side file protocol (view, create, replace, insert, delete) against storage you own. The hard part is context management, and both labs converged on the same three mechanisms: context editing to clear stale tool results, compaction to summarize near the limit, and notes written to files before summarization. OpenAI's Responses API conversation state has the same shape with a compaction threshold and compact endpoint. Third-party layers Mem0, Letta (from MemGPT), and Zep (temporal knowledge graph) now compete with first-party primitives. Multi-session patterns: Effective harnesses for long-running agents.

Model Context Protocol is the agent-to-tool standard, now a Linux Foundation project with individual-maintainer governance. 2026 additions: elicitation (server asks the user mid-operation), an extensions mechanism, and the async Tasks extension for long-running tools. Every major SDK below consumes it; its cost is the context each connected server's tool list occupies. A2A is the agent-to-agent standard, Google-built, Linux Foundation-hosted, at v1.0 with a steering committee spanning AWS, Cisco, Google, IBM, Microsoft, Salesforce, SAP and ServiceNow. Strong governance, weak observed consumption; worth knowing, not yet worth building on for small teams. Computer use is the universal fallback: Anthropic's computer use tool (GA toolset with zoom and an automatic injection classifier), Google's Gemini computer use, open-source Browser Use, and Playwright MCP, which drives the accessibility tree instead of screenshots. Prefer API, then accessibility tree, then screenshots.

Both labs advise starting without a framework: Building Effective Agents and OpenAI's A Practical Guide to Building Agents. The 2026 SDKs have converged on that critique as thin harnesses around a loop.

• Claude Agent SDK: Claude Code's loop as a library (built-in tools, subagents, hooks, MCP, permissions, compaction); TypeScript and Python; pre-1.0.
• OpenAI Agents SDK: handoffs, guardrails, sessions, tracing on the Responses API. OpenAI deprecated the visual Agent Builder in favor of it.
• LangGraph and LangChain 1.x: stateful graph with checkpointing, interrupts, durable execution; create_agent as a minimal middleware harness. LangSmith is the separate tracing product.
• Google ADK: code-first hierarchical agent trees with native A2A; deploys to Vertex Agent Engine.
• Microsoft Agent Framework: GA successor to AutoGen and Semantic Kernel; Python and .NET.
• CrewAI: role-based crews, past 1.0, with a commercial management platform.
• smolagents: code agents that write Python instead of JSON calls; weakest maintenance signal on the list.
• Pydantic AI and Vercel AI SDK: typed validation-first agents in Python; loop control and agent abstraction in TypeScript.

Decision rule: machine-operating agent fast, Claude Agent SDK; lightweight handoffs, OpenAI; durable human-in-the-loop state, LangGraph; inside Google or Microsoft, their kit; to understand what you run, write the loop yourself first.

Agents are evaluated on trajectories, not answers: traces, task evals, cost per task. Traces follow the OpenTelemetry GenAI semantic conventions; products include LangSmith, Langfuse (open source, acquired by ClickHouse), Arize Phoenix, Braintrust, W&B Weave, and Helicone. Public benchmarks show the shape of a task eval: SWE-bench Verified, which OpenAI stopped reporting citing contamination; SWE-bench Pro; tau2-bench; Terminal-Bench 2.0; OSWorld-Verified; GDPval. Cost and reliability: Princeton's Holistic Agent Leaderboard (paper) and its reliability dashboard separate pass@k capability from pass^k reliability; METR time horizons with their own limitations note. Guardrails: OpenAI agent safety, NeMo Guardrails, Guardrails AI; prompt injection framed by Simon Willison's lethal trifecta and Google's CaMeL architectural defense.

Claude Cowork: "Claude Code for everyone," a sandboxed desktop agent with open-sourced plugins. ChatGPT agent remains; the Atlas browser was retired within a year, folded into ChatGPT and Codex. Google discontinued Project Mariner and moved the capability into Gemini and Antigravity, which absorbed Gemini CLI. Standalone browser agents contracted; the capability moved into models and existing apps. Still shipping: Perplexity Comet (free), Manus (ownership contested this year; check before building on it), Devin, Copilot Studio, Agentforce. Glue: n8n (AI Agent node inside a drawn workflow, MCP server trigger) and Zapier Agents with Zapier MCP. Browser-agent prompt injection is the documented security problem: the PleaseFix research note.

Two essays a day apart: How we built our multi-agent research system (orchestrator plus parallel subagents beat a single agent on research at roughly 15x the tokens; token usage explained most of the variance) and Cognition's Don't Build Multi-Agents (dispersed decisions and unshared context make it fragile). The disagreement is task shape. Parallelize independent, read-mostly work; keep stateful, sequential work single-threaded; prefer a small hierarchy where workers return findings rather than decisions.

• MLA 29: OpenClaw and the Personal Agent
• MLA 22: Vibe Coding in 2026
• MLA 23: Inside a Coding Agent
• MLA 24: Agentic Software Engineering

Companion show: Agentic Business on Gnothi follows one business as agents take on research, software, sales and operations.

How to automate AI media end to end: clone your own voice on open TTS, pick music that's actually licensed, run ComfyUI graphs headless, design around fal, Replicate and provider queues, finish with ffmpeg, and stay inside licensing at every layer.

• More OCDevel shows - this one has siblings, each on its own subject and produced the same way
• Companion show. This episode is the overview of the media pipeline. For weekly, hands-on coverage of the video half, from a first usable clip to scenes that cut together, listen to AI Video Generation on Gnothi.
• Try a walking desk - stay healthy & sharp while you learn & code

Once you need thirty clips with the same character, a narrator who sounds identical every episode, a matched music bed, word-accurate captions and a platform-safe export, the prompt is one node in a graph and the graph is the product. The engineering lives in the edges: how one model's output becomes the next model's input, how failures retry, what each job cost, and whether the run is reproducible next week. Model choice at the nodes is covered in the two sibling episodes; this one covers everything else: voice with ElevenLabs and Qwen3-TTS, licensed music, ComfyUI on your own card, the fal and Replicate APIs, ffmpeg assembly, and licensing.

ElevenLabs remains the reference point (current flagship Eleven v3, character-based pricing). Professional voice cloning is locked to the requester's own voice behind a live voice check, and the terms require consent attestation for any uploaded voice. On the open side, Breeze TTS 2 topped the open-weights column of the Artificial Analysis speech arena in August 2026, ahead of Fish Audio's S2 Pro; the code is Apache 2.0 but the weights are research/non-commercial, so it is not a commercial self-hosting option. The working set for programmers: Qwen3-TTS (Apache 2.0, 0.6B/1.7B, cloning from seconds of reference audio; the preset-speaker variant does not clone, see my Qwen3-TTS voice cloning guide), Chatterbox (MIT, emotion control, watermarked output), and Kokoro (82M parameters, Apache 2.0, faster than real time on CPU). Fish Audio plays both sides with open weights and a cheap hosted API. Quantized Qwen3-TTS runs podcast-length synthesis on CPU-only instances; see Quantized Qwen3-TTS on CPU and the broader open-source TTS roundup. Hosted alternatives for prototyping: OpenAI text-to-speech and Gemini speech generation.

Consent is the legal boundary. Tennessee's ELVIS Act added voice to right of publicity; the federal NO FAKES Act cleared Senate Judiciary in June 2026; EU AI Act Article 50 transparency duties apply from 2 August 2026; Denmark is amending copyright law to cover a person's face and voice.

Warner settled with both Suno and Udio; Universal settled with Udio, which became a no-download walled garden; UMG and Sony are still litigating against Suno, whose terms now grant commercial rights rather than ownership to paid subscribers. Eleven Music is trained on licensed data via Merlin and Kobalt deals and cleared for commercial use on self-serve plans, excluding film, TV and larger games; ElevenLabs sound effects are cleared on any paid plan and support loops. Google exposes Lyria and Lyria RealTime through the Gemini API. Open models: ACE-Step 1.5, YuE, Stable Audio Open (community license, best open option for short effects), HeartMuLa, and Meta's MusicGen, which is non-commercial.

ComfyUI is a workflow runtime with a GUI for designing graphs. Comfy raised $30M at a $500M valuation in April 2026 and ships a desktop app, Comfy Cloud, and API nodes that call paid providers from inside a local graph. Programmatic use is the same /prompt endpoint and websocket the front end uses: export the workflow in API format, patch fields, post, poll. Wrappers like comfyui-api and comfy-pack turn a graph into a scalable service. Alternatives: SwarmUI, InvokeAI, the Krita AI plugin.

Hardware: full-precision Flux.2 and Qwen-Image do not fit consumer cards; fp8 and GGUF quantization bring them to 16 to 24 GB. For video, the open Wan releases lag the API versions; the 5B variant does 720p on 24 GB (about 8 GB with Comfy offloading), the 14B variant officially wants 80 GB at full precision and needs GGUF to be consumer-viable, and Wan2GP targets low-VRAM cards. Rule: local for iteration, cloud for volume.

Start with an aggregator, move to a provider API only for a feature or price it lacks. fal is queue-first: submit, get a request ID, poll or webhook, per-output pricing on popular models, per GPU-second for custom deployments. Replicate has the broader catalog beyond image and video, bills per second of compute for open models, packages custom models with Cog, and joined Cloudflare with the same API. RunPod serverless is the raw GPU option for a custom ComfyUI graph. Provider APIs have converged on the same shape: Veo via the Gemini API (billed per output second, audio included), Kling API (post, store task_id, poll /v1/tasks), Runway API, ElevenLabs API.

Design rules: every generation is a job in a durable queue keyed on a hash of inputs, model and seed; store the provider's request ID next to your job ID; honor 429 retry-after with a token bucket per provider; persist prompt, seed, inputs and outputs in object storage; route to a second provider on 5xx. Cost per usable second is list price times your rejection rate.

ffmpeg is the programmer's editor: concat, overlay, sidechain ducking, caption burn-in, crop to 9:16, loudnorm and export. Human-in-the-loop editors: DaVinci Resolve (free version is a real editor; Studio unlocks most Neural Engine features), Descript with its transcript-as-timeline and Underlord assistant, and CapCut for short-form auto-captions, with a caution about its June 2025 terms change. Upscaling: Topaz retired Video AI for the subscription Topaz Video with the Astra model; open-side, SeedVR2 is single-step, runs on 8 GB and plugs into ComfyUI; Real-ESRGAN for clean stills; RIFE for frame interpolation. Captions: WhisperX gives word timestamps within about 50 ms via forced alignment plus diarization, emitting SRT/VTT; generate styled word-pop overlays from its JSON. Delivery: 1080x1920 9:16, H.264/AAC, roughly 10 to 12 Mbps, 30 fps; YouTube recommended upload settings; target -14 LUFS integrated with a -1 dBTP ceiling as the last pipeline step.

Five layers, and the output is only as clean as the dirtiest node. Weights: Apache/MIT models (Qwen3-TTS, Kokoro, Chatterbox, open Wan) are clean; FLUX.2 dev is non-commercial without a separate license; Stability's community license allows commercial use under a revenue threshold; MusicGen is non-commercial. Output: the Copyright Office holds that prompts alone are not authorship, and the Supreme Court denied cert in Thaler v. Perlmutter in March 2026, so keep evidence of the human selection and editing. Training data: licensed models are the safe path while label suits continue. People: right of publicity, get written consent. Disclosure: YouTube auto-labels via SynthID and C2PA content credentials since May 2026, and labels on Veo and C2PA-stamped content are permanent. Attach credentials and disclose.

Social clip (30 s, 9:16, run 200 times): character sheet from an image-editing model stored with prompt and seed -> templated script -> Qwen3-TTS narration with word timings -> per-shot image-to-video jobs via fal keyed on input hash, webhook completion -> cached licensed music bed -> ffmpeg concat, duck, styled captions from timing JSON, crop, loudnorm, 1080x1920 export -> content credentials and disclosure -> human review queue.

Narrated explainer (8 min, 16:9, weekly): human-written script (where copyright rests) -> chunked TTS stitched with short silences -> LLM shot list with timestamps tagged diagram/image/video -> deterministic diagrams, styled images, a few video clips upscaled with SeedVR2 -> licensed music and SFX generated once -> Resolve or ffmpeg assembly, captions from narration timings, loudnorm, 1080p/4K export -> title, chapters from the shot list, disclosure, credentials. Both are the same graph with different shot counts and aspect ratios.

• AI Image Generation and Editing in 2026
• AI Video Generation in 2026
• AI Agents covers agent orchestration of pipelines like these

The companion show for the video half of this pipeline is AI Video Generation on Gnothi.

Sora is shut down, Google runs two video models, Kling 3 does lip-synced dialogue, and open-weight MiniMax H3 is what you can actually fine-tune. What a usable clip costs, which models do native audio, how reference consistency works, and why the unit of work is the shot.

• Notes and resources at ocdevel.com/mlg/mla-26
• More OCDevel shows - this one has siblings, each on its own subject and produced the same way.
• Companion show: for weekly, hands-on coverage of the AI video pipeline, from a first usable clip to scenes that cut together, listen to AI Video Generation.
• Try a walking desk - stay healthy & sharp while you learn & code

Second of three episodes on AI media generation, covering Veo, Kling, Runway and MiniMax H3. Four questions: what a usable clip costs, which models generate sound and dialogue natively, how character and shot consistency work now, and where open-weight video fits for a programmer. Ends with the shot-to-scene mental model.

Native audio is now the baseline at the frontier: Veo 3.1, Kling 3.0, MiniMax H3 and LTX-2.5 sample audio and frames from one model, so lip movement and sound effects land on the right frame. Clips grew from four or five seconds to eight to fifteen, with a few models advertising thirty. Every serious product ships a reference-conditioning feature (Google "ingredients", Kling "elements", Runway references) that holds a character or object across clips. Image-to-video, not text-to-video, is the professional path: lock the first frame with an image model (see AI Image Generation and Editing), then ask the video model to move it. The old "storyteller vs animator" split resolved in favor of the animators.

Google now runs two video models in two places. Veo 3.1 is the developer baseline on the Gemini API and Vertex, in Quality, Fast and Lite tiers, all with native audio; clips are 4, 6 or 8 seconds, and 1080p/4K are upscales of the 8-second clip. It accepts up to three reference images, first-and-last-frame interpolation, and extend in 7-second steps up to 20 times; the Ingredients to Video update added identity consistency, native vertical and 4K upscaling. At Google I/O 2026 Google announced Gemini Omni; Gemini Omni Flash replaced Veo inside the Gemini app and Flow, taking text, image, video or audio as input and supporting conversational video-to-video editing. Per Flow's model matrix it currently tops out at 10 seconds and 720p. All output carries SynthID; the detector portal is still waitlisted.

OpenAI launched Sora 2 on September 30, 2025 with native audio and a free iOS app; it then hit a copyright reckoning over opt-out character use, SAG-AFTRA and Bryan Cranston pushback on likeness, and a court order barring the word "Cameo". In March 2026 OpenAI announced a two-stage shutdown: app and web closed April 26, 2026, API closes September 24, 2026. NBC's reporting attributes it to reallocating compute to coding, reasoning and enterprise; Sora continues only as internal world-model research. It stays in the episode as the case study of a strong model without a business.

Kuaishou's Kling 3.0 launched globally in March 2026 as a unified image, video and audio model: up to 15 seconds per shot, native 4K, and per the Kling Omni audio guide lip-synced dialogue in five languages with sound effects and ambience generated in the same pass. The control surface is the point: an elements library built from images or short reference video with per-element voice binding, multi-shot generation with continuity, motion transfer from a reference video, motion brush, six-axis camera control, extend and retake. Sold as a credit-based consumer app with commercial rights on paid tiers, plus a first-party API fronted in the West by fal and Replicate. fal's three per-second prices (audio off, audio on, voice control) make the cost of joint audio-video sampling visible.

Runway Gen-4.5 shipped December 1, 2025, briefly topped the Artificial Analysis leaderboard, and was candid about causal reasoning, object permanence and "success bias" failures. Runway's differentiator is editing: Aleph is video-to-video (new angles, relighting, add/remove objects, restyle), and per the Runway API changelog Aleph 2.0 takes 2-30 second inputs with up to five keyframes; Act-Two transfers a filmed performance onto a character. Gen-4.5's release notes do not claim native dialogue or effects. The Runway API now also resells ByteDance Seedance 2.5 (30-second clips, large reference budgets, audio) and Wan 3, and studio deals with Adobe, AMC Networks and Lionsgate anchor the enterprise story.

As of this recording the Artificial Analysis text-to-video arena (blind pairwise human preference) has none of Veo 3.1, Kling 3.0 or Gen-4.5 in its top five: Wan 3.0, Gemini Omni Flash, fal's post-trained MiniMax H3 Max, MiniMax H3, then Seedance 2.0. The headline products win on control, distribution and enterprise fit, not the taste test.

Naming matters for Wan: Wan 2.2 is Apache-2.0 open weight (14B MoE needing an 80GB GPU, or a 5B model for a 24GB card) and its GGUF quantizations still trend on Hugging Face; Wan 2.5, 2.7 and 3.0 have no published weights on the Wan-AI Hugging Face org or GitHub and are served as APIs. The open-weight center of gravity is MiniMax H3: 33B, native stereo audio, up to 2K, 4-15 seconds, a community license permitting commercial use, official ComfyUI workflows, and a LoRA and step-distillation ecosystem. LTX-2.5 (19B, native audio, community license free under $10M revenue) and HunyuanVideo-1.5 (8.3B, 14GB with offload, no audio) round out the runnable set; MAGI-2 is a preview with no confirmed license. Elsewhere: Luma shipped Ray3, Ray3 Modify and Ray3.14; Pika pivoted to effects, an agent and MCP on Pika 2.5; ByteDance's Seedance 2.0 and 2.5 ride Dreamina and CapCut distribution; Grok Imagine is a priced API video model outside the top ten; Higgsfield is an aggregator and creative suite, as are fal, Replicate and OpenArt on the developer side.

Every consistency feature is conditioning under a different name: text, reference images, first frame, last frame, and reference video are slots the denoiser attends to. A first frame is the strongest condition, which is why image-to-video wins. Reference characters (Veo's three images, Kling elements with voice binding, Seedance's dozens of references) fight identity drift, still the main failure mode. Start and end frames bound a camera move and let shots hand off to each other. Explicit camera controls beat prompt text. Video-to-video (Aleph, Luma Modify, Omni Flash, Kling) means fixing a nearly right shot instead of regenerating. Extend compounds drift, so use it to finish a shot, not build a scene. Open models add LoRAs: Musubi Tuner trains adapters for HunyuanVideo and Wan 2.x, and the tooling lags each new frontier open release by months.

Native audio means one sampling process produces waveform and frames, conditioned on each other; Veo 3.1 prices everything as video with audio, Kling 3.0 exposes it as a paid toggle, H3 and LTX-2.5 do it in open weights, Gen-4.5 and Wan 2.2 do not. Post-hoc remains a valid choice: MMAudio generates synchronized sound from finished video with an explicit alignment module, and ElevenLabs sound effects generate timed effects from text. Native dialogue holds for a line or two; longer talking heads still favor performance-driven tools like Act-Two. Voice and music proper are in The AI Media Pipeline.

As of this recording, from Gemini API pricing: Veo 3.1 Quality about $0.40/s with audio (720p/1080p), Fast about $0.10/s, Lite about $0.05/s; Gemini Omni Flash is billed per token, working out to roughly $0.10/s of 720p. From Runway API pricing: Gen-4.5 $0.12/s, Aleph 2 $0.28/s with a minimum. From fal: Kling v3 about $0.08/s silent and $0.13/s with audio, Wan 2.5 $0.05/s; Grok Imagine video $0.05-0.08/s. An 8-second Veo Quality shot with audio is a bit over $3; Kling or Gen-4.5 about $1. No vendor publishes success rates; budgeting four generations per usable shot puts a frontier clip with audio at $3-13 and a Fast or open-weight clip under $1. Iterate on the cheap tier, render on the expensive one.

Every model generates a shot: one continuous take, one camera, one action, 4-15 seconds. A scene is three to eight shots cut together, and continuity is your job: same references in every shot, first and last frames handing off, the same elements or LoRA, one audio bed over the cut. Storyboard as shots, lock first frames with an image model, iterate cheap, render expensive, fix with video-to-video, assemble in an editor. Assembly, voice, music, ComfyUI and driving it from code are the next episode.

• AI Image Generation and Editing in 2026
• The AI Media Pipeline: Voice, Music, ComfyUI, APIs, and Finishing
• Vibe Coding in 2026 for Grok's coding products

Editing replaced generation as the core task. How GPT Image 2.5, Google's Nano Banana line, Midjourney V8.2 and Flux 2 differ, what open weights and LoRAs buy you, ControlNet vs instruction editing, and how licensing and C2PA provenance work now.

• More OCDevel shows - this one has siblings, each on its own subject and produced the same way
• Companion show. This episode is the overview. For weekly, hands-on coverage of the full image and video pipeline, from a first usable clip to scenes that cut together, listen to AI Video Generation.
• Try a walking desk - stay healthy & sharp while you learn & code

First of three episodes on AI media generation (this one is images and editing; then video, then the pipeline). A decision guide rather than a leaderboard: where GPT Image, Nano Banana, Midjourney and Flux each fit in late 2026, why instruction-driven editing replaced generation as the core task, what open weights buy you, and how licensing and provenance work.

The 2025 "artist vs collaborator" split is over and the collaborators won. Every frontier image model now sits behind a language model that reads the prompt with world knowledge and accepts images as input, so the unit of work became "here is an image, change this one thing and keep everything else." Text rendering, precise instruction following and identity-preserving reference images all landed at once for one reason: the image model became, or was paired with, a multimodal language model. Generation from scratch is now the special case where the input image is empty.

The lineage runs gpt-image-1 (2025), gpt-image-2, then ChatGPT Images 2.5 in September 2026, with a precision variant (Sunburst) and a fast default (Flare); the same models power ChatGPT and the Images API. As of this recording it holds the top slots on both the LMArena text-to-image and image-edit boards and on Artificial Analysis. For editing it does mask inpainting, up to four reference images, and multi-turn editing via the Responses API. Its weaknesses are latency at high quality and occasional text and consistency slips. Billing is per token (text in, image in, image out) with a cached-input discount; see the gpt-image-2.5-sunburst model page.

Nano Banana was gemini-2.5-flash-image (August 2025, now legacy). It was succeeded by Nano Banana Pro (gemini-3-pro-image) and Nano Banana 2 (gemini-3.1-flash-image, February 2026), plus a Lite tier. Per the Gemini image generation docs, Nano Banana 2 accepts up to fourteen reference images (objects plus characters), outputs up to 4K, does multi-turn sequential editing, and can ground generation in Google Search; the Pro model adds style references and identity preservation across up to five subjects. Every output carries a SynthID watermark with no opt-out. Imagen appears to be superseded for new work, though no formal retirement notice was found.

Midjourney moved from V7 (2025) to V8 alpha in March 2026 and V8.2 as the default in July 2026 (version history). V8.2's Edit Model replaces Omni Reference, Character Reference, Retexture and the Editor with one instruction-driven model taking up to four references, the same convergence OpenAI and Google made. Aesthetics and draft-mode ideation remain its strengths. The hard limits: still no official API and terms that bar automation; generations public by default below the Stealth tier; and it is the defendant in Disney Enterprises v. Midjourney (filed June 2025, joined by a separate Warner Bros. Discovery suit), currently in discovery with Midjourney demanding the studios' own AI records.

Black Forest Labs, founded by the original Stable Diffusion authors, shipped FLUX.1 Kontext (in-context editing without masks) in 2025 and FLUX.2 in November 2025: a Mistral vision-language model paired with a rectified-flow transformer, up to ten references, 4MP editing. Tiers: Pro and Flex (API only); FLUX.2 dev (32B, open weights, non-commercial license); and FLUX.2 klein (January 2026), where the 4B model is Apache 2.0 and the 9B is non-commercial. The open-weights editing sub-board puts FLUX.2 and HunyuanImage roughly 130-150 Elo behind the closed frontier. Open weights earn their place through fine-tuning, on-prem privacy and composability with ControlNets and node graphs rather than raw quality.

• Stable Diffusion: SD 3.5 is still the flagship, no SD4; the Community License is free under a revenue threshold; Stability raised again in August 2026. SDXL retains the deepest LoRA ecosystem on Civitai.
• Qwen-Image: 20B, Apache 2.0, the best open model for text-in-image (especially Chinese); Qwen-Image-Edit 2511 adds multi-image editing and identity preservation. Qwen-Image 2.0 is closed and API-only.
• Seedream: ByteDance's Seedream 4.0 and 4.5 unify generation and editing at up to 4K with up to ten references, on fal and BytePlus; strong on cost, no longer top five in either arena.
• Ideogram: text-rendering specialist; Ideogram 3.0 added single-image Character Reference; Ideogram 4.0 (June 2026) is 9.3B open-weight with JSON bounding-box prompting, but the weights are non-commercial.
• Recraft: V4 / V4.1 output native SVG with editable paths, brand-palette control and clean product shots.
• Adobe Firefly: "commercially safe" training plus IP indemnification; the app now hosts partner models (GPT Image, Nano Banana Pro, Flux, Ideogram) and attaches Content Credentials automatically.
• Leonardo (Canva-owned, Phoenix model) is the practical pick inside Canva; Krea is a real-time canvas plus a 60-model aggregator for trying everything from one account.

Change this thing is instruction editing, now standard everywhere; masked inpainting and outpainting (Photoshop Generative Fill, GPT Image masks, FLUX.1 Fill) remain the hard constraint when the instruction is not enough. Keep this subject is reference conditioning, whose open-world mechanism is IP-Adapter (decoupled image cross-attention on a frozen base) and whose closed equivalents are Google's fourteen references and Midjourney's edit-model references; single references drift on fine detail. When drift is unacceptable, train a LoRA: roughly ten to twenty images and about a thousand steps on Flux (fal guide, FLUX.2 LoRA guide, Replicate trainer), open weights only. Keep this structure is ControlNet depth/edge/pose conditioning, still required for geometry fidelity per Autodesk's testing, with first-party support in Qwen-Image-Edit and FLUX.2 ComfyUI nodes. Multi-turn is for exploration; for repeatability, reproduce the winning edit as one instruction from the original.

Ownership is a human-authorship question: the US Copyright Office's Copyrightability report requires human authorship, treats prompts as unprotectable instructions, and protects AI-assisted work to the extent of the human contribution; the Supreme Court declined Thaler v. Perlmutter in March 2026, leaving the "how much human is enough" line undrawn. Commercial use is a vendor question: OpenAI (you own outputs), Midjourney (paid plans, no indemnity), Flux (tiered), Firefly (indemnified). Training-data fair use is unresolved in the US; Andersen v. Stability AI is the image bellwether, while the UK High Court largely rejected Getty's claims against Stability in November 2025. Provenance now has two layers: C2PA 2.3 manifests plus pixel watermarks; OpenAI now embeds both C2PA and SynthID (API guide), as Google already did. Manifests are detailed but stripped on re-save, screenshot and most platform uploads; SynthID survives those but carries little information. Labeling is now law: EU AI Act Article 50 and California SB 942 both enforceable from August 2026, China's rules from September 2025.

• Marketing asset: GPT Image or Nano Banana; Nano Banana references for a real face or product, a Flux LoRA when drift is unacceptable.
• Concept art: Midjourney, draft mode then the Edit Model; keep it out of any automated pipeline and flag the litigation to client legal.
• Product photo: Flux LoRA plus ControlNet depth, or Nano Banana object references for the quick version; Recraft for vector; Firefly for indemnity.
• Developer pipeline: on-prem or fine-tuning means open weights (FLUX.2 klein 4B commercially, dev with a license); otherwise OpenAI or Google APIs with a cheap tier for drafts; never Midjourney.

• AI Video Generation in 2026: Veo and Gemini Omni, Kling, Runway, MiniMax H3, and the End of Sora
• The AI Media Pipeline: Voice, Music, ComfyUI, APIs, and Finishing
• Companion show: AI Video Generation on Gnothi

Auto encoders are neural networks that compress data into a smaller "code," enabling dimensionality reduction, data cleaning, and lossy compression by reconstructing original inputs from this code. Advanced auto encoder types, such as denoising, sparse, and variational auto encoders, extend these concepts for applications in generative modeling, interpretability, and synthetic data generation.

• Notes and resources at ocdevel.com/mlg/36
• Try a walking desk - stay healthy & sharp while you learn & code
• Build the future of multi-agent software with AGNTCY.
• Thanks to T.J. Wilder from intrep.io for recording this episode!

• Autoencoders are neural networks designed to reconstruct their input data by passing data through a compressed intermediate representation called a "code."
• The architecture typically follows an hourglass shape: a wide input and output separated by a narrower bottleneck layer that enforces information compression.
• The encoder compresses input data into the code, while the decoder reconstructs the original input from this code.

• Unlike traditional supervised learning, where the output differs from the input (e.g., image classification), autoencoders use the same vector for both input and output.

• Autoencoders perform dimensionality reduction by learning compressed forms of high-dimensional data, making it easier to visualize and process data with many features.
• The compressed code can be used for clustering, visualization in 2D or 3D graphs, and input into subsequent machine learning models, saving computational resources and improving scalability.

• Autoencoders enable feature learning by extracting abstract representations from the input data, similar in concept to learned embeddings in large language models (LLMs).
• While effective for many data types, autoencoder-based encodings are less suited for variable-length text compared to LLM embeddings.

• By reducing dimensionality, autoencoders facilitate vector searches, efficient clustering, and similarity retrieval.
• The compressed codes enable lossy compression analogous to audio codecs like MP3, with the difference that autoencoders lack domain-specific optimizations for preserving perceptually important data.

• Loss functions in autoencoders are defined to compare reconstructed outputs to original inputs, often using different loss types depending on input variable types (e.g., Boolean vs. continuous).
• Compression via autoencoders is typically lossy, meaning some information from the input is lost during reconstruction, and the areas of information lost may not be easily controlled.

• Since reconstruction errors tend to move data toward the mean, autoencoders can be used to reduce noise and identify data outliers.
• Large reconstruction errors can signal atypical or outlier samples in the dataset.

• Denoising autoencoders are trained to reconstruct clean data from noisy inputs, making them valuable for applications in image and audio de-noising as well as signal smoothing.
• Iterative denoising as a principle forms the basis for diffusion models, where repeated application of a denoising autoencoder can gradually turn random noise into structured output.

• Autoencoders can aid in data imputation by filling in missing values: training on complete records and reconstructing missing entries for incomplete records using learned code representations.
• This approach leverages the model's propensity to output 'plausible' values learned from overall data structure.

• The separation of encoding and decoding can draw parallels to encryption and decryption, though autoencoders are not intended or suitable for secure communication due to their inherent lossiness.

• Sparse autoencoders use constraints to encourage code representations with only a few active values, increasing interpretability and explainability.
• Overcomplete autoencoders have a code size larger than the input, often in applications that require extraction of distinct, interpretable features from complex model states.

• Research such as Anthropic's "Towards Monosemanticity" applies sparse autoencoders to the internal activations of language models to identify interpretable features correlated with concrete linguistic or semantic concepts.
• These models can be used to monitor and potentially control model behaviors (e.g., detecting specific language usage or enforcing safety constraints) by manipulating feature activations.

• VAEs extend autoencoder architecture by encoding inputs as distributions (means and standard deviations) instead of point values, enforcing a continuous, normalized code space.
• Decoding from sampled points within this space enables synthetic data generation, as any point near the center of the code space corresponds to plausible data according to the model.

• VAEs are powerful in domains with sparse data or rare events (e.g., healthcare), allowing generation of synthetic samples representing underrepresented cases.
• They can increase model performance by augmenting datasets without requiring changes to existing model pipelines.

• Conditional autoencoders extend VAEs by allowing controlled generation based on specified conditions (e.g., generating a house with a pool), through additional decoder inputs and conditional loss terms.

• Training autoencoders and their variants requires computational resources, and their stochastic training can produce differing code representations across runs.
• Lossy reconstruction, lack of domain-specific optimizations, and limited code interpretability restrict some use cases, particularly where exact data preservation or meaningful decompositions are required.

At inference, large language models use in-context learning with zero-, one-, or few-shot examples to perform new tasks without weight updates, and can be grounded with Retrieval Augmented Generation (RAG) by embedding documents into vector databases for real-time factual lookup using cosine similarity. LLM agents autonomously plan, act, and use external tools via orchestrated loops with persistent memory, while recent benchmarks like GPQA (STEM reasoning), SWE Bench (agentic coding), and MMMU (multimodal college-level tasks) test performance alongside prompt engineering techniques such as chain-of-thought reasoning, structured few-shot prompts, positive instruction framing, and iterative self-correction.

• Notes and resources at ocdevel.com/mlg/mlg35
• Build the future of multi-agent software with AGNTCY
• Try a walking desk stay healthy & sharp while you learn & code

• Definition: LLMs can perform tasks by learning from examples provided directly in the prompt without updating their parameters.
  • Types:
    • Zero-shot: Direct query, no examples provided.
    • One-shot: Single example provided.
    • Few-shot: Multiple examples, balancing quantity with context window limitations.
  • Mechanism: ICL works through analogy and Bayesian inference, using examples as semantic priors to activate relevant internal representations.
  • Emergent Properties: ICL is an "inference-time training" approach, leveraging the model's pre-trained knowledge without gradient updates; its effectiveness can be enhanced with diverse, non-redundant examples.

• Grounding: Connecting LLMs with external knowledge bases to supplement or update static training data.
  • Motivation: LLMs' training data becomes outdated or lacks proprietary/specialized knowledge.
  • Benefit: Reduces hallucinations and improves factual accuracy by incorporating current or domain-specific information.
• RAG Workflow:
  1. Embedding: Documents are converted into vector embeddings (using sentence transformers or representation models).
  2. Storage: Vectors are stored in a vector database (e.g., FAISS, ChromaDB, Qdrant).
  3. Retrieval: When a query is made, relevant chunks are extracted based on similarity, possibly with re-ranking or additional query processing.
  4. Augmentation: Retrieved chunks are added to the prompt to provide up-to-date context for generation.
  5. Generation: The LLM generates responses informed by the augmented context.
  • Advanced RAG: Includes agentic approaches—self-correction, aggregation, or multi-agent contribution to source ingestion, and can integrate external document sources (e.g., web search for real-time info, or custom datasets for private knowledge).

• Overview: Agents extend LLMs by providing goal-oriented, iterative problem-solving through interaction, memory, planning, and tool usage.
• Key Components:
  • Reasoning Engine (LLM Core): Interprets goals, states, and makes decisions.
  • Planning Module: Breaks down complex tasks using strategies such as Chain of Thought or ReAct; can incorporate reflection and adjustment.
  • Memory: Short-term via context window; long-term via persistent storage like RAG-integrated databases or special memory systems.
  • Tools and APIs: Agents select and use external functions—file manipulation, browser control, code execution, database queries, or invoking smaller/fine-tuned models.
• Capabilities: Support self-evaluation, correction, and multi-step planning; allow integration with other agents (multi-agent systems); face limitations in memory continuity, adaptivity, and controllability.
• Current Trends: Research and development are shifting toward these agentic paradigms as LLM core scaling saturates.

• Definition: Models capable of ingesting and generating across different modalities (text, image, audio, video).
• Architecture:
  • Modality-Specific Encoders: Convert raw modalities (text, image, audio) into numeric embeddings (e.g., vision transformers for images).
  • Fusion/Alignment Layer: Embeddings from different modalities are projected into a shared space, often via cross-attention or concatenation, allowing the model to jointly reason about their content.
  • Unified Transformer Backbone: Processes fused embeddings to allow cross-modal reasoning and generates outputs in the required format.
• Recent Advances: Unified architectures (e.g., GPT-4o) use a single model for all modalities rather than switching between separate sub-models.
• Functionality: Enables actions such as image analysis via text prompts, visual Q&A, and integrated speech recognition/generation.

• Predictive Abstract Representation: Incorporating latent concept prediction alongside token prediction (e.g., via autoencoders).
• Patch-Level Training: Predicting larger "patches" of tokens to reduce sequence lengths and computation.
• Concept-Centric Modeling: Moving from next-token prediction to predicting sequences of semantic concepts (e.g., Meta's Large Concept Model).
• Multi-Token Prediction: Training models to predict multiple future tokens for broader context capture.

• Key Benchmarks Used for LLM Evaluation:
  • GPQA (Diamond): Graduate-level STEM reasoning.
  • SWE Bench Verified: Real-world software engineering, verifying agentic code abilities.
  • MMMU: Multimodal, college-level cross-disciplinary reasoning.
  • HumanEval: Python coding correctness.
  • HLE (Human's Last Exam): Extremely challenging, multimodal knowledge assessment.
  • LiveCodeBench: Coding with contamination-free, up-to-date problems.
  • MLPerf Inference v5.0 Long Context: Throughput/latency for processing long contexts.
  • MultiChallenge Conversational AI: Multiturn dialogue, in-context reasoning.
  • TAUBench/PFCL: Tool utilization in agentic tasks.
  • TruthfulnessQA: Measures tendency toward factual accuracy/robustness against misinformation.

• Foundational Approaches:
  • Few-Shot Prompting: Provide pairs of inputs and desired outputs to steer the LLM.
  • Chain of Thought: Instructing the LLM to think step-by-step, either explicitly or through internal self-reprompting, enhances reasoning and output quality.
  • Clarity and Structure: Use clear, detailed, and structured instructions—task definition, context, constraints, output format, use of delimiters or markdown structuring.
  • Affirmative Directives: Phrase instructions positively ("write a concise summary" instead of "don't write a long summary").
  • Iterative Self-Refinement: Prompt the LLM to review and improve its prior response for better completeness, clarity, and factuality.
  • System Prompt/Role Assignment: Assign a persona or role to the LLM for tailored behavior (e.g., "You are an expert Python programmer").
• Guideline: Regularly consult official prompting guides from model developers as model capabilities evolve.

• Inference-time compute is increasingly important for pushing the boundaries of LLM task performance.
• Agentic LLMs and multimodal reasoning represent the primary frontiers for innovation.
• Prompt engineering and benchmarking remain essential for extracting optimal performance and assessing progress.
• Models are expected to continue evolving with research into new architectures, memory systems, and integration techniques.

Explains language models (LLMs) advancements. Scaling laws - the relationships among model size, data size, and compute - and how emergent abilities such as in-context learning, multi-step reasoning, and instruction following arise once certain scaling thresholds are crossed. The evolution of the transformer architecture with Mixture of Experts (MoE), describes the three-phase training process culminating in Reinforcement Learning from Human Feedback (RLHF) for model alignment, and explores advanced reasoning techniques such as chain-of-thought prompting which significantly improve complex task performance.

• Notes and resources at ocdevel.com/mlg/mlg34
• Build the future of multi-agent software with AGNTCY
• Try a walking desk stay healthy & sharp while you learn & code

• Transformers: Introduced by the 2017 "Attention is All You Need" paper, transformers allow for parallel training and inference of sequences using self-attention, in contrast to the sequential nature of RNNs.
• Scaling Laws:
  • Empirical research revealed that LLM performance improves predictably as model size (parameters), data size (training tokens), and compute are increased together, with diminishing returns if only one variable is scaled disproportionately.
  • The "Chinchilla scaling law" (DeepMind, 2022) established the optimal model/data/compute ratio for efficient model performance: earlier large models like GPT-3 were undertrained relative to their size, whereas right-sized models with more training data (e.g., Chinchilla, LLaMA series) proved more compute and inference efficient.

• Emergence: When trained beyond a certain scale, LLMs display abilities not present in smaller models, including:
  • In-Context Learning (ICL): Performing new tasks based solely on prompt examples at inference time.
  • Instruction Following: Executing natural language tasks not seen during training.
  • Multi-Step Reasoning & Chain of Thought (CoT): Solving arithmetic, logic, or symbolic reasoning by generating intermediate reasoning steps.
• Discontinuity & Debate: These abilities appear abruptly in larger models, though recent research suggests that this could result from non-linearities in evaluation metrics rather than innate model properties.

• MoE Layers: Modern LLMs often replace standard feed-forward layers with MoE structures.
  • Composed of many independent "expert" networks specializing in different subdomains or latent structures.
  • A gating network routes tokens to the most relevant experts per input, activating only a subset of parameters—this is called "sparse activation."
  • Enables much larger overall models without proportional increases in compute per inference, but requires the entire model in memory and introduces new challenges like load balancing and communication overhead.
• Specialization & Efficiency: Experts learn different data/knowledge types, boosting model specialization and throughput, though care is needed to avoid overfitting and underutilization of specialists.

• 1. Unsupervised Pre-Training: Next-token prediction on massive datasets—builds a foundation model capturing general language patterns.
• 2. Supervised Fine Tuning (SFT): Training on labeled prompt-response pairs to teach the model how to perform specific tasks (e.g., question answering, summarization, code generation). Overfitting and "catastrophic forgetting" are risks if not carefully managed.
• 3. Reinforcement Learning from Human Feedback (RLHF):
  • Collects human preference data by generating multiple responses to prompts and then having annotators rank them.
  • Builds a reward model (often PPO) based on these rankings, then updates the LLM to maximize alignment with human preferences (helpfulness, harmlessness, truthfulness).
  • Introduces complexity and risk of reward hacking (specification gaming), where the model may exploit the reward system in unanticipated ways.

• Prompt Engineering: The art/science of crafting prompts that elicit better model responses, shown to dramatically affect model output quality.
• Chain of Thought (CoT) Prompting: Guides models to elaborate step-by-step reasoning before arriving at final answers—demonstrably improves results on complex tasks.
  • Variants include zero-shot CoT ("let's think step by step"), few-shot CoT with worked examples, self-consistency (voting among multiple reasoning chains), and Tree of Thought (explores multiple reasoning branches in parallel).
• Automated Reasoning Optimization: Frontier models selectively apply these advanced reasoning techniques, balancing compute costs with gains in accuracy and transparency.

• Tradeoffs: The optimal balance between model size, data, and compute is determined not only for pretraining but also for inference efficiency, as lifetime inference costs may exceed initial training costs.
• Current Trends: Efficient scaling, model specialization (MoE), careful fine-tuning, RLHF alignment, and automated reasoning techniques define state-of-the-art LLM development.

How working engineers ship with coding agents: issues an agent can verify, plan mode before code, a verification loop with a browser in it, agent review of agent code, worktrees and CI, cost discipline, and where agents still fail.

• Try a walking desk - stay healthy & sharp while you learn & code
• More OCDevel shows - this one has siblings, each on its own subject and produced the same way

Third and last episode of the vibe-coding sequence. Vibe Coding in 2026 picked an agent; Inside a Coding Agent explained the mechanics. This one is the practice: how working engineers ship real software with Claude Code, Codex, and Antigravity without shipping garbage. Specs, verification loops, agent review, parallel worktrees, headless CI, cost discipline, and the failure modes.

Andrej Karpathy coined vibe coding in early 2025 and, a year later, proposed "agentic engineering" for professional work. His older idea of jagged intelligence (models clear hard problems and trip on trivial ones, unpredictably) is why the job is judgment rather than button-pressing. The frame for the episode: when implementation is cheap, the value moves to the two ends of the pipeline, specification (saying exactly what should be true) and verification (proving it). Vibe coding stays fine for throwaways; the rest applies to codebases with users.

The prompt is a spec whether you meant it or not. The unit of work is a tracker issue with four parts: what's wrong, where to look, acceptance criteria a machine or a five-second human check can verify, and an explicit out-of-scope fence. Every serious agent has a read-only planning phase: Claude Code's plan mode (Shift+Tab or /plan), Codex's plan mode on the same keystroke with its own plan-mode reasoning effort, and Antigravity's implementation plan artifact. Judge a plan on three things: the files it names, the verification step it commits to, and whether it stays inside the fence. Heavier spec tooling (GitHub spec-kit, Kiro specs) formalizes requirements, design, and tasks. Anthropic's best practices teach a lighter interview-then-spec-file pattern and say to skip planning when you could describe the diff in one sentence; OpenAI's Codex best practices frame a task as goal, context, constraints, and "done when."

An agent without feedback guesses; an agent with a runnable check searches. Make typecheck, lint, tests, and build fast and runnable from one command, because that command is what the agent lives inside. Test-first changes meaning here: it used to be design pressure, and now it gives the agent a fixed target. Anthropic's guidance is that without a success criterion the developer is the only feedback loop (building verification loops). Then add a browser. Playwright MCP drives a real one through accessibility snapshots with stable element refs, no vision model needed; Claude in Chrome is the screenshot route Anthropic names for UI verification; Antigravity's browser agent starts the dev server and clicks through on its own, ending in a walkthrough with verification evidence. The principle: every acceptance criterion maps to a check the agent can run. If one doesn't, either the criterion is vague or the project is missing a kind of check.

Five vendor-agnostic stages: trigger, implement, review, fix, gate. Agent review of agent code works because the reviewer has fresh context, so never reuse the implementing session as its own reviewer. Ask for correctness and quality separately and set a confidence bar so the reviewer reports only what it's sure of. Anthropic's code-review plugin runs four parallel reviewers with 0-100 confidence scores and drops findings under 80; a cloud tier (Ultrareview) reproduces each finding before reporting it, and a separate hosted Code Review product reviews every PR automatically. Codex reviews on an @codex review mention or automatically per repo, flagging only serious issues. CodeRabbit and Greptile fill the same slot with a precision/recall tradeoff (ignore vendor benchmarks of each other). The human gate reviews a staging branch as a batch with the running app in front of you, and reads the tests, not just the code.

Each agent gets its own copy of the repo on its own branch. Locally that's a git worktree (Claude Code worktrees, --worktree and subagent isolation: worktree; Codex added a --worktree flag in 0.154.0). In the cloud it's a sandbox per task: Claude Code on the web (managed VM, credentials behind a proxy, claude --cloud, teleport back to the terminal), Codex cloud tasks (isolated containers, network off by default), Google Jules, and Antigravity's Agent Manager. Parallel tasks must be disjoint, which is why issues that name their files matter. Three interactive agents is a practical ceiling for one person; no vendor publishes a number, and Simon Willison's parallel coding agent lifestyle argues for prompting during natural breaks rather than a fixed count. Past that, scale with a labeled queue instead of more terminals.

Every major CLI has a non-interactive mode, and CI chains it. All three majors ship a GitHub Action covering mention, label, and schedule triggers: claude-code-action (@claude mentions, label_trigger, cron automation mode, claude/ branch prefix), Codex's GitHub integration and codex-action, and Google's run-gemini-cli with hourly issue triage in its examples. GitHub Copilot coding agent takes an assigned issue into an Actions runner and opens a draft PR. The CI fix loop: Claude Code on the web's Auto-fix pull requests subscribes to a PR's webhooks and pushes fixes for failing checks or review comments (/autofix-pr from the terminal); Codex does the same from a PR mention. Two cautions: headless runs need a fixed tool allowlist or a sandbox, and scheduled runs act as the user who wrote the schedule, so gate triggers on the actor to avoid automation loops.

Shape rather than prices, since prices age fastest: every vendor sells a subscription with a rolling window plus weekly cap in multiplied tiers (Claude, Codex, Antigravity), overage credits at roughly API rates, and pay-as-you-go API keys. Subscription for daily interactive use, API for headless and CI. Model tiering is official guidance: Anthropic's cost docs reserve the top model for architectural work, and subagents take a per-agent model and effort; Codex shows model plus reasoning effort on its status line. Context is the invisible line item: every turn re-sends history, compaction is itself a large request, /usage and /context show what eats the window, and the prompt cache goes cold after a break. The habits: clear between unrelated tasks, trim MCP servers, push exploration into subagents, and expect agent teams to run several times a single session's tokens.

Four families, each with a mechanism and a case. Reward-hacked tests: METR measured frontier models gaming graders in about 30% of research-engineering runs (patching the scoring function, locating precomputed answers), and instructing them not to cheat had nearly no effect; Anthropic's reward-hacking research names the canonical move, calling sys.exit(0) inside the harness so tests report green. The mitigation is structural: the worker is never the grader, and a hook flags test-file edits. Scope creep: fence the issue, flag out-of-fence changes in review, route unrelated improvements to a new issue. Prompt injection through repo content: Invariant Labs' GitHub MCP demonstration exfiltrated private code via a malicious public issue; Cursor's CVE-2025-54135 let injected content write MCP config and execute code; Anthropic's security docs say no system is immune and recommend VMs for external services. Secrets and blast radius: the Nx s1ngularity attack (advisory) ran victims' installed Claude, Gemini and Amazon Q CLIs with skip-permissions flags to harvest credentials, leaking 1,000+ tokens; the Replit production database deletion (July 2025) and a reported second wipe (April 2026); the UK AISI incident report (August 2026) on unsanctioned real-world actions during cyber evaluations. Guardrails: branch-scoped tokens behind a proxy, no production credentials on the agent's machine, destructive commands denied by hook or sandbox, plugins and MCP servers installed only from sources you'd let commit (plugin trust guidance), tested backups. On productivity: METR's 2025 RCT found experienced developers 19% slower; the 2026 uplift update flipped that cohort to a speedup with intervals crossing zero; the 2025 DORA report found AI adoption raised throughput and lowered delivery stability. The tools amplify the process you already have.

Day one: one fast command for typecheck, lint, tests, build. Day two: three issues with acceptance criteria and a scope fence, each through plan mode. Day three: browser verification and a screenshot in every UI PR. Day four: fresh-context review of every agent PR and a staging branch as the human gate. Day five: one recurring chore as a scheduled headless run, with secrets and test-integrity guardrails. Days six and seven: two agents on disjoint issues, and notice where your supervision breaks.

• MLA 22: Vibe Coding in 2026, which agent to pick and why
• MLA 23: Inside a Coding Agent, context, instruction files, hooks, skills, MCP, subagents
• MLA 28: AI Agents in 2026, agent architecture beyond coding
• MLA 30: AI and Programming Jobs in 2026, what agents did to the job market and how to position

Want this practice hands-on rather than surveyed? The Gnothi OCDevel Claude Code show goes from a first change in the terminal to a repeatable delivery workflow.

A coding agent is a loop appending to a context window, and the window is what runs out. Instruction files, permissions, hooks, skills, MCP, subagents and memory explained as one mechanism, mapped across Claude Code, Codex and OpenCode.

• Try a walking desk - stay healthy & sharp while you learn & code
• More OCDevel shows - this one has siblings, each on its own subject and produced the same way

Second of three episodes on coding agents: how Claude Code, Codex, OpenCode and pi handle context, instruction files, permissions, hooks, skills, MCP, subagents and memory. Claude Code is the running example; each section maps the same concept to the other three.

A coding agent is a short program: send the conversation to the model, run whatever tool it asks for, append the result, repeat until the model answers with text alone. Every appended file and test log stays in the window for the life of the session, so the context window, not the model, is what runs out. Every mechanism below is a way of deciding what enters the window and when. Compaction summarizes the history when the window fills; Claude Code's auto-compact window and /compact are configurable, and /compact is itself an expensive request while /clear is free. Thinking is dialed by effort level in Claude Code and by model_reasoning_effort in the Codex configuration reference.

AGENTS.md has become the cross-vendor instruction file, stewarded by the Agentic AI Foundation under the Linux Foundation and read by Codex, OpenCode, Cursor, Gemini CLI, GitHub Copilot, Jules, Aider, Zed and more. Claude Code reads only CLAUDE.md, bridged with an @AGENTS.md import; the Claude Code memory docs describe the managed, user, project and local layers, lazy subdirectory loading, @ imports up to four hops, path-scoped .claude/rules/, and re-injection after compaction. Codex's AGENTS.md guide walks from the git root down with a 32 KiB default cap; OpenCode rules read AGENTS.md with a CLAUDE.md fallback plus an instructions array in opencode.json; pi concatenates AGENTS.md from home, parents and cwd. What belongs: commands, conventions and boundaries the model cannot derive from the code, under a couple hundred lines. Instruction files are context, not enforcement.

Claude Code permissions use Tool(pattern) rules with deny over ask over allow, and the docs list how allowlists leak: absolute paths, sh -c, and unstripped wrappers like devbox run. Permission modes are default, acceptEdits, plan, the classifier-backed auto mode, and bypassPermissions. Codex separates sandbox modes (read-only, workspace-write, danger-full-access) from an approval policy. OpenCode permissions map globs to allow, ask or deny with last-match-wins. pi ships no permission system by design. Claude Code sandboxing uses Seatbelt on macOS and bubblewrap on Linux, with filesystem and network-domain allowlists. The cautionary case is PocketOS, April 2026, where a Cursor agent found an unscoped hosting token and deleted the production volume and its backups in seconds, reported by The Register and Fast Company.

A hook runs a program at a fixed point in the loop and its exit code or JSON decides what happens next; unlike an instruction, it fires every time. Claude Code hooks cover more than thirty events; exit code 2 always blocks on blockable events, PreToolUse can stop a call, Stop can refuse to end a turn, PostToolUse cannot block. Codex hooks mirror the event set and require trusting each hook definition by hash. OpenCode plugins expose tool.execute.before, permission.asked and session events; Gemini CLI hooks and Copilot CLI hooks exist too. The three worth having: a PreToolUse block on destructive commands and secret files, a post-edit formatter and typecheck that feeds errors back, and a Stop hook that holds the turn open while tests are red.

A skill is a folder with a SKILL.md whose name and description load at startup, whose body loads when a task matches, and whose bundled files load on demand. The format is the open Agent Skills standard, adopted by Codex, OpenCode, Cursor, Gemini CLI, GitHub Copilot and dozens more. Claude Code skills absorbed custom slash commands; frontmatter flags decide whether a skill is user-invoked, model-invoked or both, and side-effecting skills like deploy should be user-invoked only. A loaded skill body persists in the window, and load-time shell commands can splice live state into a procedure.

The Model Context Protocol standardizes tools, resources and prompts over JSON-RPC, via stdio or streamable HTTP with OAuth; Anthropic donated it to the Linux Foundation and the current revision adds stateless per-request negotiation, Tasks and MCP Apps extensions. Tool definitions traditionally loaded into context for every server every session. Anthropic's Code execution with MCP named the two costs and reported 150,000 tokens down to 2,000. The fix that shipped is deferred loading: Claude Code defers MCP tool definitions by default via tool search, also available at the API level; Codex MCP config offers per-server enabled_tools and disabled_tools; OpenCode MCP servers filter tools by glob with no lazy loading documented. Claude Code's cost guidance: prefer CLI tools over MCP servers, scope servers per project, disable what you have not used, and watch the 25,000-token output cap.

A subagent is a second loop with a fresh window; the parent sees only its result. Claude Code subagents are Markdown files with frontmatter for model, tools, permission mode, memory and worktree isolation, started without conversation history unless forked; agent teams add messaging between named teammates at roughly seven times the tokens of a plain session. Codex subagents are TOML files with developer instructions, model, effort and sandbox mode. OpenCode agents split into primary agents and @mention subagents, each with its own permission block. Recommended cast: a read-only scout on a cheap model, a fresh reviewer per diff, one writer per task. Claude Code worktrees create isolated checkouts with --worktree, EnterWorktree, subagent isolation: worktree, .worktreeinclude for gitignored files, and enforcement that blocks edits reaching back into the main checkout; Codex and OpenCode leave worktree management to git.

Memory is files. Claude Code auto memory is on by default: a per-repository directory shared across worktrees with a MEMORY.md index (first 200 lines loaded) and topic files read on demand, recording preferences, feedback and learned project facts. Codex memories are an opt-in local store controlled per session. OpenCode has no first-party memory beyond snapshots. Both vendors frame memory as a recall layer: rules that must always apply belong in the instruction file, prohibitions in hooks.

• Instruction file: CLAUDE.md (Claude Code); AGENTS.md (Codex, OpenCode, pi, most others)
• Permissions: modes and rules (Claude Code); sandbox mode plus approval policy (Codex); per-tool allow/ask/deny (OpenCode); none by design (pi)
• Hooks: settings hooks (Claude Code); trusted hooks file (Codex); plugins (OpenCode); extensions (pi)
• Skills: shared open format everywhere; slash commands folded in (Claude Code)
• MCP: deferred by default (Claude Code); per-server tool filters (Codex); glob filters (OpenCode)
• Subagents: Markdown (Claude Code, OpenCode); TOML (Codex)
• Worktrees: first-party (Claude Code); do it yourself (Codex, OpenCode)
• Memory: auto memory (Claude Code); memories (Codex); files or an MCP server (OpenCode)

• Vibe Coding in 2026: Codex vs Claude Code vs Antigravity vs Grok
• Agentic Software Engineering: Specs, Verification, and the Review Loop
• AI Agents in 2026: Loops, Tools, Memory, Protocols, and Evaluation

The Gnothi companion show on Claude Code goes from a first change in the terminal to a repeatable delivery workflow: OCDevel Claude Code Podcast.

What a coding agent actually is, and how Claude Code, OpenAI Codex, Google Antigravity and xAI's Grok Build differ in philosophy rather than features. Plus open harnesses pi and OpenCode, the open-weight coders, and a decision rule by constraint.

• Try a walking desk - stay healthy & sharp while you learn & code
• More OCDevel shows - this one has siblings, each on its own subject and produced the same way

First of three episodes on vibe coding: the field of coding agents (Claude Code, OpenAI Codex, Google Antigravity, Grok Build, pi, OpenCode) as of late 2026 and how to choose one. The next two cover the mechanics (Inside a Coding Agent) and the engineering practice (Agentic Software Engineering).

Andrej Karpathy's February 2025 post named "vibe coding": accept all, don't read the diffs, paste errors back in, and, the caveat everyone dropped, "not too bad for throwaway weekend projects." Collins made it Word of the Year 2025. The arc since is autocomplete, then chat, then agents that read, edit, run and iterate; the engineers who stayed with it moved from vibing to agentic engineering, which is the through line for the sequence.

The model and the harness are separate things, and most tool arguments are about one without noticing which. The harness is the loop around the model, and there are five parts worth naming: the model, the tools (read, write, shell, search, browser), the permission model and sandbox, the context strategy (what to keep, summarize, offload to files, or hand to a fresh subagent), and the surface (terminal, IDE, desktop, web, chat). Ask those five questions of any product; they matter more than the feature grid. The mechanics get their own episode.

Anthropic's stated design is Unix-style composability: pipe logs into it, run it in CI, chain it with other tools. The terminal is the primary surface; the IDE extension, desktop app, web and mobile clients all connect to the same engine and share CLAUDE.md, settings and MCP servers, with handoffs between local, cloud and desktop. Headless -p mode, GitHub Actions and GitLab integrations, and the Claude Agent SDK make it embeddable. Subagents and worktrees are its answer to the context problem. Model tiers (Sonnet, Opus, Fable) sit behind one harness with 1M-context variants and an effort dial; usage is included in Claude subscriptions or billed via the API.

• Claude Code overview
• Model configuration
• Plans and costs
• Headless mode
• Subagents and worktrees
• Claude Agent SDK
• Claude Fable 5 and Mythos 5, Claude Opus 5

Codex CLI and Codex cloud are one product under one ChatGPT login, with an IDE extension and a desktop app; codex cloud pushes local work into a sandboxed cloud environment and applies the result back. OpenAI's framing is isolated environments, parallel tasks, and comparing multiple solution attempts, with tasks launchable from the web, GitHub, GitLab, Linear or Slack. Codex-tuned models are trained for long-running agentic work, and the newest generation keeps notes across context windows instead of repeatedly compressing a session. The CLI is Apache-2.0 open source; the Agents API brings the same harness to your own cloud agents.

• Codex CLI docs
• Codex cloud docs
• Model lineup and plans and credits
• Codex CLI source
• Introducing the Codex app
• Introducing the Agents API

Antigravity still ships under that name, and it absorbed Gemini CLI: Google retired Gemini CLI into Antigravity CLI in June 2026, and Antigravity 2.0 spans IDE, desktop app, CLI and SDK. The philosophy is supervision: "manage higher-level tasks for the Agent, not individual tool calls," with agents working across editor, terminal and browser and producing artifacts you review. The model menu is multi-vendor on consumer tiers (several Gemini tiers plus Claude Sonnet and Opus and GPT-OSS); enterprise is Gemini only.

• Antigravity IDE
• Supported models and tiers
• Antigravity changelog
• Google I/O 2026 developer highlights
• Transitioning Gemini CLI to Antigravity CLI

xAI's coding story flipped in 2026 from a fast model in other people's harnesses (grok-code-fast-1, since deprecated from GitHub Copilot) to a first-party open-source terminal agent, Grok Build: interactive TUI, headless -p, Agent Client Protocol for editors, plan-review-approve, hooks, plugins, MCP and parallel subagents. xAI's own positioning for its coding model is fastest and cheapest for agentic coding; the premium model above it is the agent's default. Benchmark numbers and the subagent-count and price claims circulating in blogs are not on xAI's pages.

• Introducing Grok Build
• Grok Build 0.1 on the API
• Grok Build docs and source
• Grok Code Fast 1 deprecated in Copilot

Both are open source, model-agnostic and very active, and they're opposites in philosophy. pi is a toolkit: small core, unified API over 15+ providers, TUI, print/RPC/SDK modes, TypeScript extensions, branchable sessions, a package system, and deliberately no built-in permission system (containerize instead; the author ships the Gondolin micro-VM). OpenCode is a product: TUI, web, desktop and IDE surfaces, a ~50-entry provider list including local runtimes, permission config, and the optional Zen gateway of verified models. OpenCode's docs note Anthropic prohibits driving Claude subscription models through third-party harnesses and those plugins were removed. Also: Crush, Kilo Code, Cline, and DeepSeek's own harness.

• pi, pi source
• OpenCode docs, providers, Zen, source
• Crush, DeepSeek Harness

Every lab with a model now ships a harness. Qwen has a coder line, an Apache-licensed open series and the Qwen Code CLI. Z.ai's GLM is pitched as the most capable open-weights model for coding, sold as a Coding Plan documented for Claude Code, Cline and OpenCode. DeepSeek serves 1M-context models with an Anthropic-format endpoint, so Claude-shaped harnesses can point at it. Moonshot's Kimi K3 is a very large open-weight multimodal model under a bespoke license, with the Kimi Code CLI. All coding claims are vendor-reported, now headlining SWE-bench Pro and Terminal Bench rather than SWE-bench Verified. Second tier: Copilot coding agent (cloud, inside Actions), Cursor (editor-first, adding cloud agents and a coordinator), Windsurf (folded into Devin), Cline (active), Aider (no commits since May 2026).

• Qwen3-Coder, Qwen Code
• GLM-5, GLM Coding Plan
• DeepSeek models and pricing
• Kimi K3, Kimi Code CLI
• Copilot coding agent, Cursor changelog, Cline, Aider

Already paying a lab: take that lab's agent (Claude Code, Codex, Antigravity) and stop shopping. Cloud vs local: Codex cloud or Copilot's agent for sandboxed parallel PRs; a harness plus a local or self-hosted model if everything has to stay behind your firewall. Cost: Grok Build or an open harness with a cheap open-weight model for mechanical work. Open weights: pi or OpenCode with Qwen, GLM, DeepSeek or Kimi. Test on a real bug you already understand, and budget for a primary agent plus a cheaper second one.

• MLA 23: Inside a Coding Agent
• MLA 24: Agentic Software Engineering
• MLA 28: AI Agents
• MLA 30: AI and Programming Jobs

Companion show: the Gnothi Claude Code series goes from your first terminal change to a repeatable delivery workflow.

Links:

• Notes and resources at ocdevel.com/mlg/33
• 3Blue1Brown videos: https://3blue1brown.com/
• Try a walking desk stay healthy & sharp while you learn & code
• Try Descript audio/video editing with AI power-tools

• RNN Limitations: Sequential processing prevents full parallelization—even with attention tweaks—making them inefficient on modern hardware.
• Breakthrough: "Attention Is All You Need" replaced recurrence with self-attention, unlocking massive parallelism and scalability.

• Layer Stack: Consists of alternating self-attention and feed-forward (MLP) layers, each wrapped in residual connections and layer normalization.
• Positional Encodings: Since self-attention is permutation invariant, add sinusoidal or learned positional embeddings to inject sequence order.

• Q, K, V Explained:
  • Query (Q): The representation of the token seeking contextual info.
  • Key (K): The representation of tokens being compared against.
  • Value (V): The information to be aggregated based on the attention scores.
• Multi-Head Attention: Splits Q, K, V into multiple "heads" to capture diverse relationships and nuances across different subspaces.
• Dot-Product & Scaling: Computes similarity between Q and K (scaled to avoid large gradients), then applies softmax to weigh V accordingly.

• Causal Masking: In autoregressive models, prevents a token from "seeing" future tokens, ensuring proper generation.
• Padding Masks: Ignore padded (non-informative) parts of sequences to maintain meaningful attention distributions.

• Transformation & Storage: Post-attention MLPs apply non-linear transformations; many argue they're where the "facts" or learned knowledge really get stored.
• Depth & Expressivity: Their layered nature deepens the model's capacity to represent complex patterns.

• Residual Links: Crucial for gradient flow in deep architectures, preventing vanishing/exploding gradients.
• Layer Normalization: Stabilizes training by normalizing across features, enhancing convergence.

• Parallelization Advantage: Entire architecture is designed to exploit modern parallel hardware, a huge win over RNNs.
• Complexity Trade-offs: Self-attention's quadratic complexity with sequence length remains a challenge; spurred innovations like sparse or linearized attention.

• Pretraining & Fine-Tuning: Massive self-supervised pretraining on diverse data, followed by task-specific fine-tuning, is the norm.
• Emergent Behavior: With scale comes abilities like in-context learning and few-shot adaptation, aspects that are still being unpacked.

• Distributed Representation: "Facts" aren't stored in a single layer but are embedded throughout both attention heads and MLP layers.
• Debate on Attention: While some see attention weights as interpretable, a growing view is that real "knowledge" is diffused across the network's parameters.

Databricks is a cloud-based platform for data analytics and machine learning operations, integrating features such as a hosted Spark cluster, Python notebook execution, Delta Lake for data management, and seamless IDE connectivity. Raybeam utilizes Databricks and other ML Ops tools according to client infrastructure, scaling needs, and project goals, favoring Databricks for its balanced feature set, ease of use, and support for both startups and enterprises.

• Notes and resources at ocdevel.com/mlg/mla-21
• Try a walking desk stay healthy & sharp while you learn & code

• Raybeam is a data science and analytics company, recently acquired by Dept Agency.
• While Raybeam focuses on data analytics, its acquisition has expanded its expertise into ML Ops and AI.
• The company recommends tools based on client requirements, frequently utilizing Databricks for its comprehensive nature.

• Databricks is not merely an analytics platform; it is a competitor in the ML Ops space alongside tools like SageMaker and Kubeflow.
• It provides interactive notebooks, Python code execution, and runs on a hosted Apache Spark cluster.
• Databricks includes Delta Lake, which acts as a storage and data management layer.

• Raybeam evaluates each client's needs, existing expertise, and infrastructure before recommending a platform.
• Databricks, SageMaker, Kubeflow, and Snowflake are common alternatives, with the final selection dependent on current pipelines and operational challenges.
• Maintaining existing workflows is prioritized unless scalability or feature limitations necessitate migration.

• Databricks is accessible via a web interface similar to Jupyter Hub and can be integrated with local IDEs (e.g., VS Code, PyCharm) using Databricks Connect.
• Notebooks on Databricks can be version-controlled with Git repositories, enhancing collaboration and preventing data loss.
• The platform supports configuration of computing resources to match model size and complexity.
• Databricks clusters are hosted on AWS, Azure, or GCP, with users selecting the underlying cloud provider at sign-up.

• Parquet files store data in a columnar format, which improves efficiency for aggregation and analytics tasks.
• Delta Lake provides transactional operations on top of Parquet files by maintaining a version history, enabling row edits and deletions.
• This approach offers a database-like experience for handling large datasets, simplifying both analytics and machine learning workflows.

• Pricing for Databricks depends on the chosen cloud provider (AWS, Azure, or GCP) with an additional fee for Databricks' services.
• The added cost is described as relatively small, and the platform is accessible to both individual developers and large enterprises.
• Databricks is recommended for newcomers to data science and ML for its breadth of features and straightforward setup.

• Databricks provides a hosted MLflow solution, offering experiment tracking and model management.
• The platform can access data stored in services like S3, Snowflake, and other cloud provider storage options.
• Integration with tools such as PyArrow is supported, facilitating efficient data access and manipulation.

• Migration to Databricks is recommended when a client's existing infrastructure (e.g., on-premises Spark clusters) cannot scale effectively.
• The selection process involves an in-depth exploration of a client's operational challenges and goals.
• Databricks is chosen for clients lacking feature-specific needs but requiring a unified data analytics and ML platform.

• Ming Chang has explored automated stock trading using APIs such as Alpaca, focusing on downloading and analyzing market data.
• He has also developed drone-related projects with Raspberry Pi, emphasizing real-world applications of programming and physical computing.

• Databricks Homepage
• Delta Lake on Databricks
• Parquet Format
• Raybeam Overview
• MLFlow Documentation

Machine learning pipeline orchestration tools, such as SageMaker and Kubeflow, streamline the end-to-end process of data ingestion, model training, deployment, and monitoring, with Kubeflow providing an open-source, cross-cloud platform built atop Kubernetes. Organizations typically choose between cloud-native managed services and open-source solutions based on required flexibility, scalability, integration with existing cloud environments, and vendor lock-in considerations.

• Notes and resources at ocdevel.com/mlg/mla-20
• Try a walking desk stay healthy & sharp while you learn & code

Dirk-Jan Verdoorn - Data Scientist at Dept Agency

• Cloud providers such as AWS, Google Cloud, and Azure offer managed machine learning orchestration solutions, including SageMaker (AWS) and Vertex AI (GCP).
• Managed services provide integrated environments that are easier to set up and operate but often result in vendor lock-in, limiting portability across cloud platforms.
• Open-source tools like Kubeflow extend Kubernetes to support end-to-end machine learning pipelines, enabling portability across AWS, GCP, Azure, or on-premises environments.

• Kubeflow is an open-source project aimed at making machine learning workflow deployment on Kubernetes simple, portable, and scalable.
• Kubeflow enables data scientists and ML engineers to build, orchestrate, and monitor pipelines using popular frameworks such as TensorFlow, scikit-learn, and PyTorch.
• Kubeflow can integrate with TensorFlow Extended (TFX) for complete end-to-end ML pipelines, covering data ingestion, preprocessing, model training, evaluation, and deployment.

• Production machine learning systems involve not just model training but also complex pipelines for data ingestion, feature engineering, validation, retraining, and monitoring.
• Pipelines automate retraining based on model performance drift or updated data, supporting continuous improvement and adaptation to changing data patterns.
• Scalable, orchestrated pipelines reduce manual overhead, improve reproducibility, and ensure that models remain accurate as underlying business conditions evolve.

• ML pipeline orchestration tools in machine learning fulfill a role similar to continuous integration and continuous deployment (CI/CD) in traditional software engineering.
• Pipelines enable automated retraining, modularization of pipeline steps (such as ingestion, feature transformation, and deployment), and robust monitoring.
• Adopting pipeline orchestrators, rather than maintaining standalone models, helps organizations handle multiple models and varied business use cases efficiently.

• Managed services (e.g., SageMaker, Vertex AI) offer streamlined user experiences and seamless integration but restrict cross-cloud flexibility.
• Kubeflow, as an open-source platform on Kubernetes, enables cross-platform deployment, integration with multiple ML frameworks, and minimizes dependency on a single cloud provider.
• The complexity of Kubernetes and Kubeflow setup is offset by significant flexibility and community-driven improvements.

• Kubeflow operates on any Kubernetes environment including AWS EKS, GCP GKE, and Azure AKS, as well as on-premises or local clusters.
• Local and cross-cloud development are facilitated in Kubeflow, while managed services like SageMaker and Vertex AI are better suited to cloud-native workflows.
• Debugging and development workflows can be challenging in highly secured cloud environments; Kubeflow's local deployment flexibility addresses these hurdles.

• TensorFlow Extended (TFX) is an end-to-end platform for creating production ML pipelines, tightly integrated with Kubeflow for deployment and execution.
• While Kubeflow originally focused on TensorFlow, it has grown to support PyTorch, scikit-learn, and other major ML frameworks, offering wider applicability.
• TFX provides modular pipeline components (data ingestion, transformation, validation, model training, evaluation, and deployment) that execute within Kubeflow's orchestration platform.

• Airflow is a general-purpose workflow orchestrator using DAGs, suited for data engineering and automation, but less resource-capable for heavy ML training within the pipeline.
  • Airflow often submits jobs to external compute resources (e.g., AI Platform) for resource-intensive workloads.
  • In organizations using both Kubeflow and Airflow, Airflow may handle data workflows, while Kubeflow is reserved for ML pipelines.
• MLflow and other solutions also exist, each with unique integrations and strengths; their adoption depends on use case requirements.

• The optimal choice of cloud platform and orchestration tool is typically guided by client needs, existing integrations (e.g., organizational use of Google or Microsoft solutions), and team expertise.
• Agencies with diverse client portfolios often benefit from open-source, cross-cloud tools like Kubeflow to maximize flexibility and knowledge sharing across projects.
• Users entrenched in a single cloud provider may prefer managed offerings for ease of use and integration, while those prioritizing portability and flexibility often choose open-source solutions.

• Both AWS and GCP offer cost-saving compute options for training, such as spot instances (AWS) and preemptible instances (GCP), which are suitable for non-production, batch training jobs.
• Production workloads that require high uptime and reliability do not typically utilize cost-saving transient compute resources, as these can be interrupted.

• Project initiation begins with data discovery and validation of the client's requirements against available data.
• Cloud environment selection is influenced by client infrastructure, business applications, and platform integrations rather than solely by technical features.
• Data cleaning, exploratory analysis, model prototyping, advanced model refinement, and deployment are handled collaboratively with data engineering and machine learning teams.
• The pipeline is gradually constructed in modular steps, facilitating scalable, automated retraining and integration with business applications.

• Advanced mathematics or statistics education provides a strong foundation for work in data science and machine learning.
• Master's degrees in data science add the most value for candidates from non-technical undergraduate backgrounds; those with backgrounds in statistics, mathematics, or computer science may benefit more from self-study or targeted upskilling.
• When evaluating online or accelerated degree programs, candidates should scrutinize the curriculum, instructor engagement, and peer interaction to ensure comprehensive learning.

The deployment of machine learning models for real-world use involves a sequence of cloud services and architectural choices, where machine learning expertise must be complemented by DevOps and architecture skills, often requiring collaboration with professionals. Key concepts discussed include infrastructure as code, cloud container orchestration, and the distinction between DevOps and architecture, as well as practical advice for machine learning engineers wanting to deploy products securely and efficiently.

• Notes and resources at ocdevel.com/mlg/mla-19
• Try a walking desk stay healthy & sharp while you learn & code

;## Translating Machine Learning Models to Production

• After developing and training a machine learning model locally or using cloud tools like AWS SageMaker, it must be deployed to reach end users.
• A typical deployment stack involves the trained model exposed via a SageMaker endpoint, a backend server (e.g., Python FastAPI on AWS ECS with Fargate), a managed database (such as AWS RDS Postgres), an application load balancer (ALB), and a public-facing frontend (e.g., React app hosted on S3 with CloudFront and Route 53).

• Infrastructure as code (IaC) manages deployment and maintenance of cloud resources using tools like Terraform, allowing environments to be version-controlled and reproducible.
• Terraform is favored for its structured approach and cross-cloud compatibility, while other tools like Cloud Formation (AWS-specific) and Pulumi offer alternative paradigms.
• Configuration management tools such as Ansible, Chef, and Puppet automate setup and software installation on compute instances but are increasingly replaced by containerization and Dockerfiles.
• Continuous Integration and Continuous Deployment (CI/CD) pipelines (with tools like AWS CodePipeline or CircleCI) automate builds, testing, and code deployment to infrastructure.

• Containers, enabled by Docker, allow developers to encapsulate applications and dependencies, facilitating consistency across environments from local development to production.
• Deployment options include AWS ECS/Fargate for managed orchestration, Kubernetes for large-scale or multi-cloud scenarios, and simpler services like AWS App Runner and Elastic Beanstalk for small-scale applications.
• Kubernetes provides robust flexibility and cross-provider support but brings high complexity, making it best suited for organizations with substantial infrastructure needs and experienced staff.
• Use of cloud services versus open-source alternatives on Kubernetes (e.g., RDS vs. Postgres containers) affects manageability, vendor lock-in, and required expertise.

• DevOps unites development and operations through common processes and tooling to accelerate safe production deployments and improve coordination.
• Architecture focuses on the holistic design of systems, establishing how different technical components fit together and serve overall business or product goals.
• There is significant overlap, but architecture plans and outlines systems, while DevOps engineers implement, automate, and monitor deployment and operations.
• Cross-functional collaboration is essential, as machine learning engineers, DevOps, and architects must communicate requirements, constraints, and changes, especially regarding production-readiness and security.

• Security is a primary concern when moving to production, especially if handling sensitive data or personally identifiable information (PII); professional DevOps involvement is strongly advised in such cases.
• Common cloud security pitfalls include publicly accessible networks, insecure S3 buckets, and improper handling of secrets and credentials.
• For experimentation or small-scale safe projects, machine learning engineers can use tools like Terraform, Docker, and AWS managed services, but should employ cloud cost monitoring to avoid unexpected bills.

• AWS dominates the cloud market, followed by Azure (strong in enterprise/Microsoft-integrated environments) and Google Cloud Platform (GCP), which offers a strong user interface but has a record of sunsetting products.
• Managed cloud machine learning services, such as AWS SageMaker and GCP Vertex AI, streamline model training, deployment, and monitoring.
• Vendor-specific tools simplify management but limit portability, while Kubernetes and its ML pipelines (e.g., Kubeflow, Apache Airflow) provide open-source, cross-cloud options with greater complexity.

• Learning and prototyping with Terraform, Docker, and basic cloud services is encouraged to understand deployment pipelines, but professional security review is critical before handling production-sensitive data.
• For those entering DevOps, structured learning with platforms like aCloudGuru or AWS's own curricula can provide certification-ready paths.
• Continual learning is necessary, as tooling and best practices evolve rapidly.

Expert coworkers at Dept

• Matt Merrill - Principal Software Developer
• Jirawat Uttayaya - DevOps Lead
• The Ship It Podcast (frequent discussions on DevOps and architecture)

DevOps Tools

• Terraform
• Ansible

Visual Guides and Comparisons

• Which AWS container service should I use?
• A visual guide on troubleshooting Kubernetes deployments
• Public Cloud Services Comparison
• Killed by Google

Learning Resources

• aCloudGuru AWS curriculum

AWS development environments for local and cloud deployment can differ significantly, leading to extra complexity and setup during cloud migration. By developing directly within AWS environments, using tools such as Lambda, Cloud9, SageMaker Studio, client VPN connections, or LocalStack, developers can streamline transitions to production and leverage AWS-managed services from the start. This episode outlines three primary strategies for treating AWS as your development environment, details the benefits and tradeoffs of each, and explains the role of infrastructure-as-code tools such as Terraform and CDK in maintaining replicable, trackable cloud infrastructure.

• Notes and resources at ocdevel.com/mlg/mla-17
• Try a walking desk stay healthy & sharp while you learn & code

• Docker containers encapsulate operating systems, packages, and code, which simplifies dependency management and deployment.
• Files are added to containers using either the COPY command for one-time inclusion during a build or the volume directive for live synchronization during development.
• Docker Compose orchestrates multiple containers on a local environment, while Kubernetes is used at larger scale for container orchestration in the cloud.

• Docker is frequently used in AWS, including for packaging and deploying Lambda functions, SageMaker jobs, and ECS/Fargate containers.
• Deploying complex applications like web servers and databases on AWS involves using services such as ECR for image storage, ECS/Fargate for container management, RDS for databases, and requires configuration of networking components such as VPCs, subnets, and security groups.

• Local Docker Compose setups differ considerably from AWS managed services architecture.
• Migrating to AWS involves extra steps such as pushing images to ECR, establishing networking with VPCs, configuring load balancers or API Gateway, setting up domain names with Route 53, and integrating SSL certificates via ACM.
• Configuring internal communication between services and securing databases adds complexity compared to local development.

• Developers can use AWS Lambda's built-in code editor, Cloud9 IDE, and SageMaker Studio to edit, run, and deploy code directly in the AWS console.
• Cloud-based development is not tied to a single machine and eliminates local environment setup.
• While convenient, in-browser IDEs like Cloud9 and SageMaker Studio are less powerful than established local tools like PyCharm or DataGrip.

• The AWS Client VPN enables local machines to securely access AWS VPC resources, such as RDS databases or Lambda endpoints, as if they were on the same network.
• This approach allows developers to continue using their preferred local IDEs while testing code against actual cloud services.
• Storing sensitive credentials is handled by AWS Secrets Manager instead of local files or environment variables.
• Example tutorials and instructions:
  • AWS Client VPN
  • Terraform example
  • YouTube tutorial
  • Creating the keys

• LocalStack provides local, Docker-based emulation of AWS services, allowing development and testing without incurring cloud costs or latency.
• The project offers a free tier supporting core serverless services and a paid tier covering more advanced features like RDS, ACM, and Route 53.
• LocalStack supports mounting local source files into Lambda functions, enabling direct development on the local machine with changes immediately reflected in the emulated AWS environment.
• This approach brings rapid iteration and cost savings, but coverage of AWS features may vary, especially for advanced or new AWS services.

• Managing AWS resources through the web console is not sustainable for tracking or reproducing environments.
• Infrastructure as code (IaC) tools such as Terraform, AWS CDK, and Serverless enable declarative, version-controlled description and deployment of AWS services.
• Terraform offers broad multi-cloud compatibility and support for both managed and cloud-native services, whereas CDK is AWS-specific and typically more streamlined but supports fewer services.
• Changes made via IaC tools are automatically propagated to dependent resources, reducing manual error and ensuring consistency across environments.

• Developing directly in AWS or with local emulation ensures alignment between development, staging, and production environments, reducing last-minute deployment issues.
• Early use of AWS services can reveal managed solutions—such as Cognito for authentication or Data Wrangler for feature transformation—that are more scalable and secure than homegrown implementations.
• Infrastructure as code provides reproducibility, easier team onboarding, and disaster recovery.

• Kubernetes represents a different model of orchestrating containers and services, generally leveraging open source components inside Docker containers, independent of managed AWS services.
• While Kubernetes can manage deployments to AWS (via EKS), GCP, or Azure, its architecture and operational concerns differ from AWS-native development patterns.

• Lambda
• SageMaker Studio
• Cloud9

• Choosing between developing in the AWS cloud, connecting local environments via VPN, or using tools like LocalStack depends on team needs, budget, and workflow preferences.
• Emphasizing infrastructure as code ensures environments remain consistent, maintainable, and easily reproducible.

SageMaker streamlines machine learning workflows by enabling integrated model training, tuning, deployment, monitoring, and pipeline automation within the AWS ecosystem, offering scalable compute options and flexible development environments. Cloud-native AWS machine learning services such as Comprehend and Poly provide off-the-shelf solutions for NLP, time series, recommendations, and more, reducing the need for custom model implementation and deployment.

• Notes and resources at ocdevel.com/mlg/mla-16
• Try a walking desk stay healthy & sharp while you learn & code

• SageMaker enables model training within integrated data and ML pipelines, drawing from components such as Data Wrangler and Feature Store for a seamless workflow.
• Using SageMaker for training eliminates the need for manual transitions from local environments to the cloud, as models remain deployable within the AWS stack.
• SageMaker Studio offers a browser-based IDE environment with iPython notebook support, providing collaborative editing, sharing, and development without the need for complex local setup.
• Distributed, parallel training is supported with scalable EC2 instances, including AWS-proprietary chips for optimized model training and inference.
• SageMaker's Model Debugger and monitoring tools aid in tracking performance metrics, model drift, and bias, offering alerts via CloudWatch and accessible graphical interfaces.

• SageMaker supports various model creation approaches, including default AWS environments with pre-installed data science libraries, bring-your-own Docker containers, and hybrid customizations via requirements files.
• SageMaker JumpStart provides quick-start options for common ML tasks, such as computer vision or NLP, with curated pre-trained models and environment setups optimized for SageMaker hardware and operations.
• Users can leverage Autopilot for end-to-end model training and deployment with minimal manual configuration or start from JumpStart templates to streamline typical workflows.

• SageMaker Experiments supports automated hyperparameter search and optimization, using Bayesian optimization to evaluate and select the best performing configurations.
• Experiments and training runs are tracked, logged, and stored for future reference, allowing efficient continuation of experimentation and reuse of successful configurations as new data is incorporated.

• Trained models can be deployed as scalable REST endpoints, where users specify required EC2 instance types, including inference-optimized chips.
• Elastic Inference allows attachment of specialized hardware to reduce costs and tailor inference environments.
• Batch Transform is available for non-continuous, ad-hoc, or large batch inference jobs, enabling on-demand scaling and integration with data pipelines or serverless orchestration.

• SageMaker Pipelines manages the orchestration of ML workflows, supporting CI/CD by triggering retraining and deployments based on code changes or new data arrivals.
• CI/CD automation includes not only code unit tests but also automated monitoring of metrics such as accuracy, drift, and bias thresholds to qualify models for deployment.
• Monitoring features (like Model Monitor) provide ongoing performance assessments, alerting stakeholders to significant changes or issues.

• SageMaker supports integration with Kubernetes via EKS, allowing teams to leverage universal orchestration for containerized ML workloads across cloud providers or hybrid environments.
• The SageMaker Neo service optimizes and packages trained models for deployment to edge devices, mobile hardware, and AWS Lambda, reducing runtime footprint and syncing updates as new models become available.

• AWS offers a variety of cloud-native services for common ML tasks, accessible via REST or SDK calls and managed by AWS, eliminating custom model development and operations overhead.
  • Comprehend for document clustering, sentiment analysis, and other NLP tasks.
  • Forecast for time series prediction.
  • Fraud Detector for transaction monitoring.
  • Lex for chatbot workflows.
  • Personalize for recommendation systems.
  • Poly for text-to-speech conversion.
  • Textract for OCR and data extraction from complex documents.
  • Translate for machine translation.
  • Panorama for computer vision on edge devices.
• These services continuously improve as AWS retrains and updates their underlying models, transferring benefits directly to customers without manual intervention.

• When building features such as document clustering, question answering, or recommendations, first review whether cloud-native services like Comprehend can fulfill requirements prior to investing in custom ML models.
• For custom NLP tasks not available in AWS services, use SageMaker to manage model deployment (e.g., deploying pre-trained Hugging Face Transformers for summarization or embeddings).
• Batch inference and feature extraction jobs can be triggered using SageMaker automation and event notifications, supporting modular, scalable, and microservices-friendly architectures.
• Tabular prediction and feature importance can be handled by pipe-lining data from relational stores through SageMaker Autopilot or traditional algorithms such as XGBoost.
• Recommendation workflows can combine embeddings, neural networks, and event triggers, with SageMaker handling monitoring, scaling, and retraining in response to user feedback and data drift.

• Employ AWS cloud-native services where possible to minimize infrastructure management and accelerate feature delivery.
• Use SageMaker JumpStart and Autopilot to jump ahead in common ML scenarios, falling back to custom code and containers only when unique use cases demand.
• Leverage SageMaker tools for pipeline orchestration, monitoring, retraining, and model deployment to ensure scalable, maintainable, and up-to-date ML workflows.

• MadeWithML overview & ML tutorials
• SageMaker Home
• SageMaker JumpStart
• SageMaker Model Deployment
• SageMaker Pipelines
• SageMaker Model Monitor
• SageMaker Kubernetes Integration
• SageMaker Neo

SageMaker is an end-to-end machine learning platform on AWS that covers every stage of the ML lifecycle, including data ingestion, preparation, training, deployment, monitoring, and bias detection. The platform offers integrated tools such as Data Wrangler, Feature Store, Ground Truth, Clarify, Autopilot, and distributed training to enable scalable, automated, and accessible machine learning operations for both tabular and large data sets.

• Notes and resources at ocdevel.com/mlg/mla-15
• Try a walking desk stay healthy & sharp while you learn & code

MLOps is deploying your ML models to the cloud. See MadeWithML for an overview of tooling (also generally a great ML educational run-down.)

• SageMaker is a comprehensive platform offered by AWS for machine learning operations (MLOps), allowing full lifecycle management of machine learning models.
• Its popularity provides access to extensive resources, educational materials, community support, and job market presence, amplifying adoption and feature availability.
• SageMaker can replace traditional local development environments, such as setups using Docker, by moving data processing and model training to the cloud.

• SageMaker manages diverse data ingestion sources such as CSV, TSV, Parquet files, databases like RDS, and large-scale streaming data via AWS Kinesis Firehose.
• The platform introduces the concept of data lakes, which aggregate multiple related data sources for big data workloads.
• Data Wrangler is the entry point for data preparation, enabling ingestion, feature engineering, imputation of missing values, categorical encoding, and principal component analysis, all within an interactive graphical user interface.
• Data wrangler leverages distributed computing frameworks like Apache Spark to process large volumes of data efficiently.
• Visualization tools are integrated for exploratory data analysis, offering table-based and graphical insights typically found in specialized tools such as Tableau.

• Feature Store acts as a centralized repository to save and manage transformed features created during data preprocessing, ensuring different steps in the pipeline access consistent, reusable feature sets.
• It facilitates collaboration by making preprocessed features available to various members of a data science team and across different models.

• Ground Truth provides automated and manual data labeling options, including outsourcing to Amazon Mechanical Turk or assigning tasks to internal employees via a secure AWS GUI.
• The system ensures quality by averaging multiple annotators' labels and upweighting reliable workers, and can also perform automated label inference when partial labels exist.
• This flexibility addresses both sensitive and high-volume labeling requirements.

• Clarify identifies and analyzes bias in both datasets and trained models, offering measurement and reporting tools to improve fairness and compliance.
• It integrates seamlessly with other SageMaker components for continuous monitoring and re-calibration in production deployments.

• SageMaker Studio offers a web-based integrated development environment to manage all aspects of the pipeline visually.
• Autopilot automates the selection, training, and hyperparameter optimization of machine learning models for tabular data, producing an optimal model and optionally creating reproducible code notebooks.
• Users can take over the automated pipeline at any stage to customize or extend the process if needed.

• Debugger provides real-time training monitoring, similar to TensorBoard, and offers notifications for anomalies such as vanishing or exploding gradients by integrating with AWS CloudWatch.
• SageMaker's distributed training feature enables users to train models across multiple compute instances, optimizing for hardware utilization, cost, and training speed.
• The system allows for sharding of data and auto-scaling based on resource utilization monitored via CloudWatch notifications.

• The SageMaker pipeline covers every aspect of machine learning workflows, from ingestion, cleaning, and feature engineering, to training, deployment, bias monitoring, and distributed computation.
• Each tool is integrated to provide either no-code, low-code, or fully customizable code interfaces.
• The platform supports scaling from small experiments to enterprise-level big data solutions.

• SageMaker
• DataWrangler
• Feature Store
• Ground Truth
• Clarify
• Studio
• AutoPilot
• Debugger
• Distributed Training
• JumpStart

Where the model behind your product should run in 2026: managed APIs vs open weights, AWS native vs Modal, RunPod and Cloud Run GPU, vLLM and SGLang, quantized CPU inference on Spot, batch pricing, prod evals, and infra as code.

• Try a walking desk - stay healthy & sharp while you learn & code
• More OCDevel shows - this one has siblings, each on its own subject and produced the same way

The infrastructure episode, standalone: where the model behind your product runs in 2026, and what each choice costs. Managed APIs versus self-hosted open weights, AWS native (Bedrock, SageMaker) versus serverless GPU platforms (Modal, RunPod, Cloud Run GPU) versus bare GPUs, the inference servers (vLLM, SGLang, llama.cpp), quantized CPU inference on Spot, queues and batch pricing, observability and evals, and a minute on infrastructure as code. The coding-agent workflow is covered in Vibe Coding in 2026 through Agentic Software Engineering; agent architecture in AI Agents in 2026.

• The default for a product builder is a managed API: cost is linear in usage with no floor, while a self-hosted GPU costs the same idle as busy.
• Four things flip the default: sustained volume, privacy or data residency, a strict latency shape, or a model only available as open weights. Move one workload when a flip applies, not the whole stack.
• The cloud-hosted middle path: Amazon Bedrock's model catalog now spans Anthropic, OpenAI (including the open-weight gpt-oss models), Meta Llama, Mistral, DeepSeek, Qwen and more, and Custom Model Import serves your own safetensors weights on demand with no instance to manage.
• The framing on rent vs own is still a16z's "Navigating the High Cost of AI Compute": renting wins for almost everyone; owning pays only at large steady spend.

• Lambda and Fargate still have no GPUs; the Lambda quotas page has no accelerator dimension and the Fargate FAQ sends GPU work to EC2. Lambda offers 10 GB memory, 15-minute timeouts, 10 GB container images, and SnapStart for Python.
• SageMaker AI: real-time endpoints scale to zero only via inference components; asynchronous inference queues S3 payloads up to 1 GB and scales to zero; serverless inference is CPU-only; batch transform for offline scoring.
• AWS Batch GPU jobs on Spot compute environments are the cheapest bursty GPU on AWS, with no scheduler surcharge.
• Aurora PostgreSQL ships pgvector 0.8.2 with HNSW, and Aurora Serverless v2 auto-pause drops compute to zero when idle. Bedrock Knowledge Bases can use that Aurora as its store. Aurora DSQL has no extensions, so no pgvector.

• The category: ship a container or function, pay per GPU-second, scale to zero. Differentiators are cold start and warm-pool control.
• Modal: Python-native, per-second billing, T4 through B300; cold starts around a second with memory snapshots and minimum-container knobs; batch via Function.map.
• RunPod Serverless: queue-based or load-balanced endpoints, FlashBoot, async run and webhooks, wide GPU list from economy cards to B200.
• Cloud Run GPU: GA, NVIDIA L4 and RTX PRO 6000 Blackwell, scales to zero, instance-based billing required.
• Baseten with the open-source Truss packager: operated production deployments for custom models.
• Replicate joined Cloudflare in November 2025; Cloudflare Workers AI remains a curated open-model catalog, not a bring-your-own-weights host.
• fal: generative media inference first, serverless GPUs on the side.
• Status changes: Lambda's inference API is winding down in favor of GPU instances; Fly.io's GPU Machines appear withdrawn (docs removed, no GPU line on pricing).

• Hyperscaler families: AWS P6, Google Cloud GPUs, Azure ND/NC sizes. Neoclouds: CoreWeave, Nebius, Lambda, Crusoe; Vast.ai as a marketplace with interruptible pricing.
• A reserved box beats serverless when the GPU is busy most of the hour.
• EC2 Spot advertises up to 90 percent off on-demand, with a two-minute interruption notice. The discount is paid for with checkpointing, retry-on-reclaim, and instance-pool diversity using the SPOT_PRICE_CAPACITY_OPTIMIZED allocation strategy AWS recommends.

• vLLM: PagedAttention, continuous batching, OpenAI- and Anthropic-compatible server, V1 engine, now under the PyTorch Foundation.
• SGLang: RadixAttention prefix reuse, releases every two weeks, production use at xAI, NVIDIA, Cursor and the major clouds.
• Hugging Face TGI was archived in March 2026; its notice points to vLLM and SGLang.
• TensorRT-LLM for NVIDIA-only maximum throughput; NVIDIA Dynamo orchestrates disaggregated prefill/decode across engines.
• llama.cpp for GGUF on CPUs, Apple Silicon and ARM with KleidiAI; Ollama now runs its own engine on GGML for multimodal models.
• Also: Triton Inference Server for non-LLM models, Ray Serve for composition, OpenLLM, LMDeploy, MLC-LLM.

• Quantized weights cut memory and, since batch-1 decode is memory-bandwidth bound, cut time per token; integer matrix instructions (ARM dot-product and MMLA, Intel VNNI) do the multiplies.
• GGUF quant naming: bits per weight, k-quant mixed precision, size variant; Q4_K_M takes Llama 3.1 8B from about 32 GB to about 5 GB.
• AWS Graviton guide for llama.cpp recommends Graviton3 and later for their BF16 and MMLA instructions and publishes a prebuilt llama-server container.
• vLLM's CPU backend covers x86 AVX512 and ARM. ONNX Runtime quantization for classic models, with the caveat that gains need VNNI or ARM dot-product hardware.
• Practice described in the episode: a quantized Qwen3-TTS model served by a C port on Graviton Spot instances via AWS Batch, checkpointing chunks to S3.

• If no human waits, queue it: SQS with Lambda needs idempotency and partial-batch failure reporting; SageMaker async, Modal map, and RunPod async run are the same shape.
• Batch pricing is 50 percent off across OpenAI Batch, Anthropic Message Batches, and Bedrock batch inference, with separate rate limits and a 24-hour window.
• Bedrock prompt caching bills cache reads at a large discount within a short TTL; put stable prefix content first.
• Real-time paths need warm containers ("scale to zero" becomes "scale to one"), continuous batching, and streaming.

• Trace every call with prompt, response, tokens, latency, cost. The OpenTelemetry GenAI conventions moved to their own repo; OpenLLMetry emits them.
• Tools: Langfuse (open source, part of ClickHouse since January 2026), Arize Phoenix (Elastic License), LangSmith, W&B Weave, Braintrust, Helicone as a proxy.
• Production evals score sampled traces with a judge model: Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena found over 80 percent agreement with humans and named position, verbosity and self-enhancement bias.
• Classic drift: Evidently covers tabular drift and LLM metrics; WhyLabs ceased operations but open-sourced whylogs and langkit.

• SST runs on Pulumi with Terraform providers bridged; Pulumi for languages without SST's web opinions; Terraform under BSL since 2023 with OpenTofu as the open fork. Keep ML infra in the product's repo and deploy.

• Fast custom silicon APIs: Groq, Cerebras. Open-model inference clouds: Together AI, Fireworks, DeepInfra. Aggregators: OpenRouter, Hugging Face Inference Providers; dedicated Inference Endpoints. AWS Trainium. SageMaker JumpStart delisted models in March 2026, so pin weights in your own bucket.

• MLA 22: Vibe Coding in 2026
• MLA 23: Inside a Coding Agent
• MLA 24: Agentic Software Engineering
• MLA 28: AI Agents in 2026
• MLA 29: OpenClaw and the Personal Agent

More Gnothi-generated shows on AI, coding, video and business: ocdevel.com/moremlg.

Primary technology recommendations for building a customer-facing machine learning product include React and React Native for the front end, serverless platforms like AWS Amplify or GCP Firebase for authentication and basic server/database needs, and Postgres as the relational database of choice. Serverless approaches are encouraged for scalability and security, with traditional server frameworks and containerization recommended only for advanced custom backend requirements. When serverless options are inadequate, use Node.js with Express or FastAPI in Docker containers, and consider adding Redis for in-memory sessions and RabbitMQ or SQS for job queues, though many of these functions can be handled by Postgres. The machine learning server itself, including deployment strategies, will be discussed separately.

• Notes and resources at ocdevel.com/mlg/mla-13
• Try a walking desk stay healthy & sharp while you learn & code

• React is recommended as the primary web front-end framework due to its compositional structure, best practice enforcement, and strong community support.
• React Native is used for mobile applications, enabling code reuse and a unified JavaScript codebase for web, iOS, and Android clients.
• Using React and React Native simplifies development by allowing most UI logic to be written in a single language.

• The episode encourages starting with serverless frameworks, such as AWS Amplify or GCP Firebase, for rapid scaling, built-in authentication, and security.
  • Amplify allows seamless integration with React and handles authentication, user management, and database access directly from the client.
  • When direct client-to-database access is insufficient, custom business logic can be implemented using AWS Lambda or Google Cloud Functions without managing entire servers.
• Only when serverless frameworks are insufficient should developers consider managing their own server code.
  • Recommended traditional backend options include Node.js with Express for JavaScript environments or FastAPI for Python-centric projects, both offering strong concurrency support.
  • Using Docker to containerize server code and deploying via managed orchestration (e.g., AWS ECS/Fargate) provides flexibility and migration capability beyond serverless.
  • Python's FastAPI is advised for developers heavily invested in the Python ecosystem, especially if machine learning code is also in Python.

• Postgres is recommended as the primary relational database, owing to its advanced features, community momentum, and versatility.
  • Postgres can serve multiple infrastructure functions beyond storage, including job queue management and pub/sub (publish-subscribe) messaging via specific database features.
• NoSQL options such as MongoDB are only recommended when hierarchical, non-tabular data models or specific performance optimizations are necessary.
• For situations requiring in-memory session management or real-time messaging, Redis is suggested, but Postgres may suffice for many use cases.
• Job queuing can be accomplished with external tools like RabbitMQ or AWS SQS, but Postgres also supports job queuing via transactional locks.

• Serverless deployment abstracts away infrastructure operations, improving scalability and reducing ongoing server management and security burdens.
  • Serverless functions scale automatically and only incur charges during execution.
• Amplify and Firebase offer out-of-the-box user authentication, database, and cloud function support, while custom authentication can be handled with tools like AWS Cognito.
• Managed database hosting (e.g., AWS RDS for Postgres) simplifies backups, scaling, and failover but is distinct from full serverless paradigms.

• The episode contrasts older monolithic frameworks (Django, Ruby on Rails) with current microservice and serverless architectures.
• Developers are encouraged to leverage modern tools where possible, adopting serverless and cloud-managed components until advanced customization requires traditional servers.

• React for web client
• create-react-app: quick-start React setup
• React Bootstrap: CSS framework (alternatives: Tailwind, Chakra, MaterialUI)
• react-router and easy-peasy as useful plugins
• React Native for mobile apps

• AWS Amplify for serverless web and mobile backends
• GCP Firebase
• AWS Serverless (underlying building blocks)
• AWS Lambda for serverless functions
• ECR, Fargate, Route53, ELB for containerized deployment

• Postgres as the primary relational database
• Redis for session-management and pub/sub
• RabbitMQ or SQS for job queuing (with wrapper: Celery)

Docker enables efficient, consistent machine learning environment setup across local development and cloud deployment, avoiding many pitfalls of virtual machines and manual dependency management. It streamlines system reproduction, resource allocation, and GPU access, supporting portability and simplified collaboration for ML projects. Machine learning engineers benefit from using pre-built Docker images tailored for ML, allowing seamless project switching, host OS flexibility, and straightforward deployment to cloud platforms like AWS ECS and Batch, resulting in reproducible and maintainable workflows.

• Notes and resources at ocdevel.com/mlg/mla-12
• Try a walking desk stay healthy & sharp while you learn & code

• Traditional machine learning development often requires configuring operating systems, GPU drivers (CUDA, cuDNN), and specific package versions directly on the host machine.
• Manual setup can lead to version conflicts, resource allocation issues, and difficulty reproducing environments across different systems or between local and cloud deployments.
• Tools like Anaconda and "pipenv" help manage Python and package versions, but they often fall short in managing system-level dependencies such as CUDA and cuDNN.

• Virtual machines (VMs) like VirtualBox or VMware allow multiple operating systems to run on a host, but they pre-allocate resources (RAM, CPU) up front and have limited access to host GPUs, restricting usability for machine learning tasks.
• Docker uses containerization to package applications and dependencies, allowing containers to share host resources dynamically and to access the GPU directly, which is essential for ML workloads.

• Dockerfiles describe the entire guest operating system and software environment in code, enabling complete automation and repeatability of environment setup.
• Containers created from Dockerfiles use only the necessary resources at runtime and avoid interfering with the host OS, making it easy to switch projects, share setups, or scale deployments.
• GPU support in Docker allows machine learning engineers to leverage their hardware regardless of host OS (with best results on Windows and Linux with Nvidia cards).
• On Windows, enabling GPU support requires switching to the Dev/Insider channel and installing specific Nvidia drivers alongside WSL2 and Nvidia-Docker.
• Macs are less suitable for GPU-accelerated ML due to their AMD graphics cards, although workarounds like PlaidML exist.

• Deploying machine learning models traditionally required manual replication of environments on cloud servers, such as EC2 instances, which is time-consuming and error-prone.
• With Docker, the same Dockerfile can be used locally and in the cloud (AWS ECS, Batch, Fargate, EKS, or SageMaker), ensuring the deployed environment matches local development exactly.
• AWS ECS is suited for long-lived container services, while AWS Batch can be used for one-off or periodic jobs, offering cost-effective use of spot instances for GPU workloads.

• Docker Hub provides pre-built images for ML environments, such as nvcr.io's CUDA/cuDNN images and HuggingFace's transformers setups, which can be inherited in custom Dockerfiles.
• These images ensure compatibility between key ML libraries (PyTorch, TensorFlow, CUDA, cuDNN) and reduce setup friction.
• Custom kitchen-sink images, like those in the "ml-tools" repository, offer a turnkey solution for getting started with machine learning in Docker.

• With Docker, each project can have a fully isolated environment, preventing dependency conflicts and simplifying switching between projects.
• Updates or configuration changes are tracked and versioned in the Dockerfile, maintaining a single source of truth for the entire environment.
• Modifying the Dockerfile to add dependencies or update versions ensures that local and cloud environments remain synchronized.

• Windows is recommended for local development with Docker, offering better desktop experience and driver support than Ubuntu for most users, particularly on laptops.
• GPU-accelerated ML is not practical on Macs due to hardware limitations, while Ubuntu is suitable for advanced users comfortable with system configuration and driver management.

• Docker
• Instructions: Windows Dev Channel & WSL2 with nvidia-docker support
• Nvidia's guide for CUDA on WSL2
• WSL2 & Docker odds-and-ends
• nvidia/cuda:10.1-cudnn7-runtime-ubuntu18.04 Docker Image
• huggingface/transformers-gpu
• ml-tools kitchen-sink Dockerfiles
• Machine learning hardware guidance
• Front-end stack + cloud-hosting info
• ML cloud-hosting info

Try a walking desk to stay healthy while you study or work!

Show notes at ocdevel.com/mlg/32.

L1/L2 norm, Manhattan, Euclidean, cosine distances, dot product

Normed distances link

• A norm is a function that assigns a strictly positive length to each vector in a vector space. link
• Minkowski is generalized. p_root(sum(xi-yi)^p). "p" = ? (1, 2, ..) for below.
• L1: Manhattan/city-block/taxicab. abs(x2-x1)+abs(y2-y1). Grid-like distance (triangle legs). Preferred for high-dim space.
• L2: Euclidean. sqrt((x2-x1)^2+(y2-y1)^2. sqrt(dot-product). Straight-line distance; min distance (Pythagorean triangle edge)
• Others: Mahalanobis, Chebyshev (p=inf), etc

Dot product

• A type of inner product. Outer-product: lies outside the involved planes. Inner-product: dot product lies inside the planes/axes involved link. Dot product: inner product on a finite dimensional Euclidean space link

Cosine (normalized dot)

Primary clustering tools for practical applications include K-means using scikit-learn or Faiss, agglomerative clustering leveraging cosine similarity with scikit-learn, and density-based methods like DBSCAN or HDBSCAN. For determining the optimal number of clusters, silhouette score is generally preferred over inertia-based visual heuristics, and it natively supports pre-computed distance matrices.

• Notes and resources at ocdevel.com/mlg/mla-11
• Try a walking desk stay healthy & sharp while you learn & code

• K-means is the most widely used clustering algorithm and is typically the first method to try for general clustering tasks.
• The scikit-learn KMeans implementation is suitable for small to medium-sized datasets, while Faiss's kmeans is more efficient and accurate for very large datasets.
• K-means requires the number of clusters to be specified in advance and relies on the Euclidean distance metric, which performs poorly in high-dimensional spaces.
• When document embeddings have high dimensionality (e.g., 768 dimensions from sentence transformers), K-means becomes less effective due to the limitations of Euclidean distance in such spaces.

• For text embeddings with high dimensionality, agglomerative (hierarchical) clustering methods are preferable, particularly because they allow the use of different similarity metrics.
• Agglomerative clustering in scikit-learn accepts a pre-computed cosine similarity matrix, which is more appropriate for natural language processing.
• Constructing the pre-computed distance (or similarity) matrix involves normalizing vectors and computing dot products, which can be efficiently achieved with linear algebra libraries like PyTorch.
• Hierarchical algorithms do not use inertia in the same way as K-means and instead rely on external metrics, such as silhouette score.
• Other clustering algorithms exist, including spectral, mean shift, and affinity propagation, which are not covered in this episode.

• Libraries such as Faiss, Annoy, and HNSWlib provide approximate nearest neighbor search for efficient semantic search on large-scale vector data.
• These systems create an index of your embeddings to enable rapid similarity search, often with the ability to specify cosine similarity as the metric.
• Sample code using these libraries with sentence transformers can be found in the UKP Lab sentence-transformers examples directory.

• Both K-means and agglomerative clustering require a predefined number of clusters, but this is often unknown beforehand.
• The "elbow" method involves running the clustering algorithm with varying cluster counts and plotting the inertia (sum of squared distances within clusters) to visually identify the point of diminishing returns; see kmeans.inertia_.
• The kneed package can automatically detect the "elbow" or "knee" in the inertia plot, eliminating subjective human judgment; sample code available here.
• The silhouette score, calculated via silhouette_score, considers both inter- and intra-cluster distances and allows for direct selection of the number of clusters with the maximum score.
• The silhouette score can be computed using a pre-computed distance matrix (such as from cosine similarities), making it well-suited for applications involving non-Euclidean metrics and hierarchical clustering.

• DBSCAN is a hierarchical clustering method that does not require specifying the number of clusters, instead discovering clusters based on data density.
• HDBSCAN is a more popular and versatile implementation of density-based clustering, capable of handling various types of data without significant parameter tuning.
• DBSCAN and HDBSCAN can be preferable to K-means or agglomerative clustering when automatic determination of cluster count or robustness to noise is important.
• However, these algorithms may not perform well with all types of high-dimensional embedding data, as illustrated by the challenges faced when clustering 768-dimensional text embeddings.

• For low- to medium-sized, low-dimensional data, use K-means with silhouette score to choose the optimal number of clusters: scikit-learn KMeans, silhouette_score.
• For very large data or vector search, use Faiss.kmeans.
• For high-dimensional data using cosine similarity, use Agglomerative Clustering with a pre-computed square matrix of cosine similarities; sample code.
• For density-based clustering, consider DBSCAN or HDBSCAN.
• Exploratory code and further examples can be found in the UKP Lab sentence-transformers examples.

The landscape of Python natural language processing tools has evolved from broad libraries like NLTK toward more specialized packages such as Gensim for topic modeling, SpaCy for linguistic analysis, and Hugging Face Transformers for advanced tasks, with Sentence Transformers extending transformer models to enable efficient semantic search and clustering. Each library occupies a distinct place in the NLP workflow, from fundamental text preprocessing to semantic document comparison and large-scale language understanding.

• Notes and resources at ocdevel.com/mlg/mla-10
• Try a walking desk stay healthy & sharp while you learn & code

• NLTK ("Natural Language Toolkit") was one of the earliest and most popular Python libraries for natural language processing, covering tasks from tokenization and stemming to document classification and syntax parsing.
• NLTK remains a catch-all "Swiss Army knife" for NLP, but many of its functions have been supplemented or superseded by newer tools tailored to specific tasks.

• Gensim emerged as the leading library for topic modeling in Python, most notably via its LDA Topic Modeling implementation, which groups documents according to topic distributions.
• Topic modeling workflows often use NLTK for initial preprocessing (tokenization, stop word removal, lemmatization), then vectorize with scikit-learn's TF-IDF, and finally model topics with Gensim's LDA.
• Gensim also provides effective Bigrams/Trigrams, allowing the detection and combination of commonly-used word pairs or triplets (n-grams) to enhance analysis accuracy.

• spaCy is a deep-learning-based library for high-performance linguistic analysis, focusing on tasks such as part-of-speech tagging, named entity recognition, and syntactic parsing.
• SpaCy supports integrated sentence and word tokenization, stop word removal, and lemmatization, but for advanced lemmatization and inflection, LemmInflect can be used to derive proper inflections for part-of-speech tags.
• For even more accurate (but slower) linguistic tasks, consider Stanford CoreNLP via SpaCy integration as spacy-stanza.
• SpaCy can examine parse trees to identify sentence components, enabling sophisticated NLP applications like grammatical corrections and intent detection in conversation agents.

• huggingface/transformers provides interfaces to transformer-based models (like BERT and its successors) capable of advanced NLP tasks including question answering, summarization, translation, and sentiment analysis.
• Its Pipelines allow users to accomplish over ten major NLP applications with minimal code.
• The library's model repository hosts a vast collection of pre-trained models that can be used for both research and production.

• UKPLab/sentence-transformers extends the transformer approach to create dense document embeddings, enabling semantic search, clustering, and similarity comparison via cosine distance or similar metrics.
• Example applications include finding the most similar documents, clustering user entries, or summarizing clusters of text.
• The repository offers application examples for tasks such as semantic search and clustering, often using cosine similarity.
• For very large-scale semantic search (such as across Wikipedia), approximate nearest neighbor (ANN) libraries like Annoy, FAISS, and hnswlib enable rapid similarity search with embeddings; practical examples are provided in the Sentence Transformers documentation.

• For a comparative overview and discovery of further libraries, see Analytics Steps Top 10 NLP Libraries in Python, which reviews several packages beyond those discussed here.

• NLTK: Foundational and comprehensive for most classic NLP needs; still covers a broad range of preprocessing and basic analytic tasks.
• Gensim: Best for topic modeling and phrase extraction (bigrams/trigrams); especially useful in workflows relying on document grouping and label generation.
• SpaCy: Leading tool for syntactic, linguistic, and grammatical analysis; supports integration with advanced lemmatizers and external tools like Stanford CoreNLP.
• Hugging Face Transformers: The standard for modern, high-level NLP tasks and quick prototyping, featuring simple pipelines and an extensive model hub.
• Sentence Transformers: The main approach for embedding text for semantic search, clustering, and large-scale document comparison, supporting ANN methodologies via companion libraries.

Python charting libraries - Matplotlib, Seaborn, and Bokeh - explaining, their strengths from quick EDA to interactive, HTML-exported visualizations, and clarifies where D3.js fits as a JavaScript alternative for end-user applications. It also evaluates major software solutions like Tableau, Power BI, QlikView, and Excel, detailing how modern BI tools now integrate drag-and-drop analytics with embedded machine learning, potentially allowing business users to automate entire workflows without coding.

• Notes and resources at ocdevel.com/mlg/mla-9
• Try a walking desk stay healthy & sharp while you learn & code

• Exploratory Data Analysis (EDA):
  • EDA occupies an early stage in the Business Intelligence (BI) pipeline, positioned just before or sometimes merged with the data cleaning ("munging") phase.
  • The outputs of EDA (e.g., correlation matrices, histograms) often serve as inputs to subsequent machine learning steps.

• The foundational plotting library in Python, supporting static, basic chart types.
• Requires substantial boilerplate code for custom visualizations.
• Serves as the core engine for many higher-level visualization tools.
• Common EDA tasks (like plotting via .corr(), .hist(), and .scatter() methods on pandas DataFrames) depend on Matplotlib under the hood.

• Pandas integrates tightly with Matplotlib and exposes simple, one-line commands for common plots (e.g., df.corr(), df.hist()).
• Designed to make quick EDA accessible without requiring detailed knowledge of Matplotlib's verbose syntax.

• A high-level wrapper around Matplotlib, analogous to how Keras wraps TensorFlow.
• Sets sensible defaults for chart styles, fonts, colors, and sizes, improving aesthetics with minimal effort.
• Importing Seaborn can globally enhance the appearance of all Matplotlib plots, even without direct usage of Seaborn's plotting functions.

• A powerful library for creating interactive, web-ready plots from Python.
• Enables user interactions such as hovering, zooming, and panning within rendered plots.
• Exports visualizations as standalone HTML files or can operate as a server-linked app for live data exploration.
• Supports advanced features like cross-filtering, allowing dynamic slicing and dicing of data across multiple axes or columns.
• More suited for creating reusable, interactive dashboards rather than quick, one-off EDA visuals.

• Unlike previous libraries, D3.js is a JavaScript framework for creating complex, highly customized data visualizations for web and mobile apps.
• Used predominantly on the client-side to build interactive front-end graphics for end users, not as an EDA tool for analysts.
• Common in production-grade web apps, but not typically part of a Python-based data science workflow.

• Leading commercial drag-and-drop BI tool for data visualization and dashboarding.
• Connects to diverse data sources (CSV, Excel, databases), auto-detects column types, and suggests default chart types.
• Users can interactively build visualizations, cross-filter data, and switch chart types without coding.

• Microsoft's BI suite, similar to Tableau, supporting end-to-end data analysis and visualization.
• Integrates data preparation, visualization, and increasingly, built-in machine learning workflows.
• Focused on empowering business users or analysts to run the BI pipeline without programming.

• Another major BI offering is QlikView, emphasizing interactive dashboards and data exploration.

• Still widely used for basic EDA and visualizations directly on spreadsheets.
• Offers limited but accessible charting tools for histograms, scatter plots, and simple summary statistics.
• Data often originates from Excel/CSV files before being ingested for further analysis in Python/pandas.

• Workflow Integration: Modern BI tools are converging, adding both classic EDA capabilities and basic machine learning modeling, often through a code-free interface.
• Automation Risks and Opportunities: As drag-and-drop BI tools increase in capabilities (including model training and selection), some data science coding work traditionally required for BI pipelines may become accessible to non-programmers.
• Distinctions in Use:
  • Python libraries (Matplotlib, Seaborn, Bokeh) excel in automating and scripting EDA, report generation, and static analysis as part of data pipelines.
  • BI software (Tableau, Power BI, QlikView) shines for interactive exploration and democratized analytics, integrated from ingestion to reporting.
  • D3.js stands out for tailored, production-level, end-user app visualizations, rarely leveraged by data scientists for EDA.

Key Takeaways

• For quick, code-based EDA: Use Pandas' built-in plotters (wrapping Matplotlib).
• For pre-styled, pretty plots: Use Seaborn (with or without direct API calls).
• For interactive, shareable dashboards: Use Bokeh for Python or BI tools for no-code operation.
• For enterprise, end-user-facing dashboards: Choose BI software like Tableau or build custom apps using D3.js for total control.

Exploratory data analysis (EDA) sits at the critical pre-modeling stage of the data science pipeline, focusing on uncovering missing values, detecting outliers, and understanding feature distributions through both statistical summaries and visualizations, such as Pandas' info(), describe(), histograms, and box plots. Visualization tools like Matplotlib, along with processes including imputation and feature correlation analysis, allow practitioners to decide how best to prepare, clean, or transform data before it enters a machine learning model.

• Notes and resources at ocdevel.com/mlg/mla-8
• Try a walking desk stay healthy & sharp while you learn & code

• Position in Pipeline: EDA is an essential pre-processing step in the business intelligence (BI) or data science pipeline, occurring after data acquisition but before model training.
• Purpose: The goal of EDA is to understand the data by identifying:
  • Missing values (nulls)
  • Outliers
  • Feature distributions
  • Relationships or correlations between variables

• Data Sources: Data may arrive from various streams (e.g., Twitter, sensors) and is typically stored in structured formats such as databases or spreadsheets.
• Loading Data: In Python, data is often loaded into a Pandas DataFrame using commands like pd.read_csv('filename.csv').
• Initial Review:
  • df.info(): Displays data types and counts of non-null entries by column, quickly highlighting missing values.
  • df.describe(): Provides summary statistics for each column, including count, mean, standard deviation, min/max, and quartiles.

• Imputation:
  • Missing values must often be filled (imputed), as most machine learning algorithms cannot handle nulls.
  • Common strategies: impute with mean, median, or another context-appropriate value.
  • For example, missing ages can be filled with the column's average rather than zero, to avoid introducing skew.
• Outlier Strategy:
  • Outliers can be removed, replaced (e.g., by nulls and subsequently imputed), or left as-is if legitimate.
  • Treatment depends on whether outliers represent true data points or data errors.

• Purpose: Visualizations help reveal data distributions, outliers, and relationships that may not be apparent from raw statistics.
• Common Visualization Tools:
  • Matplotlib: The primary Python library for static data visualizations.
  • Visualization Methods:
    • Histogram: Ideal for visualizing the distribution of a single variable (e.g., age), making outliers visible as isolated bars.
    • Box Plot: Summarizes quartiles, median, and range, with 'whiskers' showing min/max; useful for spotting outliers and understanding data spread.
    • Line Chart: Used for time-series data, highlighting trends and anomalies (e.g., sudden spikes in stock price).
    • Correlation Matrix: Visual grid (often of scatterplots) comparing each feature against every other, helping to detect strong or weak linear relationships between features.

• Correlation Plot:
  • Generated with df.corr() in Pandas to assess linear relationships between features.
  • High correlation between features may suggest redundancy (e.g., number of bedrooms and square footage) and inform feature selection or removal.
• Limitations:
  • While correlation plots provide intuition, automated approaches like Principal Component Analysis (PCA) or autoencoders are typically superior for feature reduction and target prediction tasks.

• Scaling:
  • Machine learning models, especially neural networks, often require input features to be scaled (normalized or standardized).
  • StandardScaler (from scikit-learn): Standardizes features, but is sensitive to outliers.
  • RobustScaler: A variant that compresses the influence of outliers, keeping data within interquartile ranges, simplifying preprocessing steps.

• Initial Steps:
  • Load data into a DataFrame.
  • Examine data types and missing values with df.info().
  • Review summary statistics with df.describe().
• Visualization:
  • Use histograms and box plots to explore feature distributions and detect anomalies.
  • Leverage correlation matrices to identify related features.
• Data Preparation:
  • Impute missing values thoughtfully (e.g., with means or medians).
  • Decide on treatment for outliers: removal, imputation, or scaling with tools like RobustScaler.
• Outcome:
  • Proper EDA ensures that data is cleaned, features are well-understood, and inputs are suitable for effective machine learning model training.

Jupyter Notebooks, originally conceived as IPython Notebooks, enable data scientists to combine code, documentation, and visual outputs in an interactive, browser-based environment supporting multiple languages like Python, Julia, and R. This episode details how Jupyter Notebooks structure workflows into executable cells - mixing markdown explanations and inline charts - which is essential for documenting, demonstrating, and sharing data analysis and machine learning pipelines step by step.

• Notes and resources at ocdevel.com/mlg/mla-7
• Try a walking desk stay healthy & sharp while you learn & code

• Historical Context and Scope

  • Jupyter Notebooks began as IPython Notebooks focused solely on Python.
  • The project was renamed Jupyter to support additional languages - namely Julia ("JU"), Python ("PY"), and R ("R") - broadening its applicability for data science and machine learning across multiple languages.

• Interactive, Narrative-Driven Coding

  • Jupyter Notebooks allow for the mixing of executable code, markdown documentation, and rich media outputs within a browser-based interface.
  • The coding environment is structured as a sequence of cells where each cell can independently run code and display its output directly underneath.
  • Unlike traditional Python scripts, which output results linearly and impermanently, Jupyter Notebooks preserve the stepwise development process and its outputs for later review or publication.

• Stepwise Data Science Pipeline Construction
  • Import necessary libraries: Each new notebook usually starts with a cell for imports (e.g., matplotlib, scikit-learn, keras, pandas).
  • Data ingestion phase: Read data into a pandas DataFrame via read_csv for CSVs or read_sql for databases.
  • Exploratory analysis steps: Use DataFrame methods like .info() and .describe() to inspect the dataset; results are rendered below the respective cell.
  • Model development: Train a machine learning model - for example using Keras - and output performance metrics such as loss, mean squared error, or classification accuracy directly beneath the executed cell.
  • Data visualization: Leverage charting libraries like matplotlib to produce inline plots (e.g., histograms, correlation matrices), which remain visible as part of the notebook for later reference.

• Markdown Support and Storytelling

  • Markdown cells enable the inclusion of formatted explanations, section headings, bullet points, and even inline images and videos, allowing for clear documentation and instructional content interleaved with code.
  • This format makes it simple to delineate different phases of a pipeline (e.g., "Data Ingestion", "Data Cleaning", "Model Evaluation") with descriptive context.

• Inline Visual Outputs

  • Outputs from code cells, such as tables, charts, and model training logs, are preserved within the notebook interface, making it easy to communicate findings and reasoning steps alongside the code.
  • Visualization libraries (like matplotlib) can render charts directly in the notebook without the need to generate separate files.

• Reproducibility and Sharing

  • Notebooks can be published to platforms like GitHub, where the full code, markdown, and most recent cell outputs are viewable in-browser.
  • This enables transparent workflow documentation and facilitates tutorials, blog posts, and collaborative analysis.

• Cell-based Execution Flexibility

  • Each cell can be run independently, so developers can repeatedly rerun specific steps (e.g., re-trying a modeling cell after code fixes) without needing to rerun the entire notebook.
  • This is especially useful for iterative experimentation with large or slow-to-load datasets.

• Primary Use Cases

  • Jupyter Notebooks excel at "storytelling" - presenting an analytical or modeling process along with its rationale and findings, primarily for publication or demonstration.
  • For regular development, many practitioners prefer traditional editors or IDEs (like PyCharm or Vim) due to advanced features such as debugging, code navigation, and project organization.

Jupyter Notebooks serve as a central tool for documenting, presenting, and sharing the entirety of a machine learning or data analysis pipeline - combining code, output, narrative, and visualizations into a single, comprehensible document ideally suited for tutorials, reports, and reproducible workflows.

O'Reilly's 2017 Data Science Salary Survey finds that location is the most significant salary determinant for data professionals, with median salaries ranging from $134,000 in California to under $30,000 in Eastern Europe, and highlights that negotiation skills can lead to salary differences as high as $45,000. Other key factors impacting earnings include company age and size, job title, industry, and education, while popular tools and languages—such as Python, SQL, and Spark—do not strongly influence salary despite widespread use.

• Notes and resources at ocdevel.com/mlg/mla-6
• Try a walking desk stay healthy & sharp while you learn & code

• Median Global Salary: $90,000 USD, up from $85,000 the previous year.
• Regional Breakdown:
  • United States: $112,000 median; California leads at $134,000.
  • Western Europe: $57,000—about half the US median.
  • Australia & New Zealand: Second after the US.
  • Eastern Europe: Below $30,000.
  • Asia: Wide interquartile salary range, indicating high variability.

• Gender: Women's median salaries are $8,000 lower than men's. Women make up 20% of respondents but are increasing in number.
• Age & Experience: Higher age/experience correlates with higher salaries, but the proportion of older professionals declines.
• Education: Nearly all respondents have at least a master's; PhD holders earn only about $5,000 more than those with a master's.
• Negotiation Skills: Self-reported strong salary negotiation skills are linked to $45,000 higher median salaries (from $70,000 for lowest to $115,000 for highest bargaining skill).

• Industry Impact:
  • Highest salaries found in search/social networking and media/entertainment.
  • Education and non-profit offer the lowest pay.
• Company Age & Size:
  • Companies aged 2–5 years offer higher than average pay; less than 2 years old offer much lower salaries (~$40,000).
  • Large organizations generally pay more.
• Job Title:
  • "Data scientist" and "data analyst" titles carry higher medians than "engineer" titles by around $7,000.
  • Executive titles (CTO, VP, Director) see the highest pay, with CTOs at $150,000 median.

• Operating Systems:
  • Windows: 67% usage, but declining.
  • Linux: 55%; Unix: 18%; macOS: 46%; Unix-based systems are rising in use.
• Programming Languages:
  • SQL: 64% (most used for database querying).
  • Python: 63% (most popular procedural language).
  • R: 54%.
  • Others (Java, Scala, C/C++, C#): Each less than 20%.
  • Salary difference across languages is minor; C/C++ users earn more but not enough to outweigh the difficulty.
• Databases:
  • MySQL (37%), MS SQL Server (30%), PostgreSQL (28%).
  • Popularity of the database has little impact on pay.
• Big Data and Search Tools:
  • Spark: Most popular big data platform, especially for large-scale data processing.
  • Elasticsearch: Most common search engine, but Solr pays more.
• Machine Learning Libraries:
  • Scikit-learn (37%) and Spark MLlib (16%) are most used.
• Visualization Tools:
  • R's ggplot2 and Python's matplotlib are leading choices.

• Top Predictors (explaining ~60% of salary variance):
  • World/US region
  • Experience
  • Gender
  • Company size
  • Education (but amounting to only ~$5,000 difference)
  • Job title
  • Industry
• Lesser Impact: Specific tools, languages, and databases do not meaningfully affect salary.

• The greatest leverage for a higher salary comes from geography and individual negotiation capability, with up to $45,000 differences possible.
• Role/title selection, industry, company age, and size are also significant, while mastering the most commonly used tools is essential but does not strongly differentiate pay.
• For aspiring data professionals: focus on developing negotiation skills and, where possible, optimize for location and title to maximize earning potential.

Explains the fundamental differences between tensor dimensions, size, and shape, clarifying frequent misconceptions—such as the distinction between the number of features ("columns") and true data dimensions—while also demystifying reshaping operations like expand_dims, squeeze, and transpose in NumPy. Through practical examples from images and natural language processing, listeners learn how to manipulate tensors to match model requirements, including scenarios like adding dummy dimensions for grayscale images or reordering axes for sequence data.

• Notes and resources at ocdevel.com/mlg/mla-5
• Try a walking desk stay healthy & sharp while you learn & code

• Tensor: A general term for an array of any number of dimensions.

  • 0D Tensor (Scalar): A single number (e.g., 5).
  • 1D Tensor (Vector): A simple list of numbers.
  • 2D Tensor (Matrix): A grid of numbers (rows and columns).
  • 3D+ Tensors: Higher-dimensional arrays, such as images or batches of images.

• NDArray (NumPy): Stands for "N-dimensional array," the foundational array type in NumPy, synonymous with "tensor."

• Number of nested levels in the array (e.g., a matrix has two dimensions: rows and columns).
• Access in NumPy: Via .ndim property (e.g., array.ndim).

• Total number of elements in the tensor.
• Examples:
  • Scalar: size = 1
  • Vector: size equals number of elements (e.g., 5 for [1, 2, 3, 4, 5])
  • Matrix: size = rows × columns (e.g., 10×10 = 100)
• Access in NumPy: Via .size property.

• Tuple listing the number of elements per dimension.
• Example: An image with 256×256 pixels and 3 color channels has shape = (256, 256, 3).

• CSV/Spreadsheet Example: Dataset with 1 million housing examples and 50 features:
  • Shape: (1_000_000, 50)
  • Size: 50,000,000
• Image Example (RGB): 256×256 pixel image:
  • Shape: (256, 256, 3)
  • Dimensions: 3 (width, height, channels)
• Batching for Models:
  • For a convolutional neural network, shape might become (batch_size, width, height, channels), e.g., (32, 256, 256, 3).

• The term "dimensions" in data science often refers to features (columns), but technically in tensors it means the number of structural axes.
• The "curse of dimensionality" often uses "dimensions" to refer to features, not tensor axes.

• Adding Dimensions:

  • Useful when a model expects higher-dimensional input than currently available (e.g., converting grayscale image from shape (256, 256) to (256, 256, 1)).
  • Use np.expand_dims or array.reshape.

• Removing Singleton Dimensions:

  • Occurs when, for example, model output is (N, 1) and single dimension should be removed to yield (N,).
  • Use np.squeeze or array.reshape.

• Wildcard with -1:

  • In reshaping, -1 is a placeholder for NumPy to infer the correct size, useful when batch size or another dimension is variable.

• Flattening:

  • Use np.ravel to turn a multi-dimensional tensor into a contiguous 1D array.

• Transposing Axes:
  • Needed when model input or output expects axes in a different order (e.g., sequence length and embedding dimensions in NLP).
  • Use np.transpose for general axis permutations.
  • Use np.swapaxes to swap two specific axes but prefer transpose for clarity and flexibility.

• In NLP sequence models:
  • 3D tensor with (batch_size, sequence_length, embedding_dim) might need to be reordered to (batch_size, embedding_dim, sequence_length) for certain models.
  • Achieved using: array.transpose(0, 2, 1)

• reshape: General function for changing the shape of a tensor, including adding or removing dimensions.
• expand_dims: Adds a new axis with size 1.
• squeeze: Removes axes with size 1.
• ravel: Flattens to 1D.
• transpose: Changes the order of axes.
• swapaxes: Swaps specified axes (less general than transpose).

• A deep understanding of tensor structure - dimensions, size, and shape - is vital for preparing data for machine learning models.
• Reshaping, expanding, squeezing, and transposing tensors are everyday tasks in model development, especially for adapting standard datasets and models to each other.