Managing AI Costs: Token Optimization, Caching, Model Routing

AI infrastructure costs aren't a strategy problem — they're an engineering problem. This episode is the war story session: the developer who hit $3,200 in a single month (22% from a CI/CD staging loop hitting the live API 40,000 times per commit), the 3am retry nightmare that burned $500 in one night from a primitive while-loop hitting a 429 error, and the 49-agent refactoring task that burned 887,000 tokens per minute before the actual work started. Then the fixes: 2026 model pricing head-to-head (GPT-5.2 at $1.75/$14, Gemini 3.1 Pro at $2/$12, Claude Opus 4.6 at $5/$25 per million tokens), the 200K context cliff that doubles your bill on a single token overage, prompt caching math (5-min cache breaks even on request 2, 1-hour cache breaks even on request 8), Microsoft's LLM Lingua compression framework (50–80% input reduction with near-zero quality loss), Redis semantic caching with HNSW vector search at 27ms vs several seconds for live inference, cascade model routing with RouteLLM and Bifrost's code mode (90% MCP schema compression), Upstash token bucket rate limiting with the ephemeral cache gotcha, and pre-flight tokenizer checks that kill the request before it hits the wire.