At 9:14 AM, a single ESG compliance task entered our platform. Twenty-three hours later, it had spawned 847 pods, orphaned 200 GPU-consuming workers across three orchestrator crashes, exhausted resource quotas during an end-of-quarter peak, and delivered results two hours past SLA at 340% of estimated cost. No Kubernetes alert fired until production was already broken.

This is not a misconfiguration story. Kubernetes assumes submitted workloads are static and resource footprints are known upfront. AI agent orchestration breaks every one of those assumptions - an agent processing a 500-property EU Taxonomy compliance portfolio dynamically spawns sub-agents based on complexity discovered at runtime. The execution graph is fundamentally unknowable at submission time.

This session walks through five architectural problems we solved with no off-the-shelf solution: a custom hierarchical budget controller that caps agent trees at the root envelope rather than flat namespace limits; an orphan detection system using pod ancestry labels and TTL controllers that cleans up execution chains when orchestrators crash; GPU burst scheduling redesign that cut scheduler lag from 40 minutes to under 90 seconds during correlated inference peaks; MIG partitioning on H100 hardware for regulatory-grade multi-tenant GPU memory isolation; and a cost attribution model that tags GPU-hours down to client, task, agent type, and execution tier with FinOps anomaly alerts.

We close with the three architectural decisions made during the initial build that directly caused the 847-pod incident - and why each seemed correct at the time.

聽眾收穫：

The audience will learn the following:

1. Why standard Kubernetes primitives fail structurally for dynamic agent workloads — and what those failures look like in production before you recognize them as architectural problems.
2. Hierarchical budget controller design that enforces root-level resource envelopes across dynamically-spawned agent trees.
3. Orphan detection and cleanup for pods spawned via in-cluster API calls where ownerReferences can't be established statically.
4. GPU burst scheduling for LLM inference workloads with correlated demand spikes across concurrent agent populations.
5. MIG partitioning for GPU-level tenant isolation beyond what namespace isolation provides.