At 9:14 AM, a single ESG compliance task entered our platform. Twenty-three hours later, it had spawned 847 pods, orphaned 200 GPU-consuming workers across three orchestrator crashes, exhausted resource quotas during an end-of-quarter peak, and delivered results two hours past SLA at 340% of estimated cost. No Kubernetes alert fired until production was already broken.

This is not a misconfiguration story. Kubernetes assumes submitted workloads are static and resource footprints are known upfront. AI agent orchestration breaks every one of those assumptions - an agent processing a 500-property EU Taxonomy compliance portfolio dynamically spawns sub-agents based on complexity discovered at runtime. The execution graph is fundamentally unknowable at submission time.

This session walks through five architectural problems we solved with no off-the-shelf solution: a custom hierarchical budget controller that caps agent trees at the root envelope rather than flat namespace limits; an orphan detection system using pod ancestry labels and TTL controllers that cleans up execution chains when orchestrators crash; GPU burst scheduling redesign that cut scheduler lag from 40 minutes to under 90 seconds during correlated inference peaks; MIG partitioning on H100 hardware for regulatory-grade multi-tenant GPU memory isolation; and a cost attribution model that tags GPU-hours down to client, task, agent type, and execution tier with FinOps anomaly alerts.

We close with the three architectural decisions made during the initial build that directly caused the 847-pod incident - and why each seemed correct at the time.

聽眾收穫：

The audience will learn the following:

1. Why standard Kubernetes primitives fail structurally for dynamic agent workloads — and what those failures look like in production before you recognize them as architectural problems.
2. Hierarchical budget controller design that enforces root-level resource envelopes across dynamically-spawned agent trees.
3. Orphan detection and cleanup for pods spawned via in-cluster API calls where ownerReferences can't be established statically.
4. GPU burst scheduling for LLM inference workloads with correlated demand spikes across concurrent agent populations.
5. MIG partitioning for GPU-level tenant isolation beyond what namespace isolation provides.

Kubernetes treats compute as fungible. A node with eight H100 GPUs in rack A is scheduled identically to one in rack B. The scheduler has no awareness of NVLink domains, PCIe switch topology, or InfiniBand fabric zones. For multi-node AI training, this determines whether a job runs at 60% efficiency or 95% efficiency, consuming 40% more GPU-hours than projected. The physics is straightforward: intra-node NVLink delivers ~600 GB/s bidirectional, inter-node InfiniBand HDR ~200 GB/s, and crossing congested fabric boundaries drops further. AllReduce gradient synchronization is bounded by the slowest link. Placing a four-node job across four separate leaf switches yields 40–60% throughput degradation, generating no Kubernetes alert and no dashboard signal, just jobs running slower than estimates. The common response is to provision more GPUs.

This talk builds topology-aware scheduling for multi-node AI workloads. We expose physical topology using extended resources and node labels derived from switch LLDP data and fabric management APIs, combined with the Node Topology Manager for NUMA GPU affinity. Kueue ResourceFlavors represent topology domains as scheduler-visible constructs. Gang scheduling via Kueue cohort placement ensures all pods of a training job land within the same InfiniBand leaf switch domain before any pod starts. An admission webhook validates topology constraints at submission time, avoiding placement failures after GPU allocation. For observability, DCGM-based monitoring uses Model FLOP Utilization as a first-class efficiency metric alongside SM utilization, because SM utilization alone is insufficient to prove effective GPU use.

We address the harder organizational problem: topology data lives in data center infrastructure documentation and switch management systems, not in Kubernetes-queryable APIs. We cover the data pipelines required between infrastructure, ML platform, and operations teams, and how to formalize those interfaces. Benchmark data compares topology-aware versus unaware placement, with specific Kueue configuration for H100 NVLink domains and InfiniBand HDR fabric.