Kubernetes treats compute as fungible. A node with eight H100 GPUs in rack A is scheduled identically to one in rack B. The scheduler has no awareness of NVLink domains, PCIe switch topology, or InfiniBand fabric zones. For multi-node AI training, this determines whether a job runs at 60% efficiency or 95% efficiency, consuming 40% more GPU-hours than projected. The physics is straightforward: intra-node NVLink delivers ~600 GB/s bidirectional, inter-node InfiniBand HDR ~200 GB/s, and crossing congested fabric boundaries drops further. AllReduce gradient synchronization is bounded by the slowest link. Placing a four-node job across four separate leaf switches yields 40–60% throughput degradation, generating no Kubernetes alert and no dashboard signal, just jobs running slower than estimates. The common response is to provision more GPUs.

This talk builds topology-aware scheduling for multi-node AI workloads. We expose physical topology using extended resources and node labels derived from switch LLDP data and fabric management APIs, combined with the Node Topology Manager for NUMA GPU affinity. Kueue ResourceFlavors represent topology domains as scheduler-visible constructs. Gang scheduling via Kueue cohort placement ensures all pods of a training job land within the same InfiniBand leaf switch domain before any pod starts. An admission webhook validates topology constraints at submission time, avoiding placement failures after GPU allocation. For observability, DCGM-based monitoring uses Model FLOP Utilization as a first-class efficiency metric alongside SM utilization, because SM utilization alone is insufficient to prove effective GPU use.

We address the harder organizational problem: topology data lives in data center infrastructure documentation and switch management systems, not in Kubernetes-queryable APIs. We cover the data pipelines required between infrastructure, ML platform, and operations teams, and how to formalize those interfaces. Benchmark data compares topology-aware versus unaware placement, with specific Kueue configuration for H100 NVLink domains and InfiniBand HDR fabric.

The audience will learn the following:

1. Why default Kubernetes scheduling produces 40–60% throughput degradation for multi-node distributed training and the network physics driving that number
2. How to expose GPU cluster physical topology to the Kubernetes scheduler using extended resources, node labels, and the Topology Manager API
3. Kueue ResourceFlavor and ClusterQueue configuration for topology-constrained multi-node distributed training job placement
4. Gang scheduling implementation using Kueue cohort constraints that enforce topology locality before any pod in a training job begins execution
5. Admission webhook architecture for topology validation at job submission time rather than after GPU allocation begins