A fun realization I had recently: with Blackwell (B200 and B300), we are basically back to solving classic NUMA problems inside a single GPU package. To software, a B200 looks like one big unified GPU. But physically it is two silicon dies connected by NV-HBI. And while that cross-die link is fast, local HBM is still noticeably closer. If you ignore this, memory-heavy workloads like MoE layers spend half their time pulling weights across the link, which burns latency and power for no good reason. Pay attention to MegaMoE in DeepGEMM. In fact, CUDA doesn’t have an API to tell you which SM sits on which die, so just measure it directly with a few lines of pointer chasing. 1. Slice the weights in half onto physical memory for each die, and run a persistent scheduler where SMs pull from their local die first. 2. Only steal work across dies if there are at least two full waves of tasks left. Why? Otherwise, paying the transit latency for small leftovers just isn't worth it. Then, 4D TMA descriptor can do coordinate routing so the inner loop stays completely clean. The payoff is ~100 W of package power saved and 5-10% higher sustained clocks. (Which can be hundreds of thousands of $$ saved in real deployment).