Why are memory and chip interconnects getting so much attention in AI hardware? Faster GPUs still matter, but AI performance is increasingly about how much data you can keep close to the compute, and how quickly you can move it there. Text generation is a good example. Take NVIDIA’s H100 SXM. It can theoretically do about 989 trillion operations per second for dense BF16 tensor math. It also has 80 GB of memory with about 3.35 TB/s of memory bandwidth. Now take Llama 3.1 70B. Its roughly 70.6 billion weights occupy about 141 GB in BF16, before accounting for the other memory needed to run the model. So you already need multiple 80 GB H100s just to hold it. When a dense model generates a token, it has to use essentially all those weights. The dominant operations are roughly a multiplication and addition per weight. That creates an interesting problem: enormous amounts of data movement for relatively little computation per byte. Imagine an incredibly fast worker doing calculations from number cards arriving on a conveyor belt. If the worker finishes each calculation before the next numbers arrive, giving them a faster calculator doesn’t accomplish much. The bottleneck is the belt. GPU memory bandwidth can play the role of that belt. In my simplified roofline calculation, generating tokens for one conversation can use only about 0.4% of the H100’s peak BF16 math throughput. Batch 64 conversations together and the theoretical ceiling rises to roughly 11%. Those aren’t benchmark results. They depend on assumptions including BF16, an 8K context and how the model is split across GPUs. But the intuition matters. Batching doesn’t make memory faster. It lets the GPU reuse each weight across many conversations, doing more useful math for every byte moved. And low math utilization doesn’t mean the GPU is idle. The memory system can be extremely busy while the calculating units wait for data. This also explains why reading a prompt and generating an answer behave differently. Prompt processing can work on many tokens together and reuse weights heavily, making it more compute-intensive. Token generation is sequential within each conversation and, especially at smaller batch sizes, is often much more constrained by memory bandwidth. Then add agents. A coding agent might ask a model for a fix, edit files, compile code, run tests, inspect the result and call the model again. The model inference may run on GPUs, while compilation, tests and orchestration run mostly on CPUs. Storage, networks and external services introduce still more bottlenecks. Same “AI,” very different hardware problems. Memory capacity determines how much data fits. Memory bandwidth determines how quickly it moves. Interconnects determine how efficiently work and data move between chips. CPUs handle much of the surrounding general-purpose work. And GPUs and other accelerators provide enormous amounts of specialized computation. This is why peak FLOPS alone tell you surprisingly little about how fast or economical an AI system will be. My bet is that moving, storing, and reusing data will shape the next decade of AI hardware as much as doing more math. If you’re building or buying AI infrastructure, the question is: where would the next dollar spent on hardware actually make the system faster?