A counterintuitive insight about LLM inference: LLM inference can get faster even when every GPU operation takes exactly the same amount of time. The improvement can come entirely from reducing the time between operations. An LLM forward pass appears to be one computation. But on the GPU, it becomes a sequence of smaller operations for attention, matrix multiplication, normalization, sampling, and data movement. The CPU usually prepares and submits these operations individually. That overhead is small for one launch. But token-by-token decoding repeats a similar sequence for every generated token. Some kernels complete in only a few microseconds. The GPU can finish one operation before the CPU has submitted the next. The resulting gaps contain no useful model computation, and the profiler shows the GPU waiting for its next work. This means optimizing inference is not always about accelerating the kernels themselves. Instead, sometimes the larger opportunity is removing the empty waiting time between them. CUDA Graphs are designed for this situation. A CUDA Graph is a saved execution plan containing a sequence of GPU operations and their dependencies. The application first runs a warm-up pass. This handles one-time initialization and establishes the shapes, buffers, kernels, and memory addresses required by the workload. The runtime then captures the operation sequence as a graph. Later iterations can replay the entire graph through one launch instead of submitting every operation separately. Replay does not reuse an earlier output. New input is placed into the expected buffers. The GPU runs the captured operations again and produces a new result. The computation remains almost the same. What changes is how efficiently that computation reaches the GPU. This works particularly well during LLM decoding because every step processes a new token while following a largely repeated execution path. The tradeoff is reduced flexibility. A captured graph expects compatible operations, tensor shapes, control flow, and memory addresses across replays. Real inference traffic rarely has one fixed shape. Batch sizes change, requests enter and leave, and servers switch between prefill and decoding workloads. Serving engines often capture several graphs for common execution shapes. Other requests can be padded to a captured size or handled through eager execution. More captured shapes improve coverage, but consume additional GPU memory and increase startup work. CUDA Graph replay is useful because it optimizes something kernel benchmarks can easily hide: the cost of repeatedly asking the GPU to begin its next piece of work. The visual below shows how warm-up, capture, and replay remove that repeated launch path. This is one reason LLM inference needs a specialized serving engine rather than a standard prediction API. I also wrote an article that explains five architectural differences between traditional ML inference and LLM inference. Read it below.