二舅

Why KV cache stores K and V vectors but never Q? (a popular technical LLM interview question) LLMs are autoregressive so each token is predicted from every token before it, one at a time. This autoregressive nature has a direct consequence inside the model. A forward pass over <n> tokens produces <n> hidden states, but only the last one is projected to logits and is required to generate the next token. So to understand why KV cache just stores K and V vector, we must back track to see how exactly is the last hidden state produced. Let's walk through this with a 10-token prompt. 1) Prefill: All 10 tokens go through the model in one forward pass, in parallel (with causal masking), since the whole prompt is already known. At every layer, each of the 10 positions produces a query, a key and a value vector, and attention at each position runs against all positions up to it. This pass is compute-heavy, and it's why the first token takes noticeably longer than the ones after it. TTFT is mostly prefill. 2) The first output token: To generate the 11th token, only the 10th token's hidden state is needed. So this is projected from the hidden-dim to vocab-dim to generate logits over vocab. These logits then go through softmax and sampling to generate token 11. 3) Back-track the hidden state: The last hidden state is the last row of the feedforward block's output. The feedforward block is position-wise (it's applied to each row independently) so that row comes from the last row of the attention output before it. So now we need to see how the last row of attention is computed. 4) Attention matrix: QKᵀ for a 10-token prompt will give a 10 × 10 matrix. Row <i> will have the dot product of query <i> with every key. Row 10 is therefore Q₁₀·K₁, Q₁₀·K₂, all the way to Q₁₀·K₁₀. Notice that only Q₁₀ appears in it. Q₁ through Q₉ only belong to their corresponding rows 1-9, and those rows' hidden states we already discarded because they were never needed. The last row of attention goes through softmax and multiplies the full stack of value vectors, V₁ through V₁₀, to give the last row of the attention output. So the last hidden state depends on exactly three things: Q₁₀, every key, and every value. 5) Generating token 12: Token 11 is appended, and this time, we need row 11's hidden state to generate token 12. Mathematically, attention operation turns out to be Q₁₁ against K₁ through K₁₁, then multiplied by V₁ through V₁₁. K₁ through K₁₁ and V₁ through V₁₁ are bit-for-bit what prefill + first token produced since under causal masking, a token's key and value depend on that token and the ones before it, never on anything after, so appending token 11 cannot change anything at position 3. 6) The cache state: Overall, this implies that you just need to retain the keys and values at each decoding step, and compute only the new position's Q, K and V. Each decode step requires one query vector, which is never used again, so they are never cached across the decoding process. The visual below explains the entire process. That said, KV cache is only one of four separate caching layers in an LLM stack. The other three are prefix caching on the server, prompt caching billed by a provider, and a semantic cache that skips the model entirely. I wrote a full breakdown of all four caches in LLM serving that you should know as an AI engineer, with code for each. Read it below.