Arko C

FP8 halved a speech model's weight bytes. At batch 1, cold, one stage got 20% faster. The other got 5.5% slower. The decoder waits on memory. The other waits on a chain of small dependent steps, so free dequant buys 2.2% at most. Bytes only buy latency where you wait on bytes.

Qwen3.6-27B on one H100 SXM. How long before the first token can even exist? Two limits decide it, and which one wins depends on how long the prompt is. Limit 1 is reading the weights: a text prompt only reads the decoder layers and the lm_head, 24.35B + 1.27B = 25.62B params = 51.25 GB in bf16. The vision tower and MTP head never run, and the embedding is just a row lookup. floor ≈ weight bytes ÷ bandwidth = 51.25 GB ÷ 3.35 TB/s ≈ 15.3 ms No 10-token prompt can return sooner. Limit 2 is the math. Prefill FLOPs ≈ 2 × 24.35B × n for a prompt of n tokens, with the output head only running for the last token. Assume the H100 sustains 60% of its 989.4 TFLOP/s peak, so 593.6 TFLOP/s. That's about 83 ms per 1,000 tokens. crossover n ≈ 15.3 ms × 593.6 TFLOP/s ÷ (2 × 24.35B) ≈ 186 Call it around 190 tokens. From there the first token gets later almost in a straight line: about 0.86 s at 10,000 tokens. The surprise is attention. Causal full-attention FLOPs ≈ 2 · n² · 24 heads · 256 head_dim · 16 layers, which at n = 10,000 is 19.66 TFLOP, about 4% of the math. Qwen3.6 is a hybrid. 48 of its 64 layers use Gated DeltaNet linear attention, whose cost grows with prompt length rather than its square, so only 16 layers pay the quadratic price. At 100,000 tokens that share climbs to almost 30%, and attention kernels start to matter. So at everyday prompt lengths, the kernels that decide this model's first token aren't the ones most people optimise first. Nearly all of the math is matrix multiplies, so every point of GPU utilisation won there comes straight off the first token. Going from 60% to 70% of peak takes 10,000 tokens from about 0.86 s to about 0.74 s. The 48 linear-attention layers do little math, but their chunked kernels are bound by memory traffic and launch overhead rather than tensor-core math, so they take a bigger share of the time than of the FLOPs. That's where custom kernels earn their keep.