Spheron Network

A 70B model needs about 140GB of VRAM at FP16, before you've loaded a single token of context. That number decides your GPU shortlist before anything else does. B200 fits the full 70B model in FP16 on one chip with 100GB+ left over for KV cache. H100 can't hold it at FP16 but serves anything up to 34B comfortably, over 10,000 tok/s on optimized inference. H200 pulls ahead of H100 specifically on throughput at that scale, 31,712 tok/s versus 22,290 on Llama 2 70B, about 42% more. Budget another 20 to 30% on top of the base model size for KV cache, activations and framework overhead, then add roughly 0.5 to 1GB per 1,000 tokens of context for a 7B-class model. Size the GPU to the model, not the other way around.

Two GPUs, same FLOPs headline, very different multi-GPU training behavior once you add a third and fourth. H100's NVLink 4.0 runs 900 GB/s bidirectional. A100's NVLink 3.0 runs 600 GB/s. That 33% gap doesn't show up in any single-GPU benchmark, it shows up as gradient-sync overhead once you scale out, and it's a real reason H100's 8-GPU scaling efficiency lands around 90% against A100's 85%. Both chips carry 80GB of VRAM and support up to 7 MIG instances, so capacity was never the differentiator here. Interconnect was. Benchmark your actual multi-GPU job, not the single-card spec sheet.

AWS says Trainium 3 is 30-50% cheaper than H200 for inference. The math behind that headline is real: $0.28 per million tokens versus $0.84 on H200 on-demand. What's missing from the headline: migrating a model to Trainium's stack costs about $36,000 in engineering time, modeled at three weeks and two engineers at $150/hr. A supported architecture takes 2-3 weeks to port, a custom one 4-6 weeks. Break-even against H200 spot pricing lands around 109 billion tokens processed. If your workload won't clear that before the next hardware cycle, the "cheaper" chip is the more expensive decision. Numbers

DeepSeek R2 can emit 40,000 reasoning tokens before it answers. A standard LLM emits about 400. That's 100x the KV cache pressure, and it breaks a server configured like a normal chat model. Full precision needs 8x H200 SXM5, 1,128GB of memory. FP8 fits on 8x H100 SXM5 instead, 640GB, with the checkpoint itself under 600GB. MLA compression is why it fits at all: roughly 2.1GB of KV cache for a 30,000-token chain versus 15GB+ for a comparable GQA architecture. FP8 costs under 1% accuracy on MATH-500 and under 2% on AIME, for 1.6 to 1.9x more throughput than FP16. INT4 is the one to skip: 10 to 18% accuracy loss on AIME.

KV cache is usually the thing nobody profiles until it blows the memory budget. KIVI quantizes it to 2-bit and keeps accuracy mostly intact. The compression: roughly 8x on the quantized portion versus a straight 16-bit cache. Pair that with FP8 KV cache and per-token cache cost drops to 54% of the BF16 baseline before INT2 is even applied. Worst accuracy hit observed: Mistral-7B on GSM8K lost 2.35 points. Llama-2-7B on CoQA lost only 0.83. A 128-token residual window stays at full fp16 so recent tokens hold precision. Runs on an A100 80GB at $1.48/hr on-demand. Config

vLLM and SGLang are within 4% of each other on raw throughput. At c=50 unique-prompt load they land at 1,850 versus 1,920 tok/s. The real split shows up somewhere else entirely. TTFT at c=100 concurrency: vLLM 620ms p50, SGLang 370ms, a 40%+ gap. That's RadixAttention's prefix caching doing its job, and the advantage scales with overlap: 18% TTFT reduction at 256 shared tokens, up to 42% at 2,048. It's not free everywhere. Deeply nested JSON structured output adds 42% overhead on vLLM versus 6% for flat JSON, and on MoE workloads like DeepSeek V4 the two engines close to within 7%. Pick on your traffic shape, not a single throughput number.

We benchmarked sequence packing against padding on the same fine-tuning job: single H100 SXM5, Llama 3.1 8B, LoRA rank 16, batch size 16, max sequence length 2,048. Padded run: 58 minutes wall-clock, burning 61% of tokens on padding. Packed run: 31 minutes, under 2% waste. A 1.9x speedup from one config change. Cost dropped from $2.55 to $1.36 per run, 47% cheaper. Eval loss barely moved: 0.842 padded vs 0.839 packed. If you're still padding to a fixed sequence length, you're paying for tokens that do nothing. Setup

MI300X ships 192GB of HBM3 on a single GPU, enough to fit models that need two H100s. That doesn't mean ROCm closes the software gap. At batch 1 to 4, H100 with TensorRT-LLM holds a 20 to 30% throughput lead. At batch 64 to 128 that narrows to 5 to 10%, so the gap mostly matters at low concurrency. What doesn't narrow: - TensorRT-LLM has no ROCm port and none planned - FlashAttention 3 is Hopper-only; AMD tops out at Flash Attention 2 via MIOpen - DeepSpeed works from v0.6+, but some Megatron-LM fused kernels aren't ported More memory per GPU, less tooling support. Weigh both before you pick.

RTX 5090 decodes 4x faster than DGX Spark on the same model. The catch is which model it can run at all. On GPT-OSS 20B at MXFP4: RTX 5090 hits 205 tok/s decode, DGX Spark manages 49.7. The reason is bandwidth, 1,792 GB/s on the 5090 against just 273 GB/s on Spark's unified memory. But Spark's 128GB of unified memory fits a 70B model in FP8 at all, 2.7 tok/s decode, slow but running. The 5090's 32GB GDDR7 can't load that model regardless of speed. Buy bandwidth for small models you'll run fast. Buy unified memory for big models you just need to run.