AI Software Engineer (Intern) @Intel 🇮🇳 | MTech AI, IIT Patna I make LLMs run fast. Daily posts on new models, GPUs & open-source AI
India
Joined June 2022
- Tweets274
- Following909
- Followers56
- Likes132
Pinned Tweet
IQuest-Q1 dropped today: 320B MoE, 15B active, 512K context, open weights, built for CLI agents (claude code / codex)
did the memory math on the card's own launch command: --tp-size 8 --dtype bfloat16
320B x 2 bytes = 640 GB of weights. 8x H100 80GB is 640 GB total. it doesn't fit. you need 8x H200 (1,128 GB), B200s, or an fp8 quant
the nice part is the attention: 3 sliding-window layers (4K window) per 1 full layer, 8 KV heads. KV is ~48 GB per 512K session vs ~189 GB if all 88 layers were full. on 8x H200 that leaves room for ~10 full-context sessions before overhead
also the 83.2 terminal-bench 2.1 headline ran with an 8-hour per-task limit
anyone serving this on hopper yet?
Interview question: Llama 3.1 8B in BF16 on one H100 SXM. Scenario rental: $2.50 per GPU-hour. What does 1M output tokens cost?
Batch 1 decode reads all weights per token: 8.03B × 2 bytes = 16.06 GB
Ceiling = 3.35 TB/s ÷ 16.06 GB ≈ 209 tok/s
$2.50 ÷ (209 × 3,600) × 1M ≈ $3.33 per 1M tokens
Most people answer "$3.33, so self-hosting is expensive." That's incomplete.
→ Batch 64 shares one weight read per step. Bytes/step = 16.06 GB + KV (64 × 2K tokens × 128 KiB = 17.2 GB) = 33.2 GB
→ 3.35 TB/s ÷ 33.2 GB ≈ 101 steps/s × 64 ≈ 6,450 tok/s
→ ≈ $0.11 per 1M tokens, about 31× cheaper
→ Both are bandwidth ceilings. Measure real throughput
Cost per token is set by batch size, not GPU price.
#LLMInference #LLM #MLOps #AIEngineering #GPU
Interview question: Your agent takes 8 sequential steps per task. Each step (scenario numbers): 0.4 s to first token, 200 output tokens at 80 tok/s, then a 0.8 s tool call. Users say it feels slow. How slow is it?
Decode/step = 200 ÷ 80 = 2.5 s
Step = 0.4 + 2.5 + 0.8 = 3.7 s
Task = 8 × 3.7 = 29.6 s
Most people answer "Use a faster GPU." That's incomplete.
→ Decode is 68% of each step. Cut the output to 60 tokens: 8 × 1.95 s = 15.6 s
→ Parallel tool calls: 3 independent lookups in one turn, 8 steps become 5 → 18.5 s
→ Do both: 5 × 1.95 s = 9.75 s
→ Stream progress so the wait feels shorter
Agent latency = steps × tokens. Cut both before you buy hardware.
#AgenticAI #AIAgents #LLM #GenAI #AI
Interview question: Your RAG app serves 1M queries a day. Each prompt is top-10 chunks × 512 tokens + a 1,000-token system prompt + a 50-token question. Scenario price: $3 per 1M input tokens. What's the input bill?
Input/query = 10 × 512 + 1,000 + 50 = 6,170 tokens
1M × 6,170 × $3/1M = $18,510 a day → ~$555K a month
Most people answer "Switch to a cheaper model." That's incomplete.
→ Rerank and keep the top 3: 2,586 tokens, 58% less → $7,758 a day
→ Cache the system prompt (scenario: cached tokens at 10% price) → $5,058 a day
→ 256-token chunks, top 3: 1,818 tokens per query
→ Check answer quality after each cut
Your context size is the bill. Retrieve less, rank better.
#RAG #LLM #GenAI #AIEngineering #MachineLearning
Interview question: Your RAG corpus is 10M chunks. Embeddings are 1024-dim FP32. The vector DB box has 32 GB RAM. Does the index fit?
Vectors = 10M × 1024 dims × 4 bytes = 40.96 GB
HNSW graph (M=32 → 64 links × 4 bytes per vector at layer 0) = +2.56 GB
Total ≈ 43.5 GB → doesn't fit
Most people answer "Get a 64 GB box." That's incomplete.
→ FP16: 20.48 GB, usually near-zero recall loss
→ INT8 scalar quantization: 10.24 GB
→ Binary (1 bit/dim): 1.28 GB, then rescore top-100 with full vectors from disk
→ Matryoshka model? Truncate to 256 dims → 10.24 GB in FP32
Measure recall@10 after every cut. Memory is cheap to save, recall is expensive to lose.
#RAG #VectorDatabase #LLM #AIEngineering #GenAI
Interview question: Llama 3.1 8B on one 80 GB GPU. Product wants 32 concurrent users at 32K context. Does it fit?
KV/token = 2 (K,V) × 32 layers × 8 KV heads × 128 head_dim × 2 bytes (BF16) = 128 KiB
32 users × 32K tokens = 1M tokens → 128 GiB of KV, plus ~16 GB of weights
Most people answer "No, add a GPU." That's incomplete.
→ Size for real lengths: paged KV allocates as tokens arrive. Avg 8K → 32 GiB
→ FP8 KV cache halves it: 64 KiB/token
→ Shared system prompt? Prefix caching stores it once
→ Decide what happens at p99: queue or preempt, not OOM
Size KV for your length distribution, not the max context.
#LLM #MachineLearning #DeepLearning #AIEngineering #GenAI
The engineer who led the BLOOM-176B training wrote down everything he learned.
Then put it on GitHub. For free.
Not a course. A field manual for running LLMs on real GPUs.
→ Insights: picking a cloud, when a GPU upgrade is worth it
→ Hardware: compute, storage, networking
→ Orchestration: SLURM and containers
→ Training: model parallelism, performance, fault tolerance
→ Inference: prefill vs decode, paged attention, speculative decoding
→ Frameworks: vLLM, SGLang, TensorRT-LLM, llama.cpp
→ Debugging and testing multi-GPU PyTorch
Plus copy-paste scripts to benchmark your network and measure the TFLOPS you actually get.
The lessons most people only learn after burning a lot of GPU hours.
GitHub (free, ~19.5K ⭐): github.com/stas00/ml-enginee…
Follow for more practical AI engineering resources.
#LLM #MLOps #GPU #DeepLearning #OpenSource
llm inference interview question (quantization edition)
Llama 3.1 8B on one A100 80GB, vLLM.
you swap BF16 weights for an AWQ INT4 checkpoint. weights go ~16 GB to ~5.7 GB.
1 user: 90 to 190 tok/s. great.
128 concurrent at peak: 5,200 to 3,400 tok/s total. 35% slower.
no OOMs, evals look fine.
why did it get slower, and what do you ship?
my answer: AWQ INT4 is weight-only (W4A16). at 1 user decode is memory-bound, you read ~3x fewer weight bytes per token, so ~2x faster.
at 128 concurrent the matmuls are compute-bound. saved bytes stop mattering and every GEMM now pays to dequantize INT4 to FP16, often in a less tuned kernel than cuBLAS BF16.
check which kernel vLLM picked (awq vs awq_marlin) and profile a decode step at high batch.
ship: marlin kernels (awq_marlin / gptq_marlin), or W8A8 (INT8 on A100) for the high-traffic endpoint. keep INT4 for low concurrency or tight memory.
unpopular opinion: most speculative decoding speedups you see online are measured at an acceptance rate you won't get in prod
the math is simple (leviathan et al. 2023). with acceptance rate a and k draft tokens, each target forward pass gives you on average (1 - a^(k+1)) / (1 - a) tokens
a = 0.8, k = 4: 3.36 tokens per pass
a = 0.6, k = 4: 2.31
a = 0.4, k = 4: 1.65
and going from k = 4 to k = 8 at a = 0.6 only buys you 2.31 to 2.47. more draft tokens don't save a weak draft
that's also before the draft model's own cost and the extra verify compute. at high batch you're already compute bound, so those rejected tokens aren't free anymore
what i'd do before shipping it: log acceptance rate per route in prod traffic, not on an offline greedy benchmark, and switch it off above some batch size
what acceptance rates are you actually seeing?
standard attention vs flashattention, clearly explained (bookmark this)
same math, same output. the difference is where the NxN score matrix lives.
standard: compute QK^T, write the full NxN matrix to HBM, read it back for softmax, write it again, read it again to multiply by V. at 32k tokens that matrix is 2 GiB per head per sequence in fp16.
flashattention: load Q, K, V in tiles into on-chip SRAM (~20 MB at ~19 TB/s on an A100, vs 40 GB of HBM at ~1.5 TB/s), do softmax with a running max and sum, write only the output. the NxN matrix never touches HBM, so attention memory goes from O(N^2) to O(N).
my take: long-context attention is limited by HBM traffic, not FLOPs. FA wins by moving fewer bytes, not by doing less math (backward actually recomputes).
anyone still serving without it?
llm inference interview question (cost-cut edition)
Llama 3.1 70B FP8 on 8x H100 SXM, vLLM, TP=8. ~3,000 tok/s total at 64 concurrent.
finance wants it cheaper, so you move to 8x L40S. same model, same TP=8, same traffic.
L40S has ~3.9x less memory bandwidth, so you budget ~775 tok/s.
you get 310. no OOMs, GPUs look busy.
what do you check first?
answer: the all-reduces. TP=8 does 2 per layer each decode step, so 160 per token on 80 layers, and on L40S they go over PCIe Gen4 (~64 GB/s vs 900 GB/s NVLink). comms eat the step, and NCCL kernels make GPUs look busy.
check: nsys one decode step for the all-reduce share, nvidia-smi topo -m.
fix: lower TP + more replicas (2x TP=4), keep TP groups on one PCIe switch, or keep 70B on NVLink boxes.
if i had 2 weeks to prep for an LLM inference interview, this is the order i'd go in (bookmark this)
days 1-3: prefill vs decode, KV cache math. you should be able to say llama 3.1 8B needs 128 KiB of KV per token in bf16 without looking it up
days 4-5: why decode is memory-bound. arithmetic intensity, roofline
days 6-7: paged attention, prefix caching, flashattention
days 8-9: continuous batching, chunked prefill, TTFT vs TPOT vs goodput
days 10-11: quantization. weight-only vs W8A8, fp8, KV cache quant
day 12: spec decoding and when it hurts
day 13: TP vs PP vs EP, MoE serving
day 14: one system design out loud
most people jump straight to day 12. interviewers start at day 1
what would you add?
i turned this into a free pdf: 25 llm inference interview questions with the math worked out, same order as the plan
topmate.io/harshal_dharpure
recursive just shipped flint, their first model: 40B total params, only 2.5B active per token, apache 2.0. they claim 125+ tok/s on an M4 Pro macbook
the inference math says that's believable
the 4-bit MLX weights are ~22.6 GB, so ~4.5 bits per param. a decode step only reads the active experts: 2.5B x 4.5 bits = ~1.4 GB per token. M4 Pro has 273 GB/s, so the ceiling is ~190 tok/s. 125 is ~65% of that, which is very good for a laptop
same bits on a dense 40B: 22.6 GB per token, ceiling ~12 tok/s
total params decide if it fits in RAM. active params decide how fast it runs
would you run your coding agent locally on this?
flint launch post with all the numbers: recursive.com/updates/introd…
deepseek open-sourced DeepInfer, the inference engine behind their own production serving. 18.8k stars in 6 days. the README charts are the part worth reading
decode on DeepSeek-V4.1-Flash, 64 GPUs, 128k context, 3 MTP draft tokens at 75% acceptance:
8 req per rank: ~325 tok/s per user, ~2.6k tok/s per GPU
64: ~245 per user, ~15.7k per GPU
256: ~100 per user, ~26k per GPU
~10x more tokens per GPU for ~3x slower per user. that one curve is the whole cost vs speed decision
who it's for: teams running deepseek at scale on SM100 (blackwell) or ascend 950. not your 4090
what's not in it: the KV cache system, prefill/decode disaggregation and DP load balancing. so you get the engine, not the full service
the bit i'd steal: batch-invariant inference + deterministic sampling. same tokens no matter who else is in the batch. that's huge for evals and RL
it started from vLLM, so i expect a lot of this to flow back upstream
interview scenario (llm inference, cost edition)
your chatbot runs on a hosted LLM API. 50k requests/day, same traffic as last week.
each request: ~6k token system prompt + tools, ~500 token user turn, ~300 tokens out.
bill was ~$390/day. after tuesday's deploy it's ~$1,200/day. p50 TTFT also went 0.4s to 1.1s.
no model change, no price change, output length is the same.
what do you check first?
answer: check the prompt cache hit rate first (cached input tokens in the usage field). it went from ~92% to ~0%, so the 6k prefix gets billed at full price every call and prefilled again, which is also why TTFT went up.
caching only matches an exact prefix, so diff the deployed prompt. usually a timestamp, request id or user name landed at the top of the system prompt, or tool JSON is serialized in a random order.
fix: static stuff first, dynamic stuff last, deterministic tool order, and alert on cache hit rate, not just the bill