levi (@levidiamode) header
levi (@levidiamode) profile picture

365 days of GPU programming ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓░░░░░░░░░ 314/365

out of memory
Joined June 2025
Photo by levi (@levidiamode)Photo by levi (@levidiamode)Photo by levi (@levidiamode)Photo by levi (@levidiamode)Photo by levi (@levidiamode)Photo by levi (@levidiamode)Photo by levi (@levidiamode)Photo by levi (@levidiamode)Photo by levi (@levidiamode)Photo by levi (@levidiamode)
Last Year We Built Megakernels. This Year We Deleted Them. One year ago, we helped introduce some of the first AI megakernels to the world. Since then, we’ve enjoyed seeing their wide adoption from friends and researchers. However, while whole model megakernels are a helpful abstraction, we’ve found that they overcomplicate and are not always an ideal optimization target. Below, we discuss our journey to understand and push past these limitations. We start with a deep dive into two contrasting paradigms of ML optimization – Torch Compile and Megakernels
Two weeks ago we announced Together Compiler: agents that tune inference to your model, chip, and traffic, for up to 2.6x more throughput. Today, we’re sharing the journey behind it: the experiments that challenged our assumptions of megakernels, the surprising strengths of Torch Compile, and the lessons that shaped our development of a next-generation system for inference optimization. Shout out to @dylan__lim @reyna_abhyankar @simran_s_arora and the team! Read more: together.ai/blog/road-to-tog…
38
71
20
745
78,611
I have a counterexample on TPU v7x. In a matmul kernel, issuing matmuls in bursts triggers power-management throttling, which in turn stalls weight delivery. Spacing out the matmuls yields a 25.9% speedup over an issue-when-ready greedy scheduler and 26.4% over vanilla Pallas.
Anyone who worked with wide instruction bundles / static scheduling on TPU, Hexagon, or AMD AI Engine — curious how close a greedy scheduler gets to optimal on your machines. I measured this on Elbrus (classic VLIW). Greedy is already optimal in 94% of cases, 0.51% cycle loss. Small neural net recovers most of the rest and never makes it worse. CLI that shows greedy vs exact optimum vs model on any .s file: github.com/smworklair/NEX-CL… @PatrickToulme @NormJouppi @adityazero_ @AminVahdat @JeffDean @Qualcomm @QCOMResearch @AMDembedded
11
15
4
241
15,158
Spent half a year of spare time building HuggingArch — a "compiler" that doesn't run tensors, but computes FLOPs, memory traffic & KV cache for any HuggingFace model. Agent writes the spec → sympy evaluates → one click gives theoretical perf across GPUs & parallel configs. No GPU. No weights. No runtime. web: huggingarch.com blog: shenhan.cc/blog/huggingarch.… github: github.com/shenh10/HuggingAr… Star ⭐ & come build it with me!
6
29
2
180
14,937
CUDA 13.5 is a very interesting update for GPU nerds! Excited to see what people will build using the new locality domain APIs.
4
6
93
6,303
Introducing the Together Compiler: continual compilation for AI inference! I’m super proud to finally share this work from our research team! ✨ Today’s inference is “Polymorphic” or shape-shifting, it doesn’t sit still. On OpenRouter, 80% of traffic goes to models released in the last three months and the top ten most popular open models each use a different architecture. Our customers constantly change their usage behaviors. Together Compiler leverages agents to continually optimize our deployments immediately as these changes occur. Together Compiler research highlights include: 1. 252x better throughput: TKCC lets agents try 252x more optimization experiments per hour compared to prior agent inference optimization systems. 2. State-of-the-art TPS across InferenceX workloads we tried: TKCC outperforms the best InferenceX engine across all tested settings in just 24 hours & 2 nodes per compile. Including 1111 TPS on MiniMax M3.1 B200 and 1205 TPS on DeepSeek v4.1 Flash GB300 at batch size 1. NVIDIA, AMD, Trainium2. 3. Continuous recompilation: TKCC currently serves many trillions of production tokens per day and adapts to traffic shifts within just a few hours of recompilation. 4. Bitter lessons: TKCC’s design builds on 2+ years of building frameworks to accelerate AI Inference, starting from ThunderKittens and Megakernels. Compilers have always used abstractions and IRs, but we show that agentic compilers now reach higher performance ceilings without abstractions. From an incredible team @reyna_abhyankar @dylan__lim @realDanFu @togethercompute!!
94
142
45
968
175,479
We invented a recursive megakernel generator called Hyperkernel, the most complete solution to megakernels and GPU efficiency. Hyperkernel-generated megakernels achieve state of the art performance on 11 different domains: inference (DeepSeek, Kimi and GLM), pretraining (Qwen), hardware fault recovery, robotics (π₀-FAST, ACT) diffusion and real time world models (Matrix-Game), full self driving (OpenPilot), voice models (Moshi), genomics (Evo 2) and protein folding (ESMFold). All of this in a single kernel launch. For context, Hyperkernel generated fully optimized megakernels for robotics, genomics and voice in under 50 minutes, and doubled the efficiency for >5 models generally using only about 60-100 lines of code per megakernel. This speaks to the unparalleled standard for code quality in the automated generations. Our first release today open sources the megakernels themselves. Over the next couple weeks, we will release the entire Hyperkernel stack, along with the complete serving stack for Kimi and GLM (20-30% inference efficiency gains across all concurrencies). Out of the box GPU machinery performs terribly for unorthodox research. We built Hyperkernel only 20 days ago for internal use; but given its multi-modality, we have decided to open source it. Creative research should not be bottlenecked by the lack of dedicated infra teams. Hyperkernels offer a one stop shop infra stack for all research. We would love for the open source community to add to it :) GitHub repo: github.com/Hyperstitioncc/Hy… NOTE: We primarily use NVIDIA B300s, but our kernel generation extends to other hardware (B200s, AMD, TPUs). 1/9 We absolutely crush NVIDIA and achieve the best low latency inference for DeepSeek v4.1 Flash. Compared with NVIDIA’s published vLLM results, Hyperkernel delivers: • 36.7% higher per-user generation speed at concurrency 1. • 28.2% higher per-user speed while serving two users, versus NVIDIA serving one. • 15% higher per user speed while serving four users, versus NVIDIA serving one. • 90.6% higher per-user generation speed at concurrency 4, compared with the highest published C4 AgentX interactivity for TP4 B300s. • 28.5% higher per-user generation speed at concurrency 8, compared with NVIDIA's published result. • 36% faster median per-request decode speed.
59
81
33
826
98,522
📖 The Ultimate MoE Training Handbook 📖 We spent 1/2M GPUh to optimize MoE training and wrote the most comprehensive blog about it. With 30k words and 50+ interactive figures it covers everything from profiling to large cluster training. huggingface.co/spaces/huggin…
33
244
20
1,595
112,695
Hi! this is Rohan Anil, I develop training algorithms among other things - my work trains various frontier models. At CoreAuto, I was fascinated by the problem of training on compute islands, eventually in space training. I hope you all enjoy this write up!🪐🛰️ coreauto.com/hitchhikers-gui…
66
137
39
1,580
193,891
We’ve developed the world’s most complete megakernel framework. It can perform entire training runs, serve inference, play video games, and even do all three simultaneously in a single kernel.
39
87
82
1,401
201,478
how much can models remember? there’s a neat perspective on neural networks that deserves to be more widely known here’s how you could have invented every layer of the transformer from memory (1/n) from my new blog post!
38
214
43
2,337
219,432
levi retweeted
first part of a blogseries to understand tensor core arithmetic better. simonveitner.substack.com/p/…
3
25
260
8,099
What is not covered in the report is how we scaled model training to be efficient on 768 B200 GPUs. I explain it in detail here: aleph-alpha.com/en/blog/scal… A how to guide to scale up model training without wasting your GPU hours. Even if you only have 1 or 2 GPUs and cannot scale, our optimisation philosophy can still make your GPUs go brrr. I couldn't explain ALL the details of our method in the blogpost, since the big launch came after. You can connect the dots now, the 30B MoE architecture was in fact Kolibri Origin all along.
15
76
3
738
177,263
New historic NanoGPT record at 39.9s (-27.7s) from @DevenPzak , obliterating the prior record of 67.6s! This record introduces a new paradigm of thinking to NanoGPT: instead of optimizing matmuls or adding more expressive operations, optimize at the individual flop level with incredibly clever engineering and ML judgement. If a flop is low value on a particular step, skip it. Specifically: -(~8s) Sampled softmax. If a token doesn’t appear in a batch, skip its lm_head fwd/bwd some fraction of the time. -Sparse values. Only run an optimizer step for ngram embeddings that occurred in the batch. Set beta1 to zero to enable this. Beta2 is applied retroactively when the row is later used. -Sparse updates. Only update ngram and value embeddings once every 4 steps instead of once every 2. -Sparse communication. Shard the n-gram table across GPUs, and only pass the rows receiving updates on each step. -Sparse optimizer states. For the n-gram table, reduce from 2 floats in Adam optimizer per param, to 1 float per 768 params. -Hand-rolled flash attention for 64 dim heads. There are several additions that add accuracy too: -(~4s) EMA during last 300 steps, combined with lifting final_lr to 0.3 instead of 0.15. -(~1s) A new optimizer, Anvil2, which expands muon via a second tracked momentum buffer, improves the ortho coefficients, and modifies the cautious weight decay application. -A couple additional dynamic skip connections in the network. The most striking consequence of the ‘flop aware paradigm’ is you can grow parameters arbitrarily large, only limited by the available memory, since you can selectively choose how to expend flops on those parameters on each step. NanoGPT has kept active parameters below 124M, but total is unbounded, and has grown to 640M through embedding sparsity over the last year. This PR takes that to its logical conclusion on the 8xH100, scaling up to 65B sparse embedding parameters, which accounts for 25% of the PR’s gains. At frontier scale, where one is not bounded by an 8xH100, one could imagine where this paradigm could lead. github.com/KellerJordan/modd… As this was a very notable PR, I spoke with Deven for an hour to learn how he did it. Here’s his story on the changes: hyperstition.cc/training-nan…
37
156
67
1,474
429,269
Have been busy with work, but finally I find the time to finish the blogpost on @GPU_MODE @coreauto's QR competition that happened 3 months ago. I didn't win the 1st prize, but my panel QR kernel is definitely the fastest (by a large margin) gau-nernst.github.io/b200-qr…
2
31
4
207
10,255
After this, people kept asking us how @modal is able to serve agent inference so well. So we wrote it all down. New blog: "How to serve trillions of tokens for trillion-parameter coding agents". modal.com/blog/trillion-toke…
found out we're serving 40% of Kimi K3 traffic on OpenRouter ... the guy working on it is tired. anyone wanna help us work on inference?
16
44
4
581
91,289
We trained a trillion-parameter model on data from our own physics labs. There's a lot of training infra hidden in that sentence :) Long scientific traces really put memory and parallelism to the test. Really happy we hit SOTA in memory efficiency and long-context training performance, so we can try more ideas, faster. Some of what went into Neon at periodic.com/news/ai-infrast…
We built high-throughput materials labs in Menlo Park to create a loop between experiments and models. The labs generate fresh data, the models learn from it, and then help us decide what to try next. Using only 1,300 H200s, plus months of our experimental data, we mid-trained and RL’d an open-source model to surpass GPT-6 Astra on our analysis benchmark. We call it Neon. This is real footage from our lab. We’re focusing first on hard problems in materials science, including superconductors, magnets, and semiconductor materials. Read our blog posts below.
8
16
4
172
78,336
i shed a tear reading this blog post by a deepseek kernel engineer
63
460
61
6,708
212,067
Over 15,000 submissions to One Layer Deeper. Nobody solved the problem as intended, but there are some interesting ideas for learning reusable operations and composing them into deeper computations coreauto.com/blog/what-the-m…
12
25
5
189
36,418
in the blog, we start with our original b200 gemms and progressively add rubin’s new features — wider mma steps, more tensor + shared memory, b-side collectors, early a release, and more these kernels are still early. lut gemms, hardware-native megakernels, and new engine optimizations are next huge thanks to @nvidia for the early access + support
1
2
15
1,675
⚡️ We are excited to share updates from KDA-v0.5 (kernel design agents). Our latest Cute-DSL kernels can beat the human winners at FlashInfer Kernel Contest by 16%~69%! An amazing progress in just 3 months! - GDN prefill: 1.69x Speedup over Kachua - DSA attention: 1.41x Speedup over Dogacel - FP8 MoE: 1.17x Speedup over Team Wombat Humanize2 Flame Chase: docs.humanfia.ai/humanize2/f… Results and Reproduction: github.com/mit-han-lab/mlsys… MLSys 2026 Contest: mlsys26.flashinfer.ai/ KDA-v0.5 achieved this by integrating the Cute-DSL primitive, better workflows (humanize1 -> humanize2), updated kernel-wiki (self-evolved), and better profiling skills (IKET). The results are achieved by the flame chase flow from humanize2 – using gpt-5.6-sol and fable-5 to iteratively optimize . We have released the kernels for validation and more details will come soon! By the KDA Team: Dongyun Zou, Yixin Dong, Junxian Guo, Changye Li, Yahui Cui, Zihao Ye, Junru Shao, Zijian Zhang, Sihao Liu, Song Bian and Ligeng Zhu.
2
36
4
219
27,179