levi retweeted
Last Year We Built Megakernels. This Year We Deleted Them.
One year ago, we helped introduce some of the first AI megakernels to the world. Since then, we’ve enjoyed seeing their wide adoption from friends and researchers. However, while whole model megakernels are a helpful abstraction, we’ve found that they overcomplicate and are not always an ideal optimization target.
Below, we discuss our journey to understand and push past these limitations. We start with a deep dive into two contrasting paradigms of ML optimization – Torch Compile and Megakernels
Two weeks ago we announced Together Compiler: agents that tune inference to your model, chip, and traffic, for up to 2.6x more throughput.
Today, we’re sharing the journey behind it: the experiments that challenged our assumptions of megakernels, the surprising strengths of Torch Compile, and the lessons that shaped our development of a next-generation system for inference optimization.
Shout out to @dylan__lim @reyna_abhyankar @simran_s_arora and the team!
Read more: together.ai/blog/road-to-tog…
levi retweeted
I have a counterexample on TPU v7x. In a matmul kernel, issuing matmuls in bursts triggers power-management throttling, which in turn stalls weight delivery. Spacing out the matmuls yields a 25.9% speedup over an issue-when-ready greedy scheduler and 26.4% over vanilla Pallas.
Anyone who worked with wide instruction bundles / static scheduling on TPU, Hexagon, or AMD AI Engine — curious how close a greedy scheduler gets to optimal on your machines.
I measured this on Elbrus (classic VLIW). Greedy is already optimal in 94% of cases, 0.51% cycle loss. Small neural net recovers most of the rest and never makes it worse.
CLI that shows greedy vs exact optimum vs model on any .s file:
github.com/smworklair/NEX-CL…
@PatrickToulme @NormJouppi @adityazero_ @AminVahdat @JeffDean @Qualcomm @QCOMResearch @AMDembedded
levi retweeted
Spent half a year of spare time building HuggingArch — a "compiler" that doesn't run tensors, but computes FLOPs, memory traffic & KV cache for any HuggingFace model.
Agent writes the spec → sympy evaluates → one click gives theoretical perf across GPUs & parallel configs.
No GPU. No weights. No runtime.
web: huggingarch.com
blog: shenhan.cc/blog/huggingarch.…
github: github.com/shenh10/HuggingAr…
Star ⭐ & come build it with me!
levi retweeted
CUDA 13.5 is a very interesting update for GPU nerds!
Excited to see what people will build using the new locality domain APIs.
levi retweeted
Introducing the Together Compiler: continual compilation for AI inference! I’m super proud to finally share this work from our research team! ✨
Today’s inference is “Polymorphic” or shape-shifting, it doesn’t sit still. On OpenRouter, 80% of traffic goes to models released in the last three months and the top ten most popular open models each use a different architecture. Our customers constantly change their usage behaviors. Together Compiler leverages agents to continually optimize our deployments immediately as these changes occur.
Together Compiler research highlights include:
1. 252x better throughput: TKCC lets agents try 252x more optimization experiments per hour compared to prior agent inference optimization systems.
2. State-of-the-art TPS across InferenceX workloads we tried: TKCC outperforms the best InferenceX engine across all tested settings in just 24 hours & 2 nodes per compile. Including 1111 TPS on MiniMax M3.1 B200 and 1205 TPS on DeepSeek v4.1 Flash GB300 at batch size 1. NVIDIA, AMD, Trainium2.
3. Continuous recompilation: TKCC currently serves many trillions of production tokens per day and adapts to traffic shifts within just a few hours of recompilation.
4. Bitter lessons: TKCC’s design builds on 2+ years of building frameworks to accelerate AI Inference, starting from ThunderKittens and Megakernels. Compilers have always used abstractions and IRs, but we show that agentic compilers now reach higher performance ceilings without abstractions.
From an incredible team @reyna_abhyankar @dylan__lim @realDanFu @togethercompute!!
levi retweeted
We invented a recursive megakernel generator called Hyperkernel, the most complete solution to megakernels and GPU efficiency.
Hyperkernel-generated megakernels achieve state of the art performance on 11 different domains: inference (DeepSeek, Kimi and GLM), pretraining (Qwen), hardware fault recovery, robotics (π₀-FAST, ACT) diffusion and real time world models (Matrix-Game), full self driving (OpenPilot), voice models (Moshi), genomics (Evo 2) and protein folding (ESMFold).
All of this in a single kernel launch.
For context, Hyperkernel generated fully optimized megakernels for robotics, genomics and voice in under 50 minutes, and doubled the efficiency for >5 models generally using only about 60-100 lines of code per megakernel. This speaks to the unparalleled standard for code quality in the automated generations.
Our first release today open sources the megakernels themselves. Over the next couple weeks, we will release the entire Hyperkernel stack, along with the complete serving stack for Kimi and GLM (20-30% inference efficiency gains across all concurrencies).
Out of the box GPU machinery performs terribly for unorthodox research. We built Hyperkernel only 20 days ago for internal use; but given its multi-modality, we have decided to open source it.
Creative research should not be bottlenecked by the lack of dedicated infra teams. Hyperkernels offer a one stop shop infra stack for all research. We would love for the open source community to add to it :)
GitHub repo: github.com/Hyperstitioncc/Hy…
NOTE: We primarily use NVIDIA B300s, but our kernel generation extends to other hardware (B200s, AMD, TPUs).
1/9
We absolutely crush NVIDIA and achieve the best low latency inference for DeepSeek v4.1 Flash.
Compared with NVIDIA’s published vLLM results, Hyperkernel delivers:
• 36.7% higher per-user generation speed at concurrency 1.
• 28.2% higher per-user speed while serving two users, versus NVIDIA serving one.
• 15% higher per user speed while serving four users, versus NVIDIA serving one.
• 90.6% higher per-user generation speed at concurrency 4, compared with the highest published C4 AgentX interactivity for TP4 B300s.
• 28.5% higher per-user generation speed at concurrency 8, compared with NVIDIA's published result.
• 36% faster median per-request decode speed.
levi retweeted
📖 The Ultimate MoE Training Handbook 📖
We spent 1/2M GPUh to optimize MoE training and wrote the most comprehensive blog about it. With 30k words and 50+ interactive figures it covers everything from profiling to large cluster training.
huggingface.co/spaces/huggin…
levi retweeted
Hi! this is Rohan Anil, I develop training algorithms among other things - my work trains various frontier models.
At CoreAuto, I was fascinated by the problem of training on compute islands, eventually in space training.
I hope you all enjoy this write up!🪐🛰️
coreauto.com/hitchhikers-gui…
levi retweeted
We’ve developed the world’s most complete megakernel framework. It can perform entire training runs, serve inference, play video games, and even do all three simultaneously in a single kernel.
levi retweeted
how much can models remember?
there’s a neat perspective on neural networks that deserves to be more widely known
here’s how you could have invented every layer of the transformer from memory
(1/n) from my new blog post!
first part of a blogseries to understand tensor core arithmetic better.
simonveitner.substack.com/p/…
levi retweeted
What is not covered in the report is how we scaled model training to be efficient on 768 B200 GPUs.
I explain it in detail here: aleph-alpha.com/en/blog/scal…
A how to guide to scale up model training without wasting your GPU hours. Even if you only have 1 or 2 GPUs and cannot scale, our optimisation philosophy can still make your GPUs go brrr.
I couldn't explain ALL the details of our method in the blogpost, since the big launch came after. You can connect the dots now, the 30B MoE architecture was in fact Kolibri Origin all along.
levi retweeted
New historic NanoGPT record at 39.9s (-27.7s) from @DevenPzak , obliterating the prior record of 67.6s!
This record introduces a new paradigm of thinking to NanoGPT: instead of optimizing matmuls or adding more expressive operations, optimize at the individual flop level with incredibly clever engineering and ML judgement. If a flop is low value on a particular step, skip it.
Specifically:
-(~8s) Sampled softmax. If a token doesn’t appear in a batch, skip its lm_head fwd/bwd some fraction of the time.
-Sparse values. Only run an optimizer step for ngram embeddings that occurred in the batch. Set beta1 to zero to enable this. Beta2 is applied retroactively when the row is later used.
-Sparse updates. Only update ngram and value embeddings once every 4 steps instead of once every 2.
-Sparse communication. Shard the n-gram table across GPUs, and only pass the rows receiving updates on each step.
-Sparse optimizer states. For the n-gram table, reduce from 2 floats in Adam optimizer per param, to 1 float per 768 params.
-Hand-rolled flash attention for 64 dim heads.
There are several additions that add accuracy too:
-(~4s) EMA during last 300 steps, combined with lifting final_lr to 0.3 instead of 0.15.
-(~1s) A new optimizer, Anvil2, which expands muon via a second tracked momentum buffer, improves the ortho coefficients, and modifies the cautious weight decay application.
-A couple additional dynamic skip connections in the network.
The most striking consequence of the ‘flop aware paradigm’ is you can grow parameters arbitrarily large,
only limited by the available memory, since you can selectively choose how to expend flops on those parameters on each step. NanoGPT has kept active parameters below 124M, but total is unbounded, and has grown to 640M through embedding sparsity over the last year. This PR takes that to its logical conclusion on the 8xH100, scaling up to 65B sparse embedding parameters, which accounts for 25% of the PR’s gains. At frontier scale, where one is not bounded by an 8xH100, one could imagine where this paradigm could lead.
github.com/KellerJordan/modd…
As this was a very notable PR, I spoke with Deven for an hour to learn how he did it. Here’s his story on the changes: hyperstition.cc/training-nan…
levi retweeted
Have been busy with work, but finally I find the time to finish the blogpost on @GPU_MODE @coreauto's QR competition that happened 3 months ago.
I didn't win the 1st prize, but my panel QR kernel is definitely the fastest (by a large margin)
gau-nernst.github.io/b200-qr…
levi retweeted
After this, people kept asking us how @modal is able to serve agent inference so well.
So we wrote it all down.
New blog: "How to serve trillions of tokens for trillion-parameter coding agents".
modal.com/blog/trillion-toke…
We trained a trillion-parameter model on data from our own physics labs. There's a lot of training infra hidden in that sentence :)
Long scientific traces really put memory and parallelism to the test. Really happy we hit SOTA in memory efficiency and long-context training performance, so we can try more ideas, faster.
Some of what went into Neon at periodic.com/news/ai-infrast…
We built high-throughput materials labs in Menlo Park to create a loop between experiments and models. The labs generate fresh data, the models learn from it, and then help us decide what to try next.
Using only 1,300 H200s, plus months of our experimental data, we mid-trained and RL’d an open-source model to surpass GPT-6 Astra on our analysis benchmark. We call it Neon.
This is real footage from our lab. We’re focusing first on hard problems in materials science, including superconductors, magnets, and semiconductor materials.
Read our blog posts below.
levi retweeted
Over 15,000 submissions to One Layer Deeper. Nobody solved the problem as intended, but there are some interesting ideas for learning reusable operations and composing them into deeper computations
coreauto.com/blog/what-the-m…
levi retweeted
in the blog, we start with our original b200 gemms and progressively add rubin’s new features — wider mma steps, more tensor + shared memory, b-side collectors, early a release, and more
these kernels are still early. lut gemms, hardware-native megakernels, and new engine optimizations are next
huge thanks to @nvidia for the early access + support
levi retweeted
⚡️ We are excited to share updates from KDA-v0.5 (kernel design agents). Our latest Cute-DSL kernels can beat the human winners at FlashInfer Kernel Contest by 16%~69%! An amazing progress in just 3 months!
- GDN prefill: 1.69x Speedup over Kachua
- DSA attention: 1.41x Speedup over Dogacel
- FP8 MoE: 1.17x Speedup over Team Wombat
Humanize2 Flame Chase: docs.humanfia.ai/humanize2/f…
Results and Reproduction: github.com/mit-han-lab/mlsys…
MLSys 2026 Contest: mlsys26.flashinfer.ai/
KDA-v0.5 achieved this by integrating the Cute-DSL primitive, better workflows (humanize1 -> humanize2), updated kernel-wiki (self-evolved), and better profiling skills (IKET). The results are achieved by the flame chase flow from humanize2 – using gpt-5.6-sol and fable-5 to iteratively optimize . We have released the kernels for validation and more details will come soon!
By the KDA Team: Dongyun Zou, Yixin Dong, Junxian Guo, Changye Li, Yahui Cui, Zihao Ye, Junru Shao, Zijian Zhang, Sihao Liu, Song Bian and Ligeng Zhu.



































