levi (@levidiamode) header
levi (@levidiamode) profile picture

365 days of GPU programming ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓░░░░░░░░░ 314/365

out of memory
Joined June 2025
Photo by levi (@levidiamode)Photo by levi (@levidiamode)Photo by levi (@levidiamode)Photo by levi (@levidiamode)Photo by levi (@levidiamode)Photo by levi (@levidiamode)Photo by levi (@levidiamode)Photo by levi (@levidiamode)Photo by levi (@levidiamode)Photo by levi (@levidiamode)
Day 83/365 of GPU Programming Looking at DeepSeek's Multi-Head Latent Attention today. The last part of the AMD challenge series is to optimize an MLA decode kernel for MI355X where the absorbed Q and compressed KV cache are given and your task is to do the attention computation. A resource that really helped internalize what MLA does was @rasbt's incredible visual guide to attention variants in LLMs (luckily he posted that last week!), which covers everything from MHA to GQA to MLA to SWA, et cetera. If there's one place to get a visual intuition for recent attention mechanisms, it's this blog post. @jbhuang0604's video on MQA, GQA,MLA and DSA was the best conceptual intro I found on the topic and progressively builds up the ideas from first principles. The Welch Labs analysis of MLA is a great watch as well. Beautiful visualization of the changes DeepSeek made for MLA. Tried out a few kernels once I had a basic understanding of MLA and I think I'm slowly getting more comfortable with at least analyzing kernels.
Day 82/365 of GPU Programming Taking a closer look at Mixture of Experts today, so I can write better MoE kernels. Specifically, to optimize an MXFP4 MoE fused kernel for the GPU Mode challenge. I haven't had much prior exposure to MoEs, so lots of new concepts I learned today. Luckily I found the best intro to MoEs thanks to @MaartenGr visual overview of the topic. I then watched @tatsu_hashimoto's amazing Stanford CS336 lecture on MoEs, which added deeper context around why MoEs are gaining popularity, FLOPs, OLMoE, infra complexity, routing functions (mindblown this works so well...), expert sizes, training objectives, top k routing and DeepSeek variations. Once I had a basic understanding I started playing around with the some AITER kernels but progress there is tbd. Also had a nice chat with @juscallmevyom (who was kind enough to reach out!) about the AMD kernels and the challenge of materialization overhead.
23
146
2
1,464
132,742
Day 321/365 of GPU Programming Another interesting TPU resource is this talk on "10 lessons from three generations of TPUs and how they shaped TPU v4i" (based on the corresponding Google paper) from ETH Zurich's Seminar in Computer Architecture in 2022. I thought it was a really insightful look into what goes on behind the scenes of the design process of a new TPU architecture.
Day 320/365 of GPU Programming Another really good talk on TPUs six years later with a focus on TPU v4 (this was around the time v5 was announced but not available yet). The talk goes through rack architecture, optical circuit switches, reliability, slice availability, SparseCore, boards, pods, and more topics.
2
4
56
1,287
Day 320/365 of GPU Programming Another really good talk on TPUs six years later with a focus on TPU v4 (this was around the time v5 was announced but not available yet). The talk goes through rack architecture, optical circuit switches, reliability, slice availability, SparseCore, boards, pods, and more topics.
Day 319/365 of GPU Programming There's this wonderful talk from 9 years ago (a year after the TPU was introduced) where legendary computer scientist and former Google fellow Dave Patterson talks about the motivation behind Tensor Processing Units. Super fascinating to hear about inference workloads from back then and the start of what he calls a "new architecture renaissance".
2
7
1
51
2,468
Day 319/365 of GPU Programming There's this wonderful talk from 9 years ago (a year after the TPU was introduced) where legendary computer scientist and former Google fellow Dave Patterson talks about the motivation behind Tensor Processing Units. Super fascinating to hear about inference workloads from back then and the start of what he calls a "new architecture renaissance".
Day 318/365 of GPU Programming Going through the final lectures of the efficient ML course. Today's lecture being about distributed training. It's a nice review of topics that were also covered in CS336, e.g. ZeRO, FSDP and various forms of parallelisms across the stack. I really liked the reference list at the end which has a bunch of older papers I wasn't aware of.
4
6
2
123
3,776
Day 318/365 of GPU Programming Going through the final lectures of the efficient ML course. Today's lecture being about distributed training. It's a nice review of topics that were also covered in CS336, e.g. ZeRO, FSDP and various forms of parallelisms across the stack. I really liked the reference list at the end which has a bunch of older papers I wasn't aware of.
Day 317/365 of GPU Programming Just found out today the vLLM/inferact folks released a collection of megakernels for TPUs back in September which is a really interesting repo to scroll through. Also one fascinating thing about TPUs is how topologies affect kernel implementation (e.g. in this case, a 2×2×4 TPU chip topology)
2
6
1
69
4,374
Day 317/365 of GPU Programming Just found out today the vLLM/inferact folks released a collection of megakernels for TPUs back in September which is a really interesting repo to scroll through. Also one fascinating thing about TPUs is how topologies affect kernel implementation (e.g. in this case, a 2×2×4 TPU chip topology)
Day 316/365 of GPU Programming Learning more about Pallas kernels on TPU SparseCore today. As someone not super familiar with the JAX ecosystem and only vaguely familiar with TPU architecture, it's interesting to learn about the differences between ecosystems. Especially now as we're seeing greater acceleration in "time to onboard new silicon" as Anthropic put it, we'll probably see a much larger heterogeneity in chips/programming models.
4
6
1
59
3,155
Day 317/365 of GPU Programming Spending some time going through the Pallas docs today. Docs are probably my favorite way to familiarize with a topic that doesn't have too many resources online. I like how the Jax team also has a section on creating a "snapshot of some of the earlier design decisions made" to understand how Pallas came to be.
Day 316/365 of GPU Programming Learning more about Pallas kernels on TPU SparseCore today. As someone not super familiar with the JAX ecosystem and only vaguely familiar with TPU architecture, it's interesting to learn about the differences between ecosystems. Especially now as we're seeing greater acceleration in "time to onboard new silicon" as Anthropic put it, we'll probably see a much larger heterogeneity in chips/programming models.
1
4
59
1,437
Last Year We Built Megakernels. This Year We Deleted Them. One year ago, we helped introduce some of the first AI megakernels to the world. Since then, we’ve enjoyed seeing their wide adoption from friends and researchers. However, while whole model megakernels are a helpful abstraction, we’ve found that they overcomplicate and are not always an ideal optimization target. Below, we discuss our journey to understand and push past these limitations. We start with a deep dive into two contrasting paradigms of ML optimization – Torch Compile and Megakernels
Two weeks ago we announced Together Compiler: agents that tune inference to your model, chip, and traffic, for up to 2.6x more throughput. Today, we’re sharing the journey behind it: the experiments that challenged our assumptions of megakernels, the surprising strengths of Torch Compile, and the lessons that shaped our development of a next-generation system for inference optimization. Shout out to @dylan__lim @reyna_abhyankar @simran_s_arora and the team! Read more: together.ai/blog/road-to-tog…
38
71
20
745
78,610
I have a counterexample on TPU v7x. In a matmul kernel, issuing matmuls in bursts triggers power-management throttling, which in turn stalls weight delivery. Spacing out the matmuls yields a 25.9% speedup over an issue-when-ready greedy scheduler and 26.4% over vanilla Pallas.
Anyone who worked with wide instruction bundles / static scheduling on TPU, Hexagon, or AMD AI Engine — curious how close a greedy scheduler gets to optimal on your machines. I measured this on Elbrus (classic VLIW). Greedy is already optimal in 94% of cases, 0.51% cycle loss. Small neural net recovers most of the rest and never makes it worse. CLI that shows greedy vs exact optimum vs model on any .s file: github.com/smworklair/NEX-CL… @PatrickToulme @NormJouppi @adityazero_ @AminVahdat @JeffDean @Qualcomm @QCOMResearch @AMDembedded
11
15
4
241
15,158
Day 316/365 of GPU Programming Learning more about Pallas kernels on TPU SparseCore today. As someone not super familiar with the JAX ecosystem and only vaguely familiar with TPU architecture, it's interesting to learn about the differences between ecosystems. Especially now as we're seeing greater acceleration in "time to onboard new silicon" as Anthropic put it, we'll probably see a much larger heterogeneity in chips/programming models.
Day 315/365 of GPU Programming Onto lecture 18 which is part 2 of the series on finding efficiency gains in diffusion models. This is almost the end of the course (two more lectures on distributed training and some guest lectures before final projects), so my conclusion is this course has been really helpful and enjoyable. One interesting facet of the course is that lectures often center a around the focus of the research group itself (MIT's Han Lab) with regular throwbacks to "older" research that came out of the lab but I honestly really enjoyed that part as well since it feels like you get a pretty opinionated look into efficient ML research.
3
6
1
109
5,068
Day 315/365 of GPU Programming Onto lecture 18 which is part 2 of the series on finding efficiency gains in diffusion models. This is almost the end of the course (two more lectures on distributed training and some guest lectures before final projects), so my conclusion is this course has been really helpful and enjoyable. One interesting facet of the course is that lectures often center a around the focus of the research group itself (MIT's Han Lab) with regular throwbacks to "older" research that came out of the lab but I honestly really enjoyed that part as well since it feels like you get a pretty opinionated look into efficient ML research.
Day 314/365 of GPU Programming Spending the weekend catching up on some lectures. Looks like the next section of the efficientml.ai course is about diffusion models and how to make them run efficiently. I don't know anything about the topic, so should be a fun one to explore further.
2
7
1
77
4,142
Day 314/365 of GPU Programming Spending the weekend catching up on some lectures. Looks like the next section of the efficientml.ai course is about diffusion models and how to make them run efficiently. I don't know anything about the topic, so should be a fun one to explore further.
Day 313/365 of GPU Programming Reading up on some of the great posts on the LMSys blog. Especially interested in their recent megakernel article which delineates why SGLang has been moving towards larger GPU kernels. In it, they also describe what their kernels will evolve to (with a nod to the flappy folks) with a virtual machine “that takes control of the GPU and schedules different workloads across it”.
1
6
1
109
4,781