Learning how systems run fast
NYC
Joined October 2014
- Tweets2.5K
- Following1K
- Followers2.9K
- Likes2.5K
Kimbo retweeted
感谢 xudong 推荐，如果我的一些心路历程能够帮助到大家，会非常开心~
非常感谢 @_TobiasLee 李磊分享这期播客。
他是小米 MiMo 大模型后训练的核心研究员，也是港大在读 PhD。一边参与大模型训练，一边做科研，聊的正好是很多读博的人天天在想的事：慢科研还是快科研、选学校还是选导师、内地香港还是美国、没论文没实习怎么冷启动，以及本科就能进 AI Labs 的话，读 PhD 还值不值。
播客的后面还顺带聊到了 RSI、Benchmark，以及研究怎么靠开源和社交媒体真正被看见。
正在读 PhD，或者准备读、纠结要不要读的，包括做科研的年轻人，很建议听这一期的博客。
xiaoyuzhoufm.com/episode/6ad…
I’m interested in MLSys, and I don’t limit myself on topics to learn, as long as it’s relevant to system performance
But I recently realized, deep down I still care about one aspect more than anything else
Kimbo retweeted
We ran ~1.3 billion sandboxes in our largest RL run for Beam - spanning RL training, evaluation, and ablations. With evals and other RL experiments running, the fleet peaked at 883,006 vCPUs (83% of allocatable) across 23 clusters on @googlecloud and @awscloud.
We plan to use OpenEnv (now part of @AgenticAIFdn) for future runs, bringing our learnings to the open ecosystem.
Kimbo retweeted
Alright I finally have more to share on my personal project!
Resin is a runtime for building fast LLM inference engines.
The idea is to make all the generic parts of inference reusable, and leave only the model-specific forward pass + kernels to be written for each architecture / hardware target.
To build an engine:
1. Import Resin's core.
2. Have an agent write a forward pass for the model you’re interested in, for your specific hardware (GPU or TPU or AMD), any kernels you want.
Then, boom, you have a standalone inference engine.
—
I've been hacking over the past month (early post: teapaw.org/ekzhang1/status/208950…), and recently this week I got a lot of features + parallelisms working, inspired by DeepInfer.
For DeepSeek-V4.1-Flash, it's nearly as fast as DeepInfer (within 5-10%), while being self-contained and supporting other models, plus metrics + HTTP server + prefix cache.
Yesterday I wrote a skill for authoring models and put Codex to work on it overnight. The agent built a GLM-5.3-Flash engine with MTP, reaching up to 60,000 input tok/s and 16,000 output tok/s on B200 (EP4).
So far inference "vibe coding" has not been very reliable or successful. I think it's because people start from scratch each time and reinvent lots of tricky things on the scheduler and server layers. But with a shared core, we can automate lots of inference work.
The tricky bit is writing a fast forward pass. But Resin turns that into a standalone engine by just importing a library, and it becomes a verifiable goal for the AI.
Code: github.com/ekzhang/resin
So… I have it working now. The core of an inference engine in Rust is about 3k lines (scheduling, kv/swa/recurrent state, prefix cache, multimodal, dynamic batching)
I then asked Codex to implement (with MTP, cudagraphs, TP on Blackwell):
- DeepSeekV4 Flash
- Gemma 3.8 27B
- GPT-OSS
- Inkling
It turns out that most models take a lot of code to run with fast, persistent graphable kernels and so on… on the order of 2.5k for a simple model (gpt-oss) and up to 7k for a monster like DeepSeekV4 Flash that has mHC, CSA, HCA and DSpark
Not sure if I’ll continue with this, but it was fun! The simple core is reusable, and startup times are crazy fast for an inference engine with no tech debt and not having to serve a ton of dynamic configurations - just one model forward pass in nvfp4 :’) - maybe one day SGLang/vLLM will be modular like this.
See interactivity-throughput below for Inkling-Small TP=1 after a few sconv kernel fusions. (This is without overlap scheduling btw! Don’t even really need it when your core + scheduler is small and Rust.)
Kimbo retweeted
Tinker changed how people do post-training. We're big fans, and so are many of the teams we work with. But as those teams scale up, they keep raising the same two problems: per-token pricing gets expensive quickly, and you’re limited to the models and features the provider decides to support.
We think a good API deserves more than one implementation.
Today we're releasing Spindle, an open-source, Tinker-compatible training server that runs in your Modal workspace. Your existing Tinker scripts run unchanged, and because you own the server, you pay for GPU time instead of tokens.
Listening to the pod got me thinking: How can I model the US vs China AI race dynamic from a game theoretic view? My attempt:
It’s an asymmetric, repeated leader-follower dynamic game
- Each “round of game” is a model release
- A “state” is the model capability gap
- An “outcome” is the capability and safety effects
- Labs in the US and China are “players”, US labs being leader, Chinese labs being follower
- Each player has a set of “actions” (scale compute, pause training, open source technology, etc) and responds to other players’ actions
The cooperative outcome is all players improve capabilities together safely. The bad outcome is all players race to improve capabilities, sacrificing safety.
Verification, eg capability evals, training run audits, compute allocation restrictions, is a good way to enforce cooperation. The historical analogy is the US-Soviet nuclear arms control. Designing an effective verification system will become one of the most important tasks.
Fantastic conversation - Joe asks tough questions! Did we answer them well?
I think this might be the first appearance of @the_marwell in public - he is a complete legend within Anthropic, an unbelievable amount of critical programs simply would not have progressed without him. Getting to know him via this podcast is crucial lore.
Kimbo retweeted
We’ve developed the world’s most complete megakernel framework. It can perform entire training runs, serve inference, play video games, and even do all three simultaneously in a single kernel.
Kimbo retweeted
I've been a fan of DPPO for a while. PI has been running it as the default policy-gradient method for a while, but now they're switching to "IPO".
so what's changed?
1/n
Kimbo retweeted
DistMoE is one the most elegant and capable kernels I have ever seen. It supports pretraining, post training, and even inference (if people wanna try) in a unified way with extreme performance and strict numerics. Try it even if you are in frontier labs! meta-pytorch.org/dist_moe/pr…
Kimbo retweeted
🚀 Introducing DeepInfer, our in-house inference framework, now fully open source!
Powering DeepSeek-V4.1-Flash at scale with high throughput and low latency. We’re sharing our production stack to help the community lower compute costs, reduce energy use, and make AI more affordable.
🔗 github.com/deepseek-ai/DeepI…
1/4
Kimbo retweeted
INTELLECT Policy Optimization (IPO)
In async RL, trainer and inference never agree exactly on a token's probability. IcePop masks on the ratio, which misjudges both tails
IPO masks on the probability difference instead
Megakernels are cool
BREAKING NEWS🚨: @Inferact & Google submit their FIRST preliminary TPUv7 Kimi K3 2.8T Model Submission, and it is already delivering strong perf per TCO against GB200 NVL72, B300, and MI355 in the ultra-high-interactivity regime due to their megakernel approach for external customers.
One of the most important aspects of an external compute platform is a strong community, and what makes this submission special is that Inferact (Makers of vLLM) led the submission end to end using the TorchTPU vLLM stack.👇️(1/6)🧵
Kimbo retweeted
We got our hands on @nvidia's new Vera CPU and put it to the test.
We built a benchmark using real-world agentic workloads to see how it performs under contention.
In early tests, Vera delivered up to 2x the throughput of comparable x86 hardware, maintaining low latency even as hundreds of concurrent workloads competed for resources.
For Modal Sandboxes, that could mean running more agents in parallel without sacrificing latency.
Kimbo retweeted
The secret to Modal's speed is our custom lazily loaded container filesystem, ImageFS. ImageFS is built on a massive tiered distributed filesystem, layering local NVMe disks, regional and p2p caches, and global blob stores.
When we launched VM Sandboxes this month, we wanted to make sure they could take advantage of ImageFS in the same way our gVisor containers do. To do that, we built out a new filesystem stack that tricks a VM's ext4 driver into thinking it's talking to a normal disk.
Kimbo retweeted
The evolution of interleaved SWA and global layers with window size 512 happened due to merging Routing Transformer (github.com/google-research/g…) with scan layers - XLA likes repeated blocks to reduce compilation time. Doing top-k sparse attention proved to be harder on TPUs and most of the successful sparsification has happened on GPUs (deepseek, minimax etc).
Kimbo retweeted
wow, amazing technical blog post about microsoft's new 3T pre-training with very rare and clear insights on how to train models of this size
first let's get an overview of the shape of this model, it's a sparser model than Kimi K3 (80B active vs 104B), same sliding window hybrid pattern, with this time more kv heads for the sliding window layers (a bit different from stepfun and poolside that add query head). they added a shared expert and also removed the interleaved MoE/Dense pattern of MAI 1
the scaling ladder is very similar to MAI 1 with the same mechanism, each rung depends only on the number of layers. they mention that one of the key improvements is how they handle the smaller rung data distribution where they make sure to have the same repetition as the final corpus. they already did that for MAI-1 but only for data ablation iirc
their prediction is almost identical to the final run loss, very impressive
they compare the model with different base models such as deepseek v4 pro and MAI 1, tho in the case of deepseek v4 pro i'd say it's quite hard to distinguish how much the data mixture "overfitting" to the scaling ladder has an impact here (they mention it's both), need to always be a bit careful about EG metrics for data in general
they have VERY NICE scaling laws for weight decay and learning rate, for the learning rate they express it as a function of tokens per parameter (N/D) and steps (hence depending on the batch size). they also do the same for dropout where it's a function of active params only, they also find the usual sqrt(k) scaling for learning rate and batch size (other scaling laws often also converge to that)
a bunch of other stability stuff, they use Muon this time and activation gradient clipping, the efficiency gain here is actually pretty crazy at larger scale. they also increase the training horizon to ~40T text (20T unique) + a new 10T (4.5T unique) multimodal since the model is natively multimodal
imo this is a safe but very clean scaling! very exciting for the tech report and MAI-base-3, congrats!!
We now have state-of-the-art pre-trained base models in the 3T-parameter weight class, built entirely in-house at MAI.
Post-training results coming soon -- stay tuned!
Lots of details in our technical blog post.
microsoft.ai/news/frontier-p…
Kimbo retweeted
Exciting news from Datadog AI Research: we trained a model to investigate production incidents, and it matches the best frontier models at a fraction of the cost.
Introducing Scout 1.0, the model powering Bits Investigation, our AI SRE.
On AlertEval, our benchmark of real production alerts:
- Scout 1.0 medium matches Claude Opus 5.5 high and GPT-6.1 Sol high at 3-8x lower cost
- It also performs comparably to Claude Fable 5.1 medium and GPT-6 Astra medium at 12x and 19x lower cost
- Scout 1.0 xhigh matches Claude Fable 5.1 xhigh at 4x lower cost
If we’re already tolerating off-policyness, why not double down and use different hardware with even more differences
Curious if there will be hardware performance/reliability-induced biases
🐐 @nanjiangwill
Training large models with RL typically runs on a fixed homogeneous cluster of the latest accelerators for numerical stability and faster throughput. This throws away inference optimization work done for other vendors and generations, limits the capacity you can utilize, and is unnecessarily expensive.
So we trained a model on agentic coding tasks using fully asynchronous RL and mixed-precision, mixed-accelerator samplers. With off-policy correction, training stayed stable through convergence, on par with homogeneous samplers, at a lower cost.