Ivan Zhou retweeted
Space Bunny is here! 🐰 MiniMax M3.1 Flash hits >1K tok/s, bs1 decode with SGLang, production-ready from Day 0. This integration is a joint effort by @MiniMax_AI, @NVIDIAAI, @AMD, and @radixark, and you can also run RL with Miles!
-Broad hardware support: NVIDIA Blackwell and Hopper (GB300/B300/B200/H200), plus AMD Instinct MI350X and MI355X.
-Agentic serving at scale: ~100K total tok/s per GPU on the AgentX coding-agent trace with 96 concurrent sessions.
-Speculative decoding: Supports MiniMax's DSpark draft and a DFlash2 draft trained by @radixark; supports HiCache and multimodal input.
-Ultra-low-latency inference: MegaKernels deliver 2.15× faster single-request decode.
MiniMax M3.1 Flash weights are now public. 428B parameters total, 23B activated per token.
Agentic performance: A generational leap in agentic tasks, coding, and cybersecurity — 76.2 on BankerToolBench, 73.7% on DeepSWE v1.1, 61.7 on OSWorld V2, 29.6 on Agents' Last Exam, and 82.2 on CyberGym.
Efficient by design: MiniMax Sparse Attention and DSpark speculative decoding improve inference efficiency.
Hugging Face:
huggingface.co/MiniMaxAI/Min…
ModelScope:
modelscope.cn/models/MiniMax…
API: platform.minimax.io
M Plan: minimax.io/m-planMiniMax
MCode for desktop: agent.minimax.io/download
Ivan Zhou retweeted
From inference to RL, ready from day one. 🐰
Thanks to the teams behind SGLang, and Miles @sgl_project alongside @NVIDIAAI and @AMD, for supporting MiniMax M3.1 Flash. Great building together!
Space Bunny is here! 🐰 MiniMax M3.1 Flash hits >1K tok/s, bs1 decode with SGLang, production-ready from Day 0. This integration is a joint effort by @MiniMax_AI, @NVIDIAAI, @AMD, and @radixark, and you can also run RL with Miles!
-Broad hardware support: NVIDIA Blackwell and Hopper (GB300/B300/B200/H200), plus AMD Instinct MI350X and MI355X.
-Agentic serving at scale: ~100K total tok/s per GPU on the AgentX coding-agent trace with 96 concurrent sessions.
-Speculative decoding: Supports MiniMax's DSpark draft and a DFlash2 draft trained by @radixark; supports HiCache and multimodal input.
-Ultra-low-latency inference: MegaKernels deliver 2.15× faster single-request decode.
Ivan Zhou retweeted
Miles v0.1.3 is out.
This release brings lighter weight updates, a smaller training footprint, and more resilient runs.
Rebuilt P2P transfer cuts the first weight update from 99.5s to 3.9s and pinned memory per sender from 28 GiB to 2 GiB in our benchmark.
Fully async RL can now share GPUs between training and rollout, even on a single GPU. The new fused log-prob backend cuts peak GPU memory by about a third in our long-response test.
Failures stay contained. With fault tolerance, a failed cell no longer blocks weight updates to healthy cells. Fully async runs can also resume where they left off.
LoRA now runs in Megatron raw mode across model families using a unified implementation.
Full release notes 👇
Ivan Zhou retweeted
Meta is running SGLang in production on Google's TPUs. And the stack is open source
This is one of the things I love most about working in open-source AI infrastructure. SGLang-TorchTPU is the result of a remarkable collaboration between engineers at @Google, Meta's @PyTorch team, and our SGLang team at @radixark.
What's especially exciting is that we're bringing much more than basic model execution to TPU. Prefix caching, speculative decoding, KV cache offloading, disaggregated serving, and the rest of the machinery needed for production agentic workloads can now run through a PyTorch-native TPU backend. The custom MegaKernel also delivers a reported 770 tok/s decode for Muse Spark Open at batch size 1 on InferenceX 8K/1K.
We're also opening a community TPU capacity pool at @radixark, offering free compute to developers contributing to SGLang-TorchTPU. I hope this makes it easier for more open-source engineers to experiment with TPU kernels, model support, and serving optimizations.
Huge thanks to everyone at Google, Meta, RadixArk, and the SGLang community who made this happen. It's genuinely exciting to see what collaborative open-source engineering can accomplish!
Today we launch SGLang-TorchTPU, built with @Google TPU team and Meta @PyTorch team.
We’re excited to see it powering production inference at Meta!
SGLang-TorchTPU is a TPU plugin for SGLang that brings:
- Top open models on TPU: Muse Spark Open, GLM-5.3 Flash, Kimi K3, Ling 3.1 Flash, MiniMax H3
- SGLang’s production features for large agentic workloads
- Custom MegaKernel: Muse Spark Open hits 770 tok/s decode at bs=1 on InferenceX 8K/1K
SGLang-TorchTPU is available now: github.com/sgl-project/sglan…
Ivan Zhou retweeted
Genuinely so happy to see SGLang-TorchTPU launch! Since Ironwood has been announced, TPU ecosystem has grown at incredible speed. Thanks to all the amazing colleagues and collaborators who makes this happen. We'll keep pushing to bring out TPU's full potential through SGLang! ❤️
Today we launch SGLang-TorchTPU, built with @Google TPU team and Meta @PyTorch team.
We’re excited to see it powering production inference at Meta!
SGLang-TorchTPU is a TPU plugin for SGLang that brings:
- Top open models on TPU: Muse Spark Open, GLM-5.3 Flash, Kimi K3, Ling 3.1 Flash, MiniMax H3
- SGLang’s production features for large agentic workloads
- Custom MegaKernel: Muse Spark Open hits 770 tok/s decode at bs=1 on InferenceX 8K/1K
SGLang-TorchTPU is available now: github.com/sgl-project/sglan…
Ivan Zhou retweeted
SGLang-TorchTPU gives people options to choose the hardware that best fits their workloads and economics. We're excited to see many partners already running it in production!
Huge thanks to everyone at Google, Meta, our @sgl_project team, and community who made this happen!
We're committed to helping make TPU a first-class citizen in the SGLang ecosystem and will keep building toward a more flexible, heterogeneous future for AI infrastructure.
Today we launch SGLang-TorchTPU, built with @Google TPU team and Meta @PyTorch team.
We’re excited to see it powering production inference at Meta!
SGLang-TorchTPU is a TPU plugin for SGLang that brings:
- Top open models on TPU: Muse Spark Open, GLM-5.3 Flash, Kimi K3, Ling 3.1 Flash, MiniMax H3
- SGLang’s production features for large agentic workloads
- Custom MegaKernel: Muse Spark Open hits 770 tok/s decode at bs=1 on InferenceX 8K/1K
SGLang-TorchTPU is available now: github.com/sgl-project/sglan…
Ivan Zhou retweeted
Production ready SGLang-TorchTPU with megakernel integrations. It’s really built by the community, kudos to Google TPU team (@googlecloud @googledevs), Meta @PyTorch team, @AntLingAGI @neuro_spark_ai @MiniMax_AI! And more to come 🤫
Today we launch SGLang-TorchTPU, built with @Google TPU team and Meta @PyTorch team.
We’re excited to see it powering production inference at Meta!
SGLang-TorchTPU is a TPU plugin for SGLang that brings:
- Top open models on TPU: Muse Spark Open, GLM-5.3 Flash, Kimi K3, Ling 3.1 Flash, MiniMax H3
- SGLang’s production features for large agentic workloads
- Custom MegaKernel: Muse Spark Open hits 770 tok/s decode at bs=1 on InferenceX 8K/1K
SGLang-TorchTPU is available now: github.com/sgl-project/sglan…
Ivan Zhou retweeted
Welcome SGLang-TorchTPU to the family!
We're excited to offer a new path for developers to use familiar tools and stacks to support models on TPU, and serve them for agentic workloads!
Check out our write-up on how SGLang-TorchTPU works as a plugin for SGLang, its feature and model coverage, and our early exploration of training on TPU!
lmsys.org/blog/2026-10-20-sg…
Today we launch SGLang-TorchTPU, built with @Google TPU team and Meta @PyTorch team.
We’re excited to see it powering production inference at Meta!
SGLang-TorchTPU is a TPU plugin for SGLang that brings:
- Top open models on TPU: Muse Spark Open, GLM-5.3 Flash, Kimi K3, Ling 3.1 Flash, MiniMax H3
- SGLang’s production features for large agentic workloads
- Custom MegaKernel: Muse Spark Open hits 770 tok/s decode at bs=1 on InferenceX 8K/1K
SGLang-TorchTPU is available now: github.com/sgl-project/sglan…
Ivan Zhou retweeted
Today we launch SGLang-TorchTPU, built with @Google TPU team and Meta @PyTorch team.
We’re excited to see it powering production inference at Meta!
SGLang-TorchTPU is a TPU plugin for SGLang that brings:
- Top open models on TPU: Muse Spark Open, GLM-5.3 Flash, Kimi K3, Ling 3.1 Flash, MiniMax H3
- SGLang’s production features for large agentic workloads
- Custom MegaKernel: Muse Spark Open hits 770 tok/s decode at bs=1 on InferenceX 8K/1K
SGLang-TorchTPU is available now: github.com/sgl-project/sglan…
Big day for the @PyTorch ecosystem at #PyTorchConf2026 🚀
We’re officially open-sourcing TorchTPU—enabling native, hardware-accelerated PyTorch execution across Cloud TPUs and custom infrastructure.
Here’s everything we launched to supercharge your training & inference workloads 🧵👇
goo.gle/4Bl0zdG
Ivan Zhou retweeted
It was always nice working with Google and Meta people! TPU has lots of lots of potential.
And yes we will be submitting AgentX results very soon 🚀
Today we launch SGLang-TorchTPU, built with @Google TPU team and Meta @PyTorch team.
We’re excited to see it powering production inference at Meta!
SGLang-TorchTPU is a TPU plugin for SGLang that brings:
- Top open models on TPU: Muse Spark Open, GLM-5.3 Flash, Kimi K3, Ling 3.1 Flash, MiniMax H3
- SGLang’s production features for large agentic workloads
- Custom MegaKernel: Muse Spark Open hits 770 tok/s decode at bs=1 on InferenceX 8K/1K
SGLang-TorchTPU is available now: github.com/sgl-project/sglan…
Ivan Zhou retweeted
SGLang-Diffusion is a high-performance serving framework for diffusion models, designed for both large-scale offline generation and latency-sensitive real-time inference.
At PyTorch Conference North America 2026, Yihao Wang and Kevin Mi of @RadixArk will present the design of @sgl_project-Diffusion and the PyTorch-based inference optimizations behind it, including efficient request scheduling, memory management, batching strategies, and execution-path optimizations for diffusion pipelines.
Join us in San Jose, October 20-21: hubs.la/Q04v4SL60 #PyTorchCon
Ivan Zhou retweeted
We have been working on making Beam easier to customize at @reflection_ai. Seeing the recipes and integrations go live feels pretty amazing.
We're putting the code, training configs, and examples out there so people can start building on Beam. Here's what's in the cookbook:
Data generation: Use Beam to create training data, including software-engineering trajectories that pass tests and reasoning examples for tool calling.
github.com/reflection-oss/co…
Fine-tuning: Adapt Beam to your data and tasks with LoRA, QLoRA, or full fine-tuning.
github.com/reflection-oss/co…
Distillation: Use Beam to teach a smaller model. We've included recipes for coding trajectories and tool-calling reasoning.
Coding: github.com/reflection-oss/co…
Tool calling: github.com/reflection-oss/co…
Reinforcement learning: Post-train Beam against verifiable rewards.
github.com/reflection-oss/co…
Huge thanks to our partners and the teams behind these integrations:
• NeMo AutoModel, @NVIDIAAIDev: SFT with LoRA and full fine-tuning.
docs.nvidia.com/nemo/automod…
• Megatron Bridge, @NVIDIAAIDev: Large-scale full fine-tuning.
github.com/NVIDIA-NeMo/Megat…
• Axolotl, @axolotl_ai: LoRA, QLoRA, and full fine-tuning.
github.com/axolotl-ai-cloud/…
• HF TRL, @huggingface: Training smaller models on Beam-generated data.
github.com/huggingface/trl
• Miles, @radixark: RL post-training.
miles.radixark.com/docs/mode…
• NeMo RL, @NVIDIAAIDev: RL post-training on NVIDIA hardware.
github.com/NVIDIA-NeMo/RL/bl…
• SkyRL, @NovaSkyAI: LoRA and full-parameter RL.
docs.skyrl.ai/docs/examples/…
Each recipe has its own instructions, dependencies, sample inputs, and smoke test. I'm excited to see people run these experiments, adapt Beam to their own work, and contribute recipes we haven't thought of.
Huge thanks to @Nazeri2010, @bhutanisanyam1, @kathleen_mhatre, @FGbelidji, @joespeez, and to Thibaut, Matthias, Travis, Kai, and Yoni for making this happen.
Explore the cookbook: github.com/reflection-oss/co…
Access the model: platform.reflection.ai
Documentation: developers.reflection.ai
Provide feedback on Discord: discord.com/invite/PxUfdsNny
Ivan Zhou retweeted
Beam × Miles 🤝
We've published RL post-training recipes for @reflection_ai's Beam, covering full-parameter GRPO and LoRA training with SGLang-powered rollouts.
Thanks @chetantekur and the Reflection team for the collaboration!
Check out the cookbook: miles.radixark.com/docs/mode…
We have been working on making Beam easier to customize at @reflection_ai. Seeing the recipes and integrations go live feels pretty amazing.
We're putting the code, training configs, and examples out there so people can start building on Beam. Here's what's in the cookbook:
Data generation: Use Beam to create training data, including software-engineering trajectories that pass tests and reasoning examples for tool calling.
github.com/reflection-oss/co…
Fine-tuning: Adapt Beam to your data and tasks with LoRA, QLoRA, or full fine-tuning.
github.com/reflection-oss/co…
Distillation: Use Beam to teach a smaller model. We've included recipes for coding trajectories and tool-calling reasoning.
Coding: github.com/reflection-oss/co…
Tool calling: github.com/reflection-oss/co…
Reinforcement learning: Post-train Beam against verifiable rewards.
github.com/reflection-oss/co…
Huge thanks to our partners and the teams behind these integrations:
• NeMo AutoModel, @NVIDIAAIDev: SFT with LoRA and full fine-tuning.
docs.nvidia.com/nemo/automod…
• Megatron Bridge, @NVIDIAAIDev: Large-scale full fine-tuning.
github.com/NVIDIA-NeMo/Megat…
• Axolotl, @axolotl_ai: LoRA, QLoRA, and full fine-tuning.
github.com/axolotl-ai-cloud/…
• HF TRL, @huggingface: Training smaller models on Beam-generated data.
github.com/huggingface/trl
• Miles, @radixark: RL post-training.
miles.radixark.com/docs/mode…
• NeMo RL, @NVIDIAAIDev: RL post-training on NVIDIA hardware.
github.com/NVIDIA-NeMo/RL/bl…
• SkyRL, @NovaSkyAI: LoRA and full-parameter RL.
docs.skyrl.ai/docs/examples/…
Each recipe has its own instructions, dependencies, sample inputs, and smoke test. I'm excited to see people run these experiments, adapt Beam to their own work, and contribute recipes we haven't thought of.
Huge thanks to @Nazeri2010, @bhutanisanyam1, @kathleen_mhatre, @FGbelidji, @joespeez, and to Thibaut, Matthias, Travis, Kai, and Yoni for making this happen.
Explore the cookbook: github.com/reflection-oss/co…
Access the model: platform.reflection.ai
Documentation: developers.reflection.ai
Provide feedback on Discord: discord.com/invite/PxUfdsNny
Ivan Zhou retweeted
Fast inference and verified answers go well together. Tudor (@tachim) is on the Neo Labs panel at the SGLang Summit with Igor Babuschkin, Eric Zelikman and Liam Fedus, Nov 12–13 in SF. Don't miss this one.
Proud to welcome our newest sponsors to SGLang Summit.
In just one week since our public announcement, our sponsor lineup has grown from 11 to 36. Thank you to every sponsor helping bring this community together.
Join 2,000+ engineers, researchers, and builders from across the AI ecosystem for technical sessions, hands-on workshops, and the GPU Expo.
Hear from Lip-Bu Tan, Lilian Weng, Soumith Chintala, Aravind Srinivas, and more. Meet the teams building across AI hardware, infrastructure, models, and applications.
🗓️ November 12–13, 2026
🌉 Fort Mason, San Francisco
🎟️ Get your tickets: sglang.io/summit
Ivan Zhou retweeted
ALERT🚨: SGLANG JUST MERGED MEGAKERNEL SUPPORT, GETTING CLOSE TO 1000 tok/s/user on the GB300 NVL72 AgentX Harness & increasing interactivity by 1.5x on MI355X too!
Introducing SGLang Low-Latency Mode, built with @NVIDIAAI and @AMD.
On DeepSeek-V4.1-Flash with DSpark, it delivers up to 1.38× P90 per-user token speed on NVIDIA GB300 (671 to 928 tok/s) and 1.56× on AMD MI355X (332 to 516 tok/s) in @SemiAnalysis_'s AgentX trace replay.
Highlights:
- NVIDIA: overlapping kernels cut decoder layer time from 96.8 to 68.4 µs
- AMD: 2 persistent kernels per layer replace dozens of small ones, with cross-GPU communication overlapped inside them.
- New plugin system: custom kernels plug in from external libraries, with automatic fallback.
- Low-latency kernels for GLM-5.3, K3, and MiniMax M3 are in progress and will plug in next.
Ivan Zhou retweeted
ALERT🚨: SGLANG JUST MERGED MEGAKERNEL SUPPORT, GETTING CLOSE TO 1000 tok/s/user on the GB300 NVL72 AgentX Harness & increasing interactivity by 1.5x on MI355X too! Great work by the SGLang, RadixArk, NVIDIA, and AMD teams!
Best of all, this already works out of the box on SGLang Docker images! Excited to rapidly get this on the official AgentX!
xitter.cf/lmsysorg/status/211234…
Introducing SGLang Low-Latency Mode, built with @NVIDIAAI and @AMD.
On DeepSeek-V4.1-Flash with DSpark, it delivers up to 1.38× P90 per-user token speed on NVIDIA GB300 (671 to 928 tok/s) and 1.56× on AMD MI355X (332 to 516 tok/s) in @SemiAnalysis_'s AgentX trace replay.
Highlights:
- NVIDIA: overlapping kernels cut decoder layer time from 96.8 to 68.4 µs
- AMD: 2 persistent kernels per layer replace dozens of small ones, with cross-GPU communication overlapped inside them.
- New plugin system: custom kernels plug in from external libraries, with automatic fallback.
- Low-latency kernels for GLM-5.3, K3, and MiniMax M3 are in progress and will plug in next.
Ivan Zhou retweeted
Introducing SGLang Low-Latency Mode, built with @NVIDIAAI and @AMD.
On DeepSeek-V4.1-Flash with DSpark, it delivers up to 1.38× P90 per-user token speed on NVIDIA GB300 (671 to 928 tok/s) and 1.56× on AMD MI355X (332 to 516 tok/s) in @SemiAnalysis_'s AgentX trace replay.
Highlights:
- NVIDIA: overlapping kernels cut decoder layer time from 96.8 to 68.4 µs
- AMD: 2 persistent kernels per layer replace dozens of small ones, with cross-GPU communication overlapped inside them.
- New plugin system: custom kernels plug in from external libraries, with automatic fallback.
- Low-latency kernels for GLM-5.3, K3, and MiniMax M3 are in progress and will plug in next.
Ivan Zhou retweeted
SemiAnalysis is an active supporter of open source in the ML community! As such, we are excited to sponsor the open-source SGLang Summit! SGLang is one of the engines that power production inference across many frontier labs globally!
At the summit, we’ll share more about our collaboration with SGLang, including our latest AgentX benchmarks.
xitter.cf/sgl_project/status/211…
Proud to welcome our newest sponsors to SGLang Summit.
In just one week since our public announcement, our sponsor lineup has grown from 11 to 36. Thank you to every sponsor helping bring this community together.
Join 2,000+ engineers, researchers, and builders from across the AI ecosystem for technical sessions, hands-on workshops, and the GPU Expo.
Hear from Lip-Bu Tan, Lilian Weng, Soumith Chintala, Aravind Srinivas, and more. Meet the teams building across AI hardware, infrastructure, models, and applications.
🗓️ November 12–13, 2026
🌉 Fort Mason, San Francisco
🎟️ Get your tickets: sglang.io/summit
Ivan Zhou retweeted
SGLang became the infra for serving LLMs. @PhAILabs is building the infra for AI-driven science.
Proud to see PhAI support the @sgl_project community at SGLang Summit — open infrastructure is how both AI and science scale.
📅 Nov 12–13 · Fort Mason, SF
Proud to welcome our newest sponsors to SGLang Summit.
In just one week since our public announcement, our sponsor lineup has grown from 11 to 36. Thank you to every sponsor helping bring this community together.
Join 2,000+ engineers, researchers, and builders from across the AI ecosystem for technical sessions, hands-on workshops, and the GPU Expo.
Hear from Lip-Bu Tan, Lilian Weng, Soumith Chintala, Aravind Srinivas, and more. Meet the teams building across AI hardware, infrastructure, models, and applications.
🗓️ November 12–13, 2026
🌉 Fort Mason, San Francisco
🎟️ Get your tickets: sglang.io/summit
Ivan Zhou retweeted
We're incredibly proud to support and join the @sgl_project community at the SGLang Summit. @PhAILabs
Can't wait to connect with everyone there!
Proud to welcome our newest sponsors to SGLang Summit.
In just one week since our public announcement, our sponsor lineup has grown from 11 to 36. Thank you to every sponsor helping bring this community together.
Join 2,000+ engineers, researchers, and builders from across the AI ecosystem for technical sessions, hands-on workshops, and the GPU Expo.
Hear from Lip-Bu Tan, Lilian Weng, Soumith Chintala, Aravind Srinivas, and more. Meet the teams building across AI hardware, infrastructure, models, and applications.
🗓️ November 12–13, 2026
🌉 Fort Mason, San Francisco
🎟️ Get your tickets: sglang.io/summit