The LLM training framework for RL Scaling. https://xitter.cf/t.co/CCT4tADFir
Joined September 2025
- Tweets131
- Following12
- Followers2.4K
- Likes68
Pinned Tweet
OpenAI's Navier-Stokes effort used ~10,000 concurrent agents and ~130B output tokens. It made us rethink the scale ahead for RL infrastructure.
slime v0.4.0 is a first step toward greater scale, with four architectural changes:
• Focus: synchronous training for algorithmic exploration and correctness; fully async training for throughput at scale.
• Distributed rollout: spread orchestration across cluster CPUs, with independent generation pools and event loops.
• Persistent storage: introduce straw, our new storage middleware for rollout and training data, providing durable queues and shared tensor storage.
• Independent recovery: restart Megatron while healthy SGLang engines stay running, preserving accepted rollout work.
More on the architecture below 👇
github.com/THUDM/slime/relea…
1/5
slime retweeted
Score Centering 火得有点快。9 月 17 日 @mrtnm Martin Marek 和 Max Ryabinin 提出，9 月 23 日 @slime_framework 就实现了（PR #2405），10 月 1 日 verl 也合了（PR #8010）。到 10 月 10 日，原论文、官方实现、verl 的 rollout correction 都已经就位。
而 BPO 是 9 月 14 日发的，一层梯度近似之后，它的梯度结构和完整 SC 一样。
工作被广泛关注后，人们开始从不同的问题入手重新解释 SC。文章把这五种解释摆在一起，用的都是各自的原始说法。
按原 SC 论文，它是一种解决训推不一致导致的 drift 的方法。
按 BPO 的说法，它是根据 PMD 的闭式解改写的 critic-free 目标的 token-level surrogate 的梯度。
从 advantage 的定义出发，它是人们希望得到 sampler 分布的精确 value baseline 而推出来的期望更新。
剩下两个，一个把它看成直通估计器，一个说它是构造出的局部分布族在起点处的真实 score @RichardYRLi
最后两个其实很接近，在变化靠近更新零点 δ=0 时是一样的，只是出发点不同。作者在文章里说，每一点都能独立推出 SC，这真是异曲同工。
工程上还卡着一笔账。保存和传输 sampler 完整概率分布太贵，每个生成位置都要保存约 15 万个概率，fp32 下就是 600 KB 左右，所以 verl 和官方实现都只存 top-k，尾部用 trainer 的分布去近似。
slime environment update:
- SGLang upgraded to v0.5.21
- Megatron upgraded to v0.19.2
- TransformerEngine upgraded to v2.18
Previously, slime provided both CUDA 12 and CUDA 13 images. Following SGLang's CUDA requirements since v0.5.20, we will discontinue CUDA 12 images, and the latest tag will now point to CUDA 13.
For users who cannot easily upgrade their drivers, we have included cuda-compat in the CUDA 13 image, allowing supported CUDA 12-era drivers to run the new image without an immediate driver upgrade.
Additionally, with the SGLang upgrade, slime now officially supports RL training for GLM-5.3-Flash.
slime retweeted
Recursive self-improvement needs agents that can build their own training environments. But researchers still part of the loop, making it hard to scale.
We introduce AutoEnvScaling, which turns Data Flywheel into a terminal task: a proposer agent builds complete Harbor tasks and sandbox. It builds just like human researchers. This enables self-play RL for terminal agents: one model improves as both builder and solver, without human-in-the-loop.
Amazing work from @modal and @nanjiangwill !
A refreshing glimpse into the future of large-scale RL!
Training large models with RL typically runs on a fixed homogeneous cluster of the latest accelerators for numerical stability and faster throughput. This throws away inference optimization work done for other vendors and generations, limits the capacity you can utilize, and is unnecessarily expensive.
So we trained a model on agentic coding tasks using fully asynchronous RL and mixed-precision, mixed-accelerator samplers. With off-policy correction, training stayed stable through convergence, on par with homogeneous samplers, at a lower cost.
slime retweeted
Introducing GLM-Legal
Built on GLM-5.3-Flash, GLM-Legal matches frontier models on mainstream legal benchmarks.
Designed for real legal workflows: contract review, evidence analysis, compliance, and more.
Tech Blog: z.ai/blog/glm-legal
Apply for API Access: z.ai/consultation/legal
Try AutoClaw: autoclaw.z.ai
slime retweeted
slime 是一个非常优秀的RL后训练框架！
重写后的 slime 主页里面尽可能包含了目前的 llm + rl 的一些基本逻辑，希望这个文档能对刚开始做 rl 的朋友有一些帮助，然后对于老手来说，如果想给身边的人解释自己在做啥，也可以随手打开 slime 的官网做个辅助~
We've rebuilt slime's docs with an interactive homepage for LLM + RL!
We hope it can be a place for beginners to learn, and a handy tool for practitioners to explain what they do.
Have a look at:
thudm.github.io/slime/
We've rebuilt slime's docs with an interactive homepage for LLM + RL!
We hope it can be a place for beginners to learn, and a handy tool for practitioners to explain what they do.
Have a look at:
thudm.github.io/slime/
Introducing sunabako, our second companion project for slime after straw: a lightweight sandbox runtime for agentic RL.
Setting up a separate sandbox service can slow you down when you’re still testing a training recipe. sunabako runs sandboxes inside your existing training containers, letting you use the cluster resources you already have to validate the full training loop before scaling out.
With sunabako, we’ve also added end-to-end agentic RL testing with Qwen3.8-27B powering Codex and Claude Code on coding tasks from MiMo’s newly released RL dataset. These tests cover agent execution, sandboxed grading, and policy updates, helping us catch regressions across slime’s agent stack.
Give slime + sunabako a try!
github.com/THUDM/slime/pull/…
sunabako also works independently of slime, with a Python SDK that supports:
• Running sandboxes without Docker-in-Docker
• Mounting local directories into sandboxes
• Using GPUs already exposed to the parent container
• Basic E2B-style APIs for sandbox creation, command execution, and file I/O
Install with:
pip install sunabako
Setup and examples:
github.com/zhuzilin/sunabako
Introduce Straw: storage middleware for RL.
Distributed workers need a shared data layer for exchanging payloads and preserving progress.
Straw provides durable queues and tensor storage on a shared filesystem. Rollout workers publish data; training ranks read it by reference. Bulk payloads move through storage, while Ray carries control messages and references.
Prompts, partial rollouts, completed samples, and converted training batches persist. Packed files reduce small-file overhead, and rollout state can be checkpointed alongside the model.
JuiceFS is the currently supported shared-filesystem target.
github.com/zhuzilin/straw
4/5
Recover training while preserving serving and rollout work.
A trainer OOM or configuration mistake becomes more expensive as the job grows.
v0.4.0 lets you adjust supported trainer parallelism or memory settings and resubmit to the same live Ray cluster. Healthy SGLang engines and routers remain running.
With Straw or saved rollout debug data, the new trainer restores the last committed checkpoint and replays retained batches, reusing accepted samples and saved conversion results.
This makes training retries cheaper by preserving serving resources and completed rollout work.
5/5
slime retweeted
We wanted Score Centering to be a plug-and-play mismatch fix for off-policy RL setups. That's why it is very rewarding to see our method in slime in less than a week from release
Try it out for stabilizing your next run!
We've just added score centering to slime!
You can enable sc with --use-score-centering
PR: github.com/THUDM/slime/pull/…
We've just added score centering to slime!
You can enable sc with --use-score-centering
PR: github.com/THUDM/slime/pull/…
RL with LLMs is very unstable when training and sampling policies differ. Standard fixes (matching numerics, importance sampling) work around the problem.
We find the root cause of this instability from first principles and propose a way to directly cancel it. Score Centering is competitive and compatible with existing approaches — while simple to implement!
🧵 [1/6]
slime retweeted
Huge congrats to the @radixark team on Miles v0.1! 🚀
Fully async agentic RL on a 744B MoE across 64 GB300s, with @sgl_project powering rollouts. Love the focus on getting rollout/training fidelity right alongside throughput.
huggingface.co/papers/2609.0…
slime retweeted
GLM-5.3 is now open-weight.
Our most capable model for agentic coding and cyber defense is now available to download, run, and customize.
Weights: huggingface.co/zai-org/GLM-5…
Tech blog: z.ai/blog/glm-5.3
