⚡ Working on a major speedup for AsyncGRPO in TRL: training steps are up to 3.5× faster so far! AsyncGRPO already has several packing optimizations. One packs samples together and balances the attention cost across DP ranks. You start with the same prompt G times. In multi-turn rollouts on a harness, the conversation can also start to fork: most of the history remains identical, while a tool result or a new turn changes the suffix. That gets expensive with GRPO because rollouts can share many long prefixes. You start from the same prompt G times, and multi-turn rollouts on a Harness can also share most of their conversation history (thanks to message-level tokenization). Each rollout is treated as its own sequence and packed on a single row, subject to a predefined token budget per DP-rank. Those rows are forwarded independently, even when most of their tokens are identical. Tree packing instead turns the rows into a prefix tree. Shared prefixes are stored and forwarded once, while we use FlexAttention to ensure that every token still sees exactly the context it would have seen in its original rollout. The loss is still defined on the original rollouts, so every trained token keeps its own target, advantage, and old log-prob. The video shows the algorithm pretty well 👇

Same reward curve. A quarter of the clock! 🚅 TRL's AsyncGRPOTrainer decouples rollouts from training: a background worker streams completions from a vLLM server while the training loop consumes them. The bench is the well-crafted sail/Sanity-Test-R1D-1.5B. 3.8× on the clock. 1.9× on GPU-hours.