
LLM/NLP researcher in Baichuan-Inc @BaichuanAI. Prev @AlibabaGroup, Research Intern at @MSFTResearch Asia , GitHub: https://teapaw.org/t.co/kUbOq3uc75
Joined May 2025
- Tweets14
- Following47
- Followers14
- Likes59
Pinned Tweet
🎉 Introducing our latest work !
🚀 We propose a label-free method that enables RL without ground truth answer, yet achieves impressive performance on mathematical tasks: 40.0% accuracy on AIME2024 🎯 with a 7B base model.
Paper: huggingface.co/papers/2505.1…
insightLLM retweeted
Harmony format is finally open-sourced. I still remember 3 years ago (before ChatGPT release) @shengjia_zhao, Daniel and I were brainstorming about the right abstraction for RL training, and that is the start point of the entire harmony library. github.com/openai/harmony
Why does GRPO make responses shorter as training goes on? 🤔
While reproducing Deepseek-R1, we noticed a weird thing: the longer you train with GRPO, the shorter the responses get 📉.Totally opposite to what paper shows(See Figure 1)! Some blame KL divergence, others say the model’s too small (it’s not like we all have the budget to train a 671B LLM 😢).
But we finally found the real culprit💡: length normalization in the formula(See Figure 2 in DeepSeekMath paper). Since the objective maximizes rewards, dividing by |oᵢ| punishes longer outputs. So the model prefers shorter and shorter replies.
And here’s the kicker: Deepseek-R1’s final formula actually removes this normalization!🆘 (See Figure 3)
BTW, both DAPO and Mistral’s new Magistral fixed this too. @dongxi_nlp @chen_mingyang
4
72
Meaningful!💡The reason behind this phenomenon: the base model(from pre-train) already possesses strong mathematical and logical reasoning abilities. The post-train merely serves to elicit these abilities, so the ground truth answers themselves are actually not that important.
🤯 We cracked RLVR with... Random Rewards?!
Training Qwen2.5-Math-7B with our Spurious Rewards improved MATH-500 by:
- Random rewards: +21%
- Incorrect rewards: +25%
- (FYI) Ground-truth rewards: + 28.8%
How could this even work⁉️ Here's why: 🧵
Blogpost: tinyurl.com/spurious-rewards
1
160
insightLLM retweeted
🚀 Introducing ReCall, learning to Reason with Tool Call via RL.
- Multi-turn Reinforcement Learning
- No need for supervised data on tool use or reasoning steps
- Empowers LLMs to agentically use and combine arbitrary tools
Fully open-source! A work in progress and we are actively working on it. Stay tuned!
Code: github.com/Agent-RL/ReCall
Blog: bit.ly/3Eo7xpL
#LLMs #DeepSeek #reasoning #RL #ReinforcementLearning #ToolUse #FunctionCall #GRPO
📚Experiments show that structure constraints and length optimization alone can yield performance comparable to baseline methods that rely on ground-truth answers.
2
73
✨We find that constructing surrogate signals based on format correctness and response length, combined with the GRPO algorithm, enables effective training of large models to solve mathematical problems.
2
67










