reward hacking for human-ai collaboration @StanfordNLP
Palo Alto, CA
Joined May 2020
- Tweets255
- Following359
- Followers633
- Likes663
AI is getting remarkably good at doing things for humans. We know much less about how to make it better at helping humans learn.
We study learning to teach adaptively: how AI can adapt its teaching to each learner’s evolving understanding, and how this capability can itself be optimized through reinforcement learning.
A small step toward AI that doesn’t just solve problems, but helps people understand and grow.
More details in @WeixianXu ’s thread 👇
Language models already excel at math, but human growth remains essential: we need AI to teach people, not replace them!
Teaching is different from solving. As the Chinese principle “因材施教” suggests, preferences differ from person to person, and a good teacher should adapt their teaching to each learner.
We introduce Sherpa, a framework for training LLMs to teach adaptively.
A key post-training paradigm shift from @mimo_labs to DeepSeek is the move to multi-teacher on-policy distillation - building the generalist from a diverse pool of 10+ domain experts.
Again surprised by their RL infra that supports full-vocabulary OPD with unbounded (??) number of teachers.
🚀 DeepSeek-V4 Preview is officially live & open-sourced! Welcome to the era of cost-effective 1M context length.
🔹 DeepSeek-V4-Pro: 1.6T total / 49B active params. Performance rivaling the world's top closed-source models.
🔹 DeepSeek-V4-Flash: 284B total / 13B active params. Your fast, efficient, and economical choice.
Try it now at chat.deepseek.com via Expert Mode / Instant Mode. API is updated & available today!
📄 Tech Report: huggingface.co/deepseek-ai/D…
🤗 Open Weights: huggingface.co/collections/d…
1/n
super cool work on self-play algos. feels very aligned with @CarinaLHong 's “spiral progression” vision for AI in mathematics from a recent podcast: youtube.com/watch?v=78Vyy_dz…
🏆 Excited to share GEM received the Outstanding Paper Award @SEAWorkshop of #NeurIPS2025 . What a great way to wrap up this amazing neurips journey!
Huge thanks to the workshop committee and organizers for the recognition.
Grateful for our incredible collaborators and advisors who made this project possible. Thanks to everyone involved! 🎉
Congrats to the following paper authors attaining Outstanding Paper Awards at @SEAWorkshop!
GEM: A Gym for Agentic LLMs
Zichen Liu, Anya Sims, Keyu Duan, Changyu Chen, Haotian Xu, Simon Yu, Chenmien Tan, Shaopan Xiong, Weixun Wang, Bo Liu, Hao Zhu, Weiyan Shi, Diyi Yang, Wee Sun Lee, Min Lin
Would be around the scaling environments for agents workshop #NeurIPS2025 today 🌊
Come say hi and chat about GEM 💎, RL and experience scaling 💡
gem: arxiv.org/pdf/2510.01051
In the era of experience, we're training LLM agents with RL — but something's missing...
We miss the good old Gym!
So we built 💎GEM: a suite of environments for training LLM 𝚐𝚎𝚗𝚎𝚛𝚊𝚕𝚒𝚜𝚝𝚜.
Let’s build the Gym for LLMs, together: axon-rl.notion.site/gem
Excited to see GEM (github.com/axon-rl/gem) used in the implementation example of RL for LLMs in the latest update of @sirbayes 's fantastic RL tutorial.
For anyone who wants to dive into the imple., this is the clearest walkthrough that lets you understand the details from the text and math equations (along with a full runnable demo of ~100 lines ofc).
I am pleased to announce another update to my RL tutorial (arxiv.org/abs/2412.05265). This time I have added code for RLFT for multi-turn LLM agents, using the awesome Tinker library from @thinkymachines, and the simple ReBN training loop from GEM by @zzlccc et al. With ~100 lines of simple python running on your laptop, you can train an agent based on Qwen3-4B-Instruct to play "guess the number" in 20 minutes.
A great deep-dive tutorial of our paper by the amazing @yacinelearning
🔗 youtube.com/watch?v=4egPDoZC…
