Rupesh Srivastava (@rupspace) profile picture

LLM Post-training Lead @IFM_AI. Previously (co)developed Highway Networks, Upside-Down RL, Bayesian Flow Networks, EvoTorch.

Santa Cruz, CA
Joined September 2014
Photo by Rupesh Srivastava (@rupspace)Photo by Rupesh Srivastava (@rupspace)Photo by Rupesh Srivastava (@rupspace)Photo by Rupesh Srivastava (@rupspace)Photo by Rupesh Srivastava (@rupspace)Photo by Rupesh Srivastava (@rupspace)Photo by Rupesh Srivastava (@rupspace)Photo by Rupesh Srivastava (@rupspace)Photo by Rupesh Srivastava (@rupspace)Photo by Rupesh Srivastava (@rupspace)
Do You Like Open Source AI? huggingface.co/collections/I… We just released a fleet of 6 great LLMs: 0.9B, 4B, 7B, 32B dense, 36 MoVA, 375B MoE GGUFs and FP8 quantized variants New "Uno" adapters for fast diffusion decoding! Loads of training data, logs, intermediate ckpts for pre-training, midtraining, post-training.
Introducing K2 Horizon: a connected fleet of six foundation models ranging from 0.9 billion to 375 billion parameters. - Frontier performance: Across coding and agentic tasks, K2 Horizon delivers top-tier performance in every size class—with the 0.9B, 3.7B and 7B models setting new state of the art at their respective scales. - Radical openness: K2 Horizon represents the largest fully open-source model launch in AI history. The fully open code, training data and recipes are a significant step forward in transparency. Launch page: ifm.ai/k2/ Tech blog: ifm.ai/blog/k2 Hugging Face: huggingface.co/collections/I…
1
5
1
54
3,500
The K2 Horizon tech report is out!
Looking for a reference manual for building a frontier LLM? We wrote one 👇 The K2 Horizon technical report is now on arXiv: 134 pages on how we built six models, from architecture to inference. It may be the most detailed LLM training reference you can read today. • Architecture: we bring two new attention designs, MoVA and softplus attention gating, to strengthen reasoning. With them, our 36B model nearly matches a 32B dense one on one-seventh of the active parameters. • Pre-training: we trained four models, from 3.7B to 36B, on the same 22T tokens. Their normalized loss curves collapse onto nearly the same line, so a recipe that works on a small model carries over when scaling up. • Data: 60+ pages on the part most reports skip. How we generated ~10T synthetic tokens, and why we chose the data mix on predicted scores at scale, which reversed the small-run ranking. • Post-training: we document our full production-scale run for the 375B flagship. • Inference: Uno gives an autoregressive model diffusion speed with one small adapter. Up to 2.2× faster decoding, lossless by construction, no draft model. This is the last piece of the openness puzzle. K2 Horizon already shipped with 380 checkpoints, 13 TB of pre-training data, 58 training runs and 17 code repositories. The report explains the decisions behind them. More details: arxiv.org/abs/2611.21003
2
7
341
Rupesh Srivastava retweeted
Meet PAN 2, our latest world model. Grounded in physics, time, and space, it pairs an LLM reasoner with a video diffusion backbone to simulate physical reality with action timing and long-horizon coherence. Check it out: ifm.ai/pan/. #worldmodel #physicalintelligence
2
8
215
Rupesh Srivastava retweeted
Pelicans made by PAN2
4
4
2
23
19,607
Introducing PAN 2: an open foundation world model for physical intelligence. A language model predicts the next word. PAN 2 predicts the next world: what happens after each action, grounded in time, physics and space. Towards AI that simulates “what if?” before it acts. More details: ifm.ai/pan/
86
147
168
624
2,060,497
Not a coincidence that B looks very much like a good RL reward curve!
Updated version!
2
344
Rupesh Srivastava retweeted
When I saw @rikiyatakehi and @jayden_teoh_ 's blog about Astra’s ability to read and edit weights, I got curious as to how it's doing this. I started out with their image prediction task, It turns out the input and output weights especially are very human interpretable. Hopefully for you too..? :P I wonder if we can learn more from Astra's strategies for solving these. See for yourself! claude.ai/artifact/5DivY44tr…
2
5
80
4,796
Rupesh Srivastava retweeted
Excited to share AgentBaton, our work at #NVIDIA showing how well-timed handoffs to a smaller model can cut costs by 65.9% on average while maintaining or improving accuracy.
When should an AI agent pass the baton? 🚀 Introducing our work AgentBaton from NVIDIA Research: efficient model collaboration through well-timed handoffs. We study how capable and efficient models can collaborate without fine-tuning either one: who starts, when to switch, and how to pass context. 🛠️ Our approach: a capable model starts; AgentBaton decides when an efficient model takes over. The next model inherits the workspace, with history passed in context or through log files. The same detector works across different model pairs. 📈 The results: AgentBaton extends the accuracy–cost frontier. Across 11 model pairs, AgentBaton cuts API costs by 65.9% on average with same or improved accuracy. 📄 arxiv.org/abs/2610.35959 🌐 research.nvidia.com/labs/lpr… Details in 🧵 #NVIDIA
1
1
10
471
Maximus Openus LLM training
The most detailed open LLM data release yet: K2 Horizon's pre-training data and post-training data-generation pipelines are now open! 13 trillion pre-training tokens across 5 datasets and 24 subsets. Over 10 trillion tokens of synthetic data spanning mathematics, reasoning, coding, planning, and problem-solving. Beyond the scale, four highlights for model builders (details in the threads): - Scaling synthetic data without losing diversity. Our generation methods produce diverse training examples at scale, nearly matching natural web text in measured diversity. - More verification isn't always better. Strict verification can throw away useful training signal. - Small-scale wins can mislead. Our chosen data mix actually lost in smaller experiments. - Beyond teacher imitation. Our post-training data generation methods produce training trajectories the teacher couldn't generate on its own. Open pre-training datasets, post-training data-generation pipelines, and processing tools for researchers to inspect, reuse, and build on. Pre-training data: huggingface.co/collections/I… Post-training data-generation code & tools: github.com/ifm-ai/DATA_NAVIG…
1
14
530
Me when I first understood what science fiction is about:
The most interesting thing about the current state of the world: Even though everything is accelerating a lot, the overall dynamics of the world seem to stay the same
5
360
Rupesh Srivastava retweeted
🎊 Excited to give a tutorial on diffusion LLMs at @NeurIPSConf 2026 with @StefanoErmon @JCJesseLai @mittu1204 📆 Mon, Dec 7, 9 AM AEDT 📍 TikTok Ent Centre (Sydney, Australia) Topics covered: > A beginner friendly introduction to diffusion-LLMs > Designing frontier diffusion-LLMs > Using Continuous diffusion for LLMs
2
27
2
92
7,684
8,192 concurrent sandboxes and ~1.3 million rollouts. Really impressive engineering from @IFM_AI! One thing I find particularly interesting about long-horizon agentic RL is that infrastructure failures can directly affect training correctness. If a sandbox crashes halfway through a trajectory and the system treats it as a failed task, we're effectively penalizing the model for something it never did wrong. At this scale, distinguishing environment failures from actual model failures becomes part of the learning algorithm itself. RL360 also demonstrates why modular RL infrastructure matters. With Miles handling rollouts and optimization, Harbor managing agent environments, SGLang serving inference, and Megatron-LM running training, each component can evolve and scale independently. I'm especially happy to see Miles and SGLang being used together in such a large-scale, real-world RL system. The lessons about asynchronous execution, distributed trajectory management, and sandbox reliability are incredibly valuable for the entire open-source community. Huge congratulations to the IFM team! github.com/ifm-ai/RL360
We're open-sourcing RL360, the RL infrastructure behind K2 Horizon's six domain experts: 8,192 concurrent sandboxes, 150 steps in 7 days, ~1.3M rollouts. Long-horizon agentic RL runs thousands of agents in real sandboxes on tasks that take minutes. Some of those sandboxes crash mid-task, and a crash arrives as the same low score as a wrong answer. Without infrastructure that can keep up, GPUs are underutilized and the model gets penalized for system crashes rather than its own mistakes. RL360 is built for this. We share how it works, what broke at scale, and what we learned in the thread. Code: github.com/ifm-ai/RL360 Run: wandb.ai/mbzuai-llm/rl360-pu…
7
9
83
6,839
Rupesh Srivastava retweeted
Agentic RL is a whole new beast. Things broke. A lot. Pods crashed. Registries choked. Runs stalled. I spent the last few months in the trenches isolating failures and fixing bottlenecks. RL360 is what came out of it: millions of rollouts, scaled across multiple experts. Proud of this one. Here's how 👇
We're open-sourcing RL360, the RL infrastructure behind K2 Horizon's six domain experts: 8,192 concurrent sandboxes, 150 steps in 7 days, ~1.3M rollouts. Long-horizon agentic RL runs thousands of agents in real sandboxes on tasks that take minutes. Some of those sandboxes crash mid-task, and a crash arrives as the same low score as a wrong answer. Without infrastructure that can keep up, GPUs are underutilized and the model gets penalized for system crashes rather than its own mistakes. RL360 is built for this. We share how it works, what broke at scale, and what we learned in the thread. Code: github.com/ifm-ai/RL360 Run: wandb.ai/mbzuai-llm/rl360-pu…
3
2
1
18
591
We learned a TON in the last 6 months about Agentic RL infrastructure at scale, and now we're happy to share it with everyone. RL360 was used for the training of K2 Horizon series (artificialanalysis.ai/models…). It builds on top of miles from @radixark, adding a lot of optimizations for efficient agentic RL at scale. Our amazing team at @IFM_AI built this with a lot of hard work. Thanks to @varad0309 @willis_guo @wang1999_zt @mingshanhee @DavidRBellamy @omkarenator @_matthewyang @qiancheng1231 @bhandari_jalaj and many others!
We're open-sourcing RL360, the RL infrastructure behind K2 Horizon's six domain experts: 8,192 concurrent sandboxes, 150 steps in 7 days, ~1.3M rollouts. Long-horizon agentic RL runs thousands of agents in real sandboxes on tasks that take minutes. Some of those sandboxes crash mid-task, and a crash arrives as the same low score as a wrong answer. Without infrastructure that can keep up, GPUs are underutilized and the model gets penalized for system crashes rather than its own mistakes. RL360 is built for this. We share how it works, what broke at scale, and what we learned in the thread. Code: github.com/ifm-ai/RL360 Run: wandb.ai/mbzuai-llm/rl360-pu…
2
14
1
50
4,054
Rupesh Srivastava retweeted
K2 looking strong 💪
3
4
1
19
2,292
One of these models is not like the others! @IFM_AI is not a major tech company or a highly-funded startup. K2 Horizon was our first time building an open agentic model from scratch, with a very small team. It comes with open pretraining data, data pipelines, RL codebase, regularly spaced intermediate checkpoints, and a very detailed tech report coming soon! We never evaluated on FrontierCode so it was a nice surprise to see these results :)
We're launching our leaderboard for FrontierCode 1.1, a private evaluation from Cognition that tests models on completing coding tasks in a way that would actually get merged by maintainers of real-world repositories @cognition announced FrontierCode in June 2026 and updated it to version 1.1 in July. Cognition built the dataset with open source maintainers who spent more than 40 hours per task. Each task requires the agent to make a change to a real open source repository, then checks both unit tests and qualitative tests for what maintainers would require to merge the change. We run FrontierCode 1.1 Main, the primary 100-task set of the benchmark's 150 tasks, using our Stirrup open source reference harness for apples-to-apples comparisons across models. All tasks are private to avoid contamination. Each attempt is graded with both blocking and non-blocking criteria. An attempt that fails any blocking check scores zero, and one that passes them all scores between 70% and 100%. Unlike other coding evaluations, FrontierCode has a more holistic view of success based on real usage. Models need to pass unit tests, but are also assessed on behaviors like making excessive changes and large diffs, modifying files outside the relevant scope, or adding unhelpful tests. We think this approach is valuable in the current context: models are now very capable at coding tasks, but their changes can still require substantial cleanup before a maintainer would merge them. Key takeaways: ➤ Leading models still fail more than 40% of the time: Claude Opus 5.5 (max with fallback) leads at a score of 52% after passing blockers in 58% of its attempts. Nine of the 100 tasks were not passed in any attempt by any of the models we evaluated for launch ➤ The top of the leaderboard is tightly packed: nine models from six labs score between 43% and 52%. MiMo-V2.6-Pro is the leading open weights model at 45%, followed by DeepSeek V4.1 Flash (max) and Kimi K3 (max) at 44% ➤ GPT-6.1 Sol (max) offers the strongest performance for its cost: At $0.81 per task it scores 49%, matching Claude Fable 5.1 (max with fallback). This means it achieves similar performance to leading Anthropic models, at less than a tenth of the cost per task Thank you to @silasalberti, @ybenpan, @cl571128, and the rest of the team at Cognition for building FrontierCode and collaborating with us to bring it to Artificial Analysis - we’re working to add FrontierCode to our coding agent leaderboard very soon.
1
8
479
Rupesh Srivastava retweeted
Towards Looped Models Done Right, Part II: Rethinking at Fixed Points Shaping fixed points in looped language models lets a 1.6B parameter model with a 3x smaller key-value cache match the downstream accuracy of a full-cache fixed-depth baseline. arxivb.org/abs/2610.06833
7
18
1
160
6,652
Rupesh Srivastava retweeted
Reminder: This is just the beginning.
1
9
705
Rupesh Srivastava retweeted
Most decisions don't need words. 🤫⚡ Meet K2-Type-0.9B 🧠: an open decision model. One state + many typed questions in, calibrated probabilities out, in a single forward pass. No text generated. Apache-2.0 weights 👇 huggingface.co/IFM/K2-Type-0… #OpenSource #LLM #AI
12
42
5
289
19,230
Grouped-query attention (GQA) reduces KV-cache memory by sharing keys and values across attention heads. More KV groups give more independent representations, at the cost of larger cache. MoVA adds capacity within the existing groups instead. Sparse value experts introduce additional parameters and nonlinearity into the value computation, while the value vectors keep their original dimensions. Each token selects 4 of 64 value experts, activating just 6.25% of the value-expert pool. The value matrix gains modeling capacity at no additional KV-cache cost.
4
3
27
1,446
Rupesh Srivastava retweeted
The MoVA model is here: huggingface.co/IFM/K2-Horizo… We are going to release the code and technical report soon.🔥
K2-Horizon-36B-A4B scores 25 on the Artificial Analysis Intelligence Index, matching models with over 20× the total parameters while using 4B active parameters per token. These capabilities come from our new architecture MoVA (Mixture-of-Value Attention), which incorporates MoE-based sparsity into the compute of value vectors in multi-head attention. It opens a second axis for scaling sparsity in an LLM, beyond MoE in the FFN module. Importantly, MoVA enjoys the following advantages: • Simple and compatible with efficient attention algorithms, such as flash attention, GQA, and sparse attention • No additional KV cache cost comparing to standard GQA K2-Horizon-36B-A4B available at: huggingface.co/IFM/K2-Horizo…
1
6
1
11
4,719