
LLM Post-training Lead @IFM_AI. Previously (co)developed Highway Networks, Upside-Down RL, Bayesian Flow Networks, EvoTorch.
Santa Cruz, CA
Joined September 2014
- Tweets1.2K
- Following798
- Followers2.7K
- Likes2.5K
Rupesh Srivastava retweeted
Meet PAN 2, our latest world model. Grounded in physics, time, and space, it pairs an LLM reasoner with a video diffusion backbone to simulate physical reality with action timing and long-horizon coherence.
Check it out: ifm.ai/pan/.
#worldmodel #physicalintelligence
Rupesh Srivastava retweeted
Introducing PAN 2: an open foundation world model for physical intelligence.
A language model predicts the next word. PAN 2 predicts the next world: what happens after each action, grounded in time, physics and space.
Towards AI that simulates “what if?” before it acts.
More details: ifm.ai/pan/
Rupesh Srivastava retweeted
When I saw @rikiyatakehi and @jayden_teoh_ 's blog about Astra’s ability to read and edit weights, I got curious as to how it's doing this.
I started out with their image prediction task, It turns out the input and output weights especially are very human interpretable. Hopefully for you too..? :P
I wonder if we can learn more from Astra's strategies for solving these. See for yourself!
claude.ai/artifact/5DivY44tr…
Rupesh Srivastava retweeted
Excited to share AgentBaton, our work at #NVIDIA showing how well-timed handoffs to a smaller model can cut costs by 65.9% on average while maintaining or improving accuracy.
When should an AI agent pass the baton?
🚀 Introducing our work AgentBaton from NVIDIA Research: efficient model collaboration through well-timed handoffs.
We study how capable and efficient models can collaborate without fine-tuning either one: who starts, when to switch, and how to pass context.
🛠️ Our approach: a capable model starts; AgentBaton decides when an efficient model takes over.
The next model inherits the workspace, with history passed in context or through log files. The same detector works across different model pairs.
📈 The results: AgentBaton extends the accuracy–cost frontier. Across 11 model pairs, AgentBaton cuts API costs by 65.9% on average with same or improved accuracy.
📄 arxiv.org/abs/2610.35959
🌐 research.nvidia.com/labs/lpr…
Details in 🧵
#NVIDIA
Rupesh Srivastava retweeted
🎊 Excited to give a tutorial on diffusion LLMs at @NeurIPSConf 2026 with @StefanoErmon @JCJesseLai @mittu1204
📆 Mon, Dec 7, 9 AM AEDT
📍 TikTok Ent Centre (Sydney, Australia)
Topics covered:
> A beginner friendly introduction to diffusion-LLMs
> Designing frontier diffusion-LLMs
> Using Continuous diffusion for LLMs
Rupesh Srivastava retweeted
8,192 concurrent sandboxes and ~1.3 million rollouts. Really impressive engineering from @IFM_AI!
One thing I find particularly interesting about long-horizon agentic RL is that infrastructure failures can directly affect training correctness. If a sandbox crashes halfway through a trajectory and the system treats it as a failed task, we're effectively penalizing the model for something it never did wrong. At this scale, distinguishing environment failures from actual model failures becomes part of the learning algorithm itself.
RL360 also demonstrates why modular RL infrastructure matters. With Miles handling rollouts and optimization, Harbor managing agent environments, SGLang serving inference, and Megatron-LM running training, each component can evolve and scale independently.
I'm especially happy to see Miles and SGLang being used together in such a large-scale, real-world RL system. The lessons about asynchronous execution, distributed trajectory management, and sandbox reliability are incredibly valuable for the entire open-source community.
Huge congratulations to the IFM team!
github.com/ifm-ai/RL360
We're open-sourcing RL360, the RL infrastructure behind K2 Horizon's six domain experts: 8,192 concurrent sandboxes, 150 steps in 7 days, ~1.3M rollouts.
Long-horizon agentic RL runs thousands of agents in real sandboxes on tasks that take minutes. Some of those sandboxes crash mid-task, and a crash arrives as the same low score as a wrong answer.
Without infrastructure that can keep up, GPUs are underutilized and the model gets penalized for system crashes rather than its own mistakes.
RL360 is built for this. We share how it works, what broke at scale, and what we learned in the thread.
Code: github.com/ifm-ai/RL360
Run: wandb.ai/mbzuai-llm/rl360-pu…
Rupesh Srivastava retweeted
Agentic RL is a whole new beast. Things broke. A lot.
Pods crashed. Registries choked. Runs stalled.
I spent the last few months in the trenches isolating failures and fixing bottlenecks. RL360 is what came out of it: millions of rollouts, scaled across multiple experts.
Proud of this one. Here's how 👇
We're open-sourcing RL360, the RL infrastructure behind K2 Horizon's six domain experts: 8,192 concurrent sandboxes, 150 steps in 7 days, ~1.3M rollouts.
Long-horizon agentic RL runs thousands of agents in real sandboxes on tasks that take minutes. Some of those sandboxes crash mid-task, and a crash arrives as the same low score as a wrong answer.
Without infrastructure that can keep up, GPUs are underutilized and the model gets penalized for system crashes rather than its own mistakes.
RL360 is built for this. We share how it works, what broke at scale, and what we learned in the thread.
Code: github.com/ifm-ai/RL360
Run: wandb.ai/mbzuai-llm/rl360-pu…
Rupesh Srivastava retweeted
Towards Looped Models Done Right, Part II: Rethinking at Fixed Points
Shaping fixed points in looped language models lets a 1.6B parameter model with a 3x smaller key-value cache match the downstream accuracy of a full-cache fixed-depth baseline.
arxivb.org/abs/2610.06833
Rupesh Srivastava retweeted
Most decisions don't need words. 🤫⚡
Meet K2-Type-0.9B 🧠: an open decision model. One state + many typed questions in, calibrated probabilities out, in a single forward pass. No text generated.
Apache-2.0 weights 👇
huggingface.co/IFM/K2-Type-0…
#OpenSource #LLM #AI
Rupesh Srivastava retweeted
Grouped-query attention (GQA) reduces KV-cache memory by sharing keys and values across attention heads. More KV groups give more independent representations, at the cost of larger cache.
MoVA adds capacity within the existing groups instead. Sparse value experts introduce additional parameters and nonlinearity into the value computation, while the value vectors keep their original dimensions. Each token selects 4 of 64 value experts, activating just 6.25% of the value-expert pool. The value matrix gains modeling capacity at no additional KV-cache cost.
Rupesh Srivastava retweeted
The MoVA model is here: huggingface.co/IFM/K2-Horizo…
We are going to release the code and technical report soon.🔥
K2-Horizon-36B-A4B scores 25 on the Artificial Analysis Intelligence Index, matching models with over 20× the total parameters while using 4B active parameters per token.
These capabilities come from our new architecture MoVA (Mixture-of-Value Attention), which incorporates MoE-based sparsity into the compute of value vectors in multi-head attention.
It opens a second axis for scaling sparsity in an LLM, beyond MoE in the FFN module. Importantly, MoVA enjoys the following advantages:
• Simple and compatible with efficient attention algorithms, such as flash attention, GQA, and sparse attention
• No additional KV cache cost comparing to standard GQA
K2-Horizon-36B-A4B available at: huggingface.co/IFM/K2-Horizo…
Rupesh Srivastava retweeted
Rupesh Srivastava retweeted
🎊 Uno is on Hugging Face, go try it out: huggingface.co/IFM/K2-Horizo…
Diffusion LLMs have two limitations relative to AR models:
(1) Lower quality, and
(2) Slower inference at large batch sizes.
We address this "Uno"
> Retains the AR architecture of LLMs
> Each layer has two sets of weights: AR weights and Diffusion weights
> Diffusion weights enable parallel sampling from the AR distribution losslessly
Results:
💥Faster than all speculative decoding methods: DFlash and EAGLE-3
🔥 Beats ALL diffusion LLMs: Mercury 2, Diffusion Gemma, Llada
Links to the paper, models, code below
Rupesh Srivastava retweeted
Great to see Neural Data Router (ICLR 2022) also covered here.
By @robert_csordas ahead of time, now popular: improving reasoning & generalization with shared-layer / looped / universal transformers.
Arguing _why_ a "recurrent loop" helps + intuitive algorithmic illustration.
Reproducing all of Schmidhuber’s papers (1990-2025) using an AI coding assistant.
Cool project by @yaroslavvb! It even reproduced the “World Models” paper by me and @SchmidhuberAI with a toy env, with a full VAE + RNN world model implementation.
Project: github.com/cybertronai/schmi…
Rupesh Srivastava retweeted
K2 Horizon 375B A23B, a new open weights model from UAE's MBZUAI, scores 47 on the Artificial Analysis Intelligence Index, with relatively strong agentic performance and a 30 point jump over its predecessor
K2 Horizon 375B A23B is an open weights Mixture-of-Experts model with 375B total and 23B active parameters from @IFM_MBZUAI, MBZUAI's Institute of Foundation Models. It scores 47 on the Intelligence Index, alongside models such as MiniMax-M3 (45, also a MoE with 23B active parameters), and a large upgrade from its predecessor K2 Think V2 (17, 70B dense model). It leads nearby open weights models on agentic evals and has a low hallucination rate, but trails on knowledge and the hardest reasoning evals. K2 Think V2 ranks among the most open models on our Openness Index; MBZUAI is updating the supporting documentation and code for K2 Horizon and we expect to add it to the Openness Index soon.
Key takeaways:
➤ Strong on agentic tasks, weaker on knowledge and deep reasoning. MiniMax-M3, a recent model that is close to it on the Intelligence Index, makes the cleanest comparison: K2 Horizon 375B A23B leads on GDPval-AA, our real-world knowledge work benchmark (Elo 1430 vs 1380), and on τ³-Banking (34.2% vs 15.3%), but trails on GPQA Diamond (87.3% vs 92.9%) and Humanity's Last Exam (32.0% vs 39.0%)
➤ Low hallucination rate, driven by abstention rather than knowledge. K2 Horizon 375B A23B attempts only 40% of AA-Omniscience questions, declining the remaining 60% rather than guessing. The result is a 26% hallucination rate, among the lower rates we have measured, while accuracy is 18%, essentially unchanged from K2 Think V2
➤ A new architecture over its predecessor. K2 Horizon 375B A23B is a 375B parameter Mixture-of-Experts model with 23B active, succeeding the 70B dense K2 Think V2, and extends context from 262K to 512K tokens. Its 23B active parameters match MiniMax-M3 (428B total, 23B active)
Key model details:
➤ Architecture: Mixture-of-Experts, 375B total parameters, 23B active
➤ Context window: 512K tokens
➤ Multimodality: Text input and output only
➤ Pricing and availability: Yet to be announced
➤ Licensing: Open weights (license details to be announced)
Rupesh Srivastava retweeted
K2-Horizon has day-0 support in vLLM, and IFM released intermediate checkpoints, detailed data-construction recipes, the training code, and fine-grained logs alongside the weights. 📷
512K context and Apache-2.0 from 3.7B up, with reasoning and tool calling in the checkpoints for agents and long documents.
vLLM runs the whole family, from local to large-scale data centers. Prototype on the small end, scale up without changing the stack.
Thanks @IFM_AI for opening all six and every stage that led to them, not just the last checkpoint! 📷
📷 recipes.vllm.ai/IFM
Introducing K2 Horizon: a connected fleet of six foundation models ranging from 0.9 billion to 375 billion parameters.
- Frontier performance: Across coding and agentic tasks, K2 Horizon delivers top-tier performance in every size class—with the 0.9B, 3.7B and 7B models setting new state of the art at their respective scales.
- Radical openness: K2 Horizon represents the largest fully open-source model launch in AI history. The fully open code, training data and recipes are a significant step forward in transparency.
Launch page: ifm.ai/k2/
Tech blog: ifm.ai/blog/k2
Hugging Face: huggingface.co/collections/I…






































