Mingkai Deng (@mdeng34) header
Mingkai Deng (@mdeng34) profile picture

PhD candidate @LTIatCMU | MSML @MLDCMU | BA Math-Stats + CS @Columbia | World models and agent models | @IFM_MBZUAI @MSFTResearch

San Francisco, USA
Joined September 2016
Photo by Mingkai Deng (@mdeng34)Photo by Mingkai Deng (@mdeng34)Photo by Mingkai Deng (@mdeng34)Photo by Mingkai Deng (@mdeng34)Photo by Mingkai Deng (@mdeng34)Photo by Mingkai Deng (@mdeng34)Photo by Mingkai Deng (@mdeng34)Photo by Mingkai Deng (@mdeng34)Photo by Mingkai Deng (@mdeng34)Photo by Mingkai Deng (@mdeng34)
Really proud to be part of the team that releases the most open frontier model for physical intelligence to date! Shows that foundation model is not so much about pedigree or prestige, but about hunkering down and doing the right things day-in day-out We believe physical intelligence is not about learning the "platonic ideal" of some embeddings. Nor is it about generating dexterous actions alone. We believe it's about "predicting the next world", or faithfully simulating the consequences of actions; the rest will follow. PAN 2 is our step towards that. More to come
Introducing PAN 2: an open foundation world model for physical intelligence. A language model predicts the next word. PAN 2 predicts the next world: what happens after each action, grounded in time, physics and space. Towards AI that simulates “what if?” before it acts. More details: ifm.ai/pan/
4
17
1,066
Open weights are the bare minimum of open. PAN 2 ships with the weights, the inference code, the training code and an Apache 2.0 license. Few open world models come with all four. Inference code lets you run it. Training code lets you make it your own. Apache 2.0 lets you keep it. huggingface.co/IFM/PAN2
11
50
7
439
19,059
Agreed that simulators are the bottleneck for scaling robotics models. Where we differ is whether the solution is rule-based or model-based. The difference between computers and physical world is the latter is not made of code. For physical intelligence, we believe an end-to-end world model is eventually the more scalable solution. Recent work like PAN 2 are making solid progress
in our new blog, we take a look at RoboDojo sim, which are base on NVIDIA's Isaac Sim and it is a good robo benchmark used in Fable 5.2 model card. the video pretty much show what we found. the simulation is not even wrong. I'm now fully understand why robotics experts "hate" the past simulator. but from the first principle, I do believe LLM powered agent could do much much better simulation with the minimal but necessary real-world calibration is the right way we build embodied intelligence. real-world are slow, expensive, hard to scale, hard to get labelled, but sim not. then we partially solve the data problem for robotic and we could scaling up the compute!
2
1
14
2,031
Great work! I often wonder how generalizable human muscle memory really is. Maybe the key is controlling diffusion with more abstract conceptual memory
Excited to release our NeurIPS 2026 paper, “Do Diffusion Models Learn to Generalize Basic Visual Skills?” We asked if diffusion models can generalize skills beyond what they were trained on, or do they mostly interpolate between examples in their training data? This is hard to test on today's text-to-image models, because we don't know what is in their training data. We trained small diffusion models from scratch on synthetic images where we control every value the model sees. We found that they interpolate well but rarely extrapolate.
2
243
Several world model releases in a few months, and still not sure what a world model is for? Here's one answer: it lets you ask "what if I try this?" and see the answer. An LLM can tell you how to climb a route. It can't show you whether you'll make the reach. PAN 2 does both, from one photo of the wall. 🧗 It’s made possible by two reasoners working together: • Conceptual reasoner: plans the climb in words. Which holds to use, and which hand moves next on this route. • Perceptual reasoner: shows visually on the wall the details the plan doesn't spell out: hips, reach, weight shift, timing. They decide whether the climb succeeds or fails. 👇 Concept demo. All climbing footage is generated by PAN 2. Try PAN 2: github.com/ifm-ai/PAN2
3
3
2
17
1,304
Mingkai Deng retweeted
Been playing around with PAN 2 and it's a great model. Thanks for open sourcing it!
A world model is only useful if it predicts what happens in the real world. PAN 2 is built for faithful simulation. Agents plan by simulating ahead, so every step must be right, or errors compound. • Multi-step plans: 🥇 #1 of all models tested, completing 34.0% of full plans, the most of any model • Single actions: close second to MiniMax-H3 on six dimensions of physical intelligence (84.48% vs 87.20%) • Blind human ratings: close second to MiniMax-H3 (Elo 1595 vs 1620) Open weights, training code and inference code under Apache 2.0: huggingface.co/IFM/PAN2
1
7
416
A world model is only useful if it predicts what happens in the real world. PAN 2 is built for faithful simulation. Agents plan by simulating ahead, so every step must be right, or errors compound. • Multi-step plans: 🥇 #1 of all models tested, completing 34.0% of full plans, the most of any model • Single actions: close second to MiniMax-H3 on six dimensions of physical intelligence (84.48% vs 87.20%) • Blind human ratings: close second to MiniMax-H3 (Elo 1595 vs 1620) Open weights, training code and inference code under Apache 2.0: huggingface.co/IFM/PAN2
4
1
2
20
1,670
Cannot agree more. This represents a major confusion of what a world model is and what it does A WM is a simulator of the real world meant for agents to interact with and learn from. This helps scale learning for physical agents without costly real world trials If anything, model like Fable 5.2 is the beginning of generalist agents that can interact somewhat intelligently with the WM. If anything, it makes the WM question more urgent It’s debatable whether World Labs’s Atlas is the right solution though. Aside from space, a WM should also understand the timing of events and physical dynamics. Models like PAN 2 are steps in this direction
最近看到不少人拿 Claude Fable 5.2 的机器人实验,嘲讽李飞飞的 World Labs,甚至说 Atlas 已经被 Claude 吊打了,世界模型就是个骗局 我觉得这种比较非常离谱,完全搞错了研究问题 Fable 5.2 确实很强,在 RoboDojo 的模拟双臂机器人测试中拿到了 64.9% 的成绩,超过了不少专门为机器人控制训练的系统。这说明通用大模型的视觉理解、推理和动作规划能力,正在快速向物理世界迁移。 但 World Labs 的 Atlas 根本不是在解决同一个问题。 Claude 更偏向 给定当前观测和任务,决定下一步怎么行动。而 Atlas 想解决的是:如何从有限的视觉信息中建立具有空间一致性的世界表征,如何生成、重建一个可以从不同视角观察的三维环境 一个更接近 Action Policy / Agent,另一个更接近 Generative World Model。 前者关心机器人怎么完成任务,后者关心能否构建一个可供观察、预测乃至模拟的世界 我相信两者最终会有交集。未来的机器人不仅需要知道该做什么,也需要理解环境如何变化、动作会产生什么后果 Atlas 并没有实现完整的物理模拟。 它目前展示较充分的是空间生成、几何重建和视角一致性,而接触动力学、长期状态演化、action-conditioned simulation 并没有过多体现。 World Labs 被批评的点应该是技术透明度、物理真实性和实际应用价值。 Claude 能控制机器人完成任务,和世界模型没有价值没有关系,be like 一个优秀的驾驶员,不能证明自动驾驶仿真环境没有必要 未来应该是两条路线逐渐融合:通用模型负责推理与决策,世界模型提供空间记忆、环境预测和反事实模拟。
3
34
7,969
Mingkai Deng retweeted
Meet PAN 2, our latest world model. Grounded in physics, time, and space, it pairs an LLM reasoner with a video diffusion backbone to simulate physical reality with action timing and long-horizon coherence. Check it out: ifm.ai/pan/. #worldmodel #physicalintelligence
2
8
217
I do a little surfing in my free time. The hardest call is *when* to pop up — too early and the wave rolls over you, too late and it’s already gone So I asked PAN 2, the world model we trained at @IFM_AI, to show me the same wave, three different timings. It shows what wipes me out, and what leads to a clean ride Simulate first, wipe out less #PAN2 #IFM #WorldModel #Simulation
4
1
18
1,004
Mingkai Deng retweeted
PAN2 pretrained on MILLIONS hours of ego-centric, robotic, and human-motion data, can serve as a new foundation for physical intelligence and robotics. Github: github.com/ifm-ai/PAN2/ Huggingface: huggingface.co/IFM/PAN2
6
9
2
86
50,550
Mingkai Deng retweeted
Video made by PAN 2
8
22
10,036
Mingkai Deng retweeted
Excited to share our latest edition of world model PAN 2 built on a Generative Latent Prediction (#GLP) architecture. A world model needs to operate in an "Aristotelian world" instead of a "Platonic world" where physical intelligence translates to 6 dimensions of capability: obeying the laws of physics, coherence over history, persistence beyond sight, conformity to constraints, invariance to viewpoints, adherence to objective dynamism; and furthermore provides 3 advanced functions: acting as a world simulator that is steerable with actions and plans (long sequence of sections), thereby enabling stimulative reasoning or thought-experiments, and therefore supporting agent to solve complex tasks, and doing so in a fully auditable fashion (hence Aristotelian). A full paper will be posted soon to explain in details how we achieve this. Congratulation to the @IFM_AI PAN team.
Introducing PAN 2: an open foundation world model for physical intelligence. A language model predicts the next word. PAN 2 predicts the next world: what happens after each action, grounded in time, physics and space. Towards AI that simulates “what if?” before it acts. More details: ifm.ai/pan/
25
65
41
183
319,410
Mingkai Deng retweeted
FastVideo team is excited to announce day-0 support for @IFM_AI PAN 2! 🚀 PAN 2 is a foundation world model for physical intelligence, enabling simulative reasoning, where an agent imagines the outcomes of its candidate actions before choosing among them. Simulate 20 seconds in 2 minutes on 8×B300. 10 seconds in 44 s. 📷 FastVideo: github.com/hao-ai-lab/FastVi… PAN 2 Recipe: haoailab.com/FastVideo/cookb… Congrats to the PAN 2 team at IFM! @mbzuai
3
14
3
51
4,751
Mingkai Deng retweeted
Thrilled to announce the PAN 2 world model. At IFM we studied foundation models, one of our research journey is towards world models with the ability to do simulative reasoning. Think of it as a thought experiment, where we "visualize" what would happen, just like Dr. Strange.
Introducing PAN 2: an open foundation world model for physical intelligence. A language model predicts the next word. PAN 2 predicts the next world: what happens after each action, grounded in time, physics and space. Towards AI that simulates “what if?” before it acts. More details: ifm.ai/pan/
3
3
1
36
1,269
Mingkai Deng retweeted
🎉 Great work from the PAN 2 team on an open-weight world model built for real-world simulation, grounded in time, physics and space to predict how scenes evolve as actions unfold. vLLM-Omni brings Day-0 support for PAN 2 text-to-video and image-to-video generation, making it available through an open inference stack.
Introducing PAN 2: an open foundation world model for physical intelligence. A language model predicts the next word. PAN 2 predicts the next world: what happens after each action, grounded in time, physics and space. Towards AI that simulates “what if?” before it acts. More details: ifm.ai/pan/
4
14
4
89
6,952
Mingkai Deng retweeted
Contextualized gene representations work surprisingly well for predicting multigene perturbation effects. I’m excited to introduce EMBER, a pretrained gene-centric foundation model designed specifically to learn context-specific gene representations. A joint work with @jmuiuc and @ericxing. With innovations in single-cell pretraining and dataset curation, EMBER gene representations can be combined with a lightweight probe for single gene knockouts, genetic interactions (double knockouts), and even drug perturbations. Both a 🧵 and a 🎥, got in front of a camera to share some highlights! (1/n)
10
36
4
155
15,413
We're open-sourcing RL360, the RL infrastructure behind K2 Horizon's six domain experts: 8,192 concurrent sandboxes, 150 steps in 7 days, ~1.3M rollouts. Long-horizon agentic RL runs thousands of agents in real sandboxes on tasks that take minutes. Some of those sandboxes crash mid-task, and a crash arrives as the same low score as a wrong answer. Without infrastructure that can keep up, GPUs are underutilized and the model gets penalized for system crashes rather than its own mistakes. RL360 is built for this. We share how it works, what broke at scale, and what we learned in the thread. Code: github.com/ifm-ai/RL360 Run: wandb.ai/mbzuai-llm/rl360-pu…
11
48
13
320
50,469