Zhengming Yu

✨RigidFormer: Learning Rigid Dynamics using Transformers — accepted at #NeurIPS2026! 💻 Code is now open source: github.com/Frank-ZY-Dou/Dyna… We propose an object-centric Transformer for learning rigid-body dynamics, enabling efficient and scalable mesh-free simulation. It achieves higher accuracy and faster simulation. We hope this architecture can be extended to a wide range of rigid-body dynamics scenarios and potentially to deformable-body dynamics by adapting its rigid-alignment component. Contributions are very welcome! It’s been a pleasure working with this wonderful team: Minghao Guo (@GuoMh14), Haixu Wu (@Haixu_Wu_1998), Doug Roble, Tuur Stuyck (@TuurStuyck), and Wojciech Matusik (@wojmatusik)! See you in Atlanta! #Simulation #PhysicalAI #MachineLearning #Robotics #AI #DeepLearning #Simulation

I kept seeing #GPT-6 Astra modelling results on here and got curious enough to build the thing myself: an agentic pipeline that turns a plain RGB video into 3D assets. The living room for robots @MIT_CSAIL. Input: one handheld phone walkthrough of our lab kitchen (~20s). Nothing else — no depth sensor, no CAD, no asset library. ~1 day (I actually slept overnight), including human-in-the-loop. Live: frank-zy-dou.github.io/kitch… It is still not perfect — thin and shiny things are still weak, a few objects are drafts, the room shell needs another pass. The loop, roughly: - Monocular video → metric scan (ViPE): camera poses + depth. This is the measuring instrument, not the output. - GPT-6 lists what should exist as separate objects, then open-vocabulary detection + tracking gives per-object masks; each object is fused and measured in metres. - For every asset, GPT-6 works in its own sandbox with tools: it writes the object as a program in a small Blender DSL (closed solids, PBR colours, hinges/drawers), builds it, renders it over the original video frames, compares it with the scan points in 3D, and iterates. - A separate GPT-6 session is the verifier — it can render the model on any frame it wants and must justify every complaint with a frame. The modeller never grades its own work. - A completeness pass renders the whole modelled scene from the video's own cameras, puts it next to the real frames, and says what's still missing. What the detector keeps missing (a row of identical cabinets) gets placed geometrically instead. - Out comes MJCF/URDF with joints. Now, adding simulation to this env.

Introducing ✨RigidFormer: Learning Rigid Dynamics with Transformers - our attempt to scale learning-based physical dynamics with Transformers. RigidFormer learns rigid dynamics with Transformers. It is a mesh-free, object-centric Transformer for multi-object rigid-body contact dynamics from point clouds. Learning physics with purely neural simulators, without relying on traditional physics engines, is an important and widely studied problem. Prior SOTA methods often use graph neural networks for accuracy and generalization, but still struggle with efficient, high-fidelity simulation at scale. RigidFormer uses only point inputs, matches or outperforms mesh-based baselines on standard benchmarks, runs much faster, generalizes across point resolutions and datasets, and scales to 200+ objects. We also show a preliminary extension to command-conditioned articulated bodies by treating body parts as interacting object-level components. RigidFormer is mesh-free: it does not require mesh connectivity, SDFs, or vertex-level message passing, making it well-suited for point-cloud observations and scalable simulation. This architecture can also be adapted to learn soft-body dynamics by replacing the rigid-body module (differentiable Kabsch alignment). 🎬See our video for more details. Many thanks to my amazing collaborators: Minghao Guo @GuoMh14, Haixu Wu @Haixu_Wu_1998, Doug Roble, Tuur Stuyck @TuurStuyck, and Wojciech Matusik @wojmatusik. Project page: people.csail.mit.edu/frankzy… Paper: people.csail.mit.edu/frankzy…

Excited to share that our work NeuralActuator: Neural Actuation Modeling for Robot Dynamics and External Force Perception has been accepted to #RSS2026! Your robot — even a low-cost one — can feel external forces without torque or tactile sensors. TL;DR: NeuralActuator is a neural actuator model that jointly predicts 1️⃣torque to capture the nonlinear and time-varying current–to–torque relationship of low-cost servos, 2️⃣external contact forces (and force detection gates) for sensorless force perception, 3️⃣and motor conditions that indicate each motor’s operating regime. Here is a fast-forward video clip ⬇️ We are also covering more robots like LeRobot-S101 and Franka Panda. More details coming soon.

SAD: Soft Anisotropic Diagrams for Differentiable Image Representation has been accepted by #SIGGRAPH2026 Check it out, and huge congrats to Lucky! @Luckyballa #SAD represents an image as a soft, anisotropic, differentiable diagram over learnable sites. Each pixel is modeled as a softmax blend over its top-K nearby sites under a site-dependent distance, yielding a differentiable partition of unity with explicit ownership and content-aligned boundaries. A GPU-friendly top-K propagation scheme keeps the cost constant per pixel, enabling fast fitting at matched or better quality. Classical geometric structures can still inspire fresh perspectives in modern visual computing. Voronoi and Power diagrams have long been elegant tools for 3D shape analysis, reconstruction, and geometric reasoning; here, related diagram ideas, with connections to Apollonius-style diagrams, are explored for image representations. Homepage: luckyiyi.github.io/SAD/ arXiv: arxiv.org/pdf/2604.21984 #SIGGRAPH2026 #SIGGRAPH #CV #Vision #Graphics #CG – at Cambridge, MA

We have seen many works unlock the power of pretrained models for images and videos🏞️. But what about human motion🕺💃? Can we leverage a pretrained motion prior for a wide range of downstream tasks? Yes!! UMO is a simple yet effective framework that, for the first time, unlocks the priors of a motion foundation model (i.e., HY-Motion) for 10+ tasks, including editing, reaction generation, stylization, trajectory control, obstacle avoidance, keyframe infilling, and more. Amazing work! @xiaoyan_cong and @kunkun0w0. 🏠Webpage: oliver-cong02.github.io/UMO.… 📄 Paper: arxiv.org/abs/2603.15975 With the growing number of tools for transferring SMPL motion to humanoids, we hope it could also become a source of skills for humanoid robot learning. #Graphics #Motion #Animation #AIGC #GenerativeAI #Vision #3DV #Robotics #Robot #Humanoid #Learning #GenAI #Animation

Hello 🇸🇬 Singapore! At #ICLR2025, I’ll be presenting our work 🎲DICE from @LingjieLiu1's lab! With DICE, one can explore hand-face interactions 📷 — this feedforward method simultaneously estimates hand and face poses, contact points, and deformations from a single image using a Transformer-based architecture. Come join us! 📍 Hall 3 + Hall 2B #130 Poster Session 6 🕒 Sat 26 Apr, 3–5:30 p.m. 🎥 Check out the video for more details! Huge thanks to all our amazing collaborators who made this possible: @qingxuan_wu @xu_sirui , Soshi Shimada, @chenwangcw , @Proof_Yu , @YuanLiu41955461 , @_cheng_lin , Zeyu Cao, Taku Komura, @VGolyanik , Christian Theobalt, Wenping Wang, and @LingjieLiu1. #ICLR25 #ICLR2025 #AI #Animation #CV #CG #Interaction – at Plentong, Johor

DICE 🎲was accepted by #ICLR2025. With DICE, one can learn more about hand face interactions 👤🤏. This end-to-end method also enables better scalability to learn and model hand-face interaction. Congrats to @qingxuan_wu ! Check the video for more details :) #ICLR #ICLR2025

#SIGGRAPHASIA #SIGGRAPHASIA2024 #ACMTOG 🐟 We introduce Collective Behavior Imitation Learning (CBIL), a scalable, self-supervised framework for learning fish schooling behaviors from videos, to be presented at SIGGRAPH ASIA 2024 Tokyo (journal track)🗼! 🦈Reproducing realistic collective behaviors presents a captivating challenge, as traditional rule-based methods fall short in realism, while data-driven approaches rely on hard-to-acquire motion trajectories. 🐠CBIL first leverages a Masked Video AutoEncoder (MVAE) to map 2D observations to expressive latent states. Then an adversarial imitation learning framework with bio-inspired rewards is developed for stable and realistic motion generation. We demonstrate CBIL's effectiveness across various fish body shapes and its capability to detect abnormal behaviors from in-the-wild videos (real2sim4real). I like this attempt to “inject data priors” for Visual Imitation Learning, especially given the challenges of obtaining ground truth 3D motion for imitation. A heartfelt thanks to our amazing intern, Yifan Wu @Littlecobbler! Always remember the excitement we felt during those sleepless nights! And special gratitude to our incredible collaborators: Yuko Ishiwaka, Shun Ogawa, Yuke Lou, Wenping Wang, @LingjieLiu1, and Taku Komura. 🏠Project Page: frank-zy-dou.github.io/proje… 📑Paper: ACM Transactions on Graphics dl.acm.org/doi/10.1145/36879… #AI #ImitationLearning #Animation #Animation #AI #CrowdAnimation #BehaviorAnalysis #CrowdMotion #Graphics #CG #Motion #AIGC #MotionSynthesis We showcase real-world videos and synthesized results, aligned for clearer visualization (rather than as reconstructions). – at Philadelphia, PA

Got five papers accepted by #ECCV2024 @eccvconf ! Huge thanks to all my collaborators! 😃 See you in Milan 🇮🇹 Summary of Selected Works (I made a fast-forward for them 😄) - [Shape Generation] Surf-D: Generating High-Quality Surfaces of Arbitrary Topologies Using Diffusion Models, ECCV 2024. - [Efficient Motion Generation] EMDM: Efficient Motion Diffusion Model for Fast, High-Quality Human Motion Generation, ECCV 2024. - [Controllable Motion Generation] TLControl: Trajectory and Language Control for Human Motion Synthesis, ECCV 2024. - [Avatar Generation] Disentangled Clothed Avatar Generation from Text Descriptions, ECCV 2024. Project Page: Surf-D: yzmblog.github.io/projects/S… EMDM: frank-zy-dou.github.io/proje… TLControl: tlcontrol.weilinwl.com/ SOSMPL: shanemankiw.github.io/SO-SMP…

(3/3) The clothes 🎽👖👗🩳generated by 🚀Surf-D🚀can be used for virtual try-ons with high quality and fidelity. You can just use sketches to generate whatever clothes you want, then put on your own avatar and use captured/generated human motion to animate them :D #SurfD #AIGC