John Kim

HumanCLAW: Can VLMs act through a body? Give VLMs a body, ask it to walk over and sit on the couch. VLMs always fail, in 1,218 long-horizon tasks across 41 indoor scenes: walls block... feet catch on furniture... objects get knocked aside... Real robot testing fuse decision and joint control into one system. When they fail, we may not tell what failed. So HumanClaw strips low-level control error out of the result: the body never falls. In practice: a frozen VLM picks one parametric whole-body skill every 0.5s from egocentric view. A pretrained motion generator turns it into continuous motion. So here, failure = decision failure. Where it breaks: - 1. inefficient exploration: once a target is in view, recognizing it is mostly fine. The more common failure is that the model never brings the target into view at all. - 2. it sees the couch and still can't get there. The model does not know how far away it is or when it has arrived. It stops early, or walks straight past. - 3. it is standing in contact with the couch and still does not sit. Sit success ranges from 90% down to 3.5% across models. - 4. the clearest signal is where the collisions land: mostly on the body parts the model can't see, like legs and feet. VLMs describe a wall correctly, then walk a leg into it. A clear gap in embodied self-awareness! Everything is open: - paper human-claw.github.io - benchmark huggingface.co/HumanCLAW - code github.com/Human-CLAW/HumanC… - leaderboard human-claw.github.io/#leader… Done by the amazing @siyao94 @Kuvvius 👇

Planning with the views: Can VLMs predict how each camera move changes the view, and plan many such moves ahead? We introduce ViewSuite with 6 DoF camera control and ~165K task instances, testing: Path-to-View View-to-Path Interactive View Planning A sharp Planning Gap emerges: + can roughly "track" how camera action changes views - cannot "compose" a plan towards a target view at all We then try to teach VLMs with Reinforcement Learning. - RL cannot teach VLMs such planning ability, only 2.5% success rate with Qwen2.5-VL-7B. + With View Graph Distillation (our RL-Graph-SFT framework), 2.5% → 47.8% Below, we answer these questions: Q1. What are the failure modes? Q2. How can we make RL work? Q3. What has the model learned? Can we open up the model to see before/after? Can such spatial priors transfer to other view related tasks? Led by @James_KKW, great to work with @LINJIEFUN @zhengyuan_yang @shiqi_chen17 @wzenus @drfeifei @jiajunwu_cs Leonidas Guibas, Lijuan Wang. A joint efforts with @StanfordAILab @StanfordSVL @MSFTResearch.