Learning and planning for safe, embodied autonomous systems under uncertainty. Senior Research Scientist @ToyotaResearch. PhD from @StanfordMSL. 日本語 & English
California, USA
Joined March 2018
- Tweets505
- Following768
- Followers704
- Likes1.4K
Haruki Nishimura retweeted
A new negative-result investigation: closer-to-target (non-Gaussian) action priors did not improve the fine-tuning of Large Behavior Models (LBMs), i.e., robot policies pretrained on large, diverse datasets.
Prior work found that replacing the Gaussian prior of diffusion and flow-matching policies with a closer-to-target prior substantially improves policies trained from scratch. We hypothesized that this benefit would carry over to fine-tuning pretrained LBMs.
Across 100K+ simulation rollouts and 1250 hardware rollouts, we found that it does not.
Website: cxu-tri.github.io/non_gaussi…
Paper: arxiv.org/abs/2609.27070
🧵 (1/10)
Haruki Nishimura retweeted
(1/12) How should we rigorously compare robot policies? Comparison is central to robotics research, but is inherently expensive. We introduce NSCORE, a flexible, general-purpose, and data-efficient method for rigorous policy comparison. Accepted to RSS 2026.
Haruki Nishimura retweeted
How should we evaluate robots in the age of foundation models?
We hosted the RSS RoboEval Workshop with folks from academia, industry, and policy to discuss this.
💪 We put together an actionable guide and insights to get you started on robot evaluations. Link below.
🧵1/N
Haruki Nishimura retweeted
We extended the deadline to *June 22nd*! Submit your coolest and craziest (in-progress or completed) works on generalist robot safety 😎
Workshop co-organized with the great team: @ArpitBahety @kensukenk @imp_aa @RobobertoMM @ianabraha @LihanZha
Excited to announce our #RSS2026 workshop: "Rethinking What It Means to be 'Safe' for Generalist Robots"! 🛡️🤖
Have new work or videos of robot safety failures? Submit by June 12! 👇
Haruki Nishimura retweeted
What does it actually mean for a modern robot to be safe? As generalist robots move across tasks, environments, and users, safety must encompass many dimensions: collisions, semantic constraints, perceptions of safety, privacy, and more. Join our discussion at RSS 2026!
Excited to announce our #RSS2026 workshop: "Rethinking What It Means to be 'Safe' for Generalist Robots"! 🛡️🤖
Have new work or videos of robot safety failures? Submit by June 12! 👇
Haruki Nishimura retweeted
Excited to announce our #RSS2026 workshop: "Rethinking What It Means to be 'Safe' for Generalist Robots"! 🛡️🤖
Have new work or videos of robot safety failures? Submit by June 12! 👇
Haruki Nishimura retweeted
Releasing RecGen: a collaboration between @ToyotaResearch, @toyota_europe, and @UvA_Amsterdam tackling a core 3D vision challenge: reconstructing complete multi-object scenes (parts, poses, textures, even occluded geometry) from just 1 to a few RGB-D views.
Trained purely on synthetic data, RecGen achieves SOTA on real-world robotics and 6D pose benchmarks, handling occlusions, symmetry, and complex interactions.
A step toward scalable, high-fidelity digital twins for robotics, and better evaluation and training of generalist policies.
reconstruction-by-generation…
Haruki Nishimura retweeted
I was thrilled to be back at @MIT for the Robotics Seminar! The talk recording is available now:
Rethinking Robot Safety & Alignment in the Era of Generalist Policies
youtu.be/pZM8sgLAye0?si=GG7t…
Haruki Nishimura retweeted
A few interesting rollouts from the Foundry-QwenVLA-2.5B multi-task model on seen tasks in sim – a 🧵. I really like behaviors that involve non-prehensile manipulation, like the little nudges in StoreCerealBoxUnderShelf.
Haruki Nishimura retweeted
Having control over upstream LLM/VLM training is key to training a good robotics model.
We hope VLA Foundry opens the door for researchers and practitioners to answer questions they previously wouldn’t even have thought of asking if upstream pretraining was simply inherited!
Haruki Nishimura retweeted
TRIで最後に関わったプロジェクトである、VLA Foundryがついにリリースされました！異なる言語モデルやビジョンモデルを手軽に試せるだけでなく、Drake + Blenderを用いたシミュレーション環境で複数タスクの評価も簡単に行えます。ぜひ試してみてください！
Haruki Nishimura retweeted
Releasing VLA Foundry: an open-source framework that unifies LLM, VLM, and VLA training in a single codebase. End-to-end control from language pretraining to action-expert fine-tuning — no more stitching together incompatible repos.
Haruki Nishimura retweeted
Great to see @LeRobotHF using STEP as a tool for statistically rigorous policy comparison!
arxiv.org/abs/2503.10966
Releasing the Unfolding Robotics blog!
Time to unfold robotics: we trained a robot to fold clothes using 8 bimanual setups, 100+ hours of demonstrations, and 5k+ GPU hours.
Flashy robot demos are everywhere. But you rarely see the real story: the data, the failures, the engineering.
We’re sharing everything: code, data, and details in the blog → huggingface.co/spaces/lerobo…
Haruki Nishimura retweeted
A really solid step toward scalable, high-quality robot data collection — Raiden, from colleagues at TRI @ZakharovSergeyN (and led by @s1wase) lowering the barrier to entry for bimanual data collection, with support for leader–follower setups and SpaceMouse teleop.
Big highlight - it natively supports camera calibration and integrates TRI’s learned stereo depth model out of the box, with strong improvements over vanilla ZED SDK.
If you're working on robot learning or data collection pipelines, definitely worth a look👇
tri-ml.github.io/raiden/
Our 3D Vision team (3DGR) is releasing Raiden — a data collection toolkit for YAM robots.
Built for scalable, high-quality data: supports leader–follower + SpaceMouse teleop, multi-camera setups, and modern stereo depth (incl. TRI learned stereo).
tri-ml.github.io/raiden/
Haruki Nishimura retweeted
Policy evaluation is a major bottleneck in robotics. We need better tools for statistically rigorous and efficient evaluation!
Check out this great blog post from Haruki Nishimura (@imp_aa) and @MashaItkina on how @ToyotaResearch uses new techniques we have been working with them on.
Haruki Nishimura retweeted
Meet SceneSmith: An agentic system that generates entire simulation-ready environments from a single text prompt.
VLM agents collaborate to build scenes with dozens of objects per room, articulated furniture, and full physics properties.
We believe environment generation is no longer the bottleneck for scalable robot training and evaluation in simulation.
Website: scenesmith.github.io/
👇🧵(1/8)
Haruki Nishimura retweeted
Excited to share our #NeurIPS2025✨spotlight✨and @HogoGoli 's first published paper: "STITCH-OPE: Trajectory Stitching with Guided Diffusion for Off-Policy Evaluation".
Haruki Nishimura retweeted
Highly recommended — a tremendous amount of effort to rigorously test an assumption often taken for granted by the community: "Does a multi-task pretrained vision-language policy actually outperform single-task policies?"
TRI's latest Large Behavior Model (LBM) paper landed on arxiv last night! Check out our project website: toyotaresearchinstitute.gith…
One of our main goals for this paper was to put out a very careful and thorough study on the topic to help people understand the state of the technology, and to share a lot of details for how we're achieving it.
youtube.com/watch?v=BEXFnru5…
Haruki Nishimura retweeted
have been waiting for this release! Robotics needs rigorous and careful evaluation now more than ever 🦾
TRI's latest Large Behavior Model (LBM) paper landed on arxiv last night! Check out our project website: toyotaresearchinstitute.gith…
One of our main goals for this paper was to put out a very careful and thorough study on the topic to help people understand the state of the technology, and to share a lot of details for how we're achieving it.
youtube.com/watch?v=BEXFnru5…