Bimanual arm prototype is assembled and running (scripted motion test, no AI/policy) I designed/built this without knowing any CAD software, just claude-as-cad Still a bit rough with exposed wires but good enough Next up: use bimanual umi grippers to train simple policy

A cool thing about our BIND action head: the image features seem to better encode the full trajectories in their 2D feature maps Here I’m plotting the feature map PCA from DINO with BIND action head vs at init: you can see the full trajectory of the task encoded in the feature maps E.g. check out the top left cup stack, you can see the path from the EEF to the cup and from the cup to the plate; For the teapot you can see even for the small teabag the path from the EEF from the teapot lid, to the place we drop the lid off on the right of the teapot, and from the teabag to the teapot Here’s what I like about it: most of the heavy lifting of the action space is being done in the 2D image encoder rather than some beefy action head, which is good because we’re tapping into the backbone that sees the most data rather than our from-scratch action head weights

here’s something small but useful I built with claude-as-cad: I wanted the camera pose and intrinsics relative to some cheap-amazon camera’s CAD/step file so I built a tiny camera calibration jig where it’s mounted relative onto some known place and observes some aruco board in a known place — from it we can get the camera pose and intrinsics relative to the mounting place cad agent passed it to the calibration agent to write the detection script to run this automatically and save it in addition to some policy use cases, one tricky thing is knowing where to place the wrist camera in CAD to make sure it has good observation I had another agent make a localhost program where I can move/drag the camera around in mujoco and view the corresponding render of exactly what the wrist camera will observe when mounted once I lock in the pose, the script sends the camera pose to the cad agent to update the EEF's wrist camera position with it this is one example that I think illustrates why you want a proper CAD agent and interface across the full robotics stack

introducing claude-as-cad i've been actually building robots and parts without knowing any CAD, using a claude-as-cad recipe -- here's how and you can too: language-based prompting for CAD-agents is a good start, but it's imprecise -- these builds are difficult to steer exactly, among other issues instead, you should have claude build you a claude-native CAD interface. i've had success with this and found 3 important ingredients: 1. click to reference -- click any part of the build and the [3D coordinate, mesh face, object id] are copied to your clipboard to paste into chat ➡️ lets you be hyper-specific about what part you are referring to in chat 2. 3D primitive editor -- reposition/reorient any existing object; add a cube to the scene and scale/position for claude to use for reference ➡️ often times its hard to describe a position/transform in language and easier to illustrate it yourself to claude 3. hierarchical notes -- claude organizes separate builds into nested folders and writes skills that survive conversation compacting ➡️ saving different builds in folders with common skills, e.g. servo mounts, means once you get it right and save it, you can always reference it easily link to install to your agent setup below ⬇️

Moving beyond simple pick and place, check out these other tasks. Teapot requires really high precision (tiny grasp point on the lid). Fold towel is bimanual Cup spill is reorienting a fallen cup and wiping up a spill. Cup stack involves rotating the smaller measuring cup into the bigger cup. <= 40 training demos for each task.

Inferring a robot’s 3D state (robot↔camera pose + joint angles / link poses) is still weirdly clunky Introducing Fiducial Exoskeletons: Image-Centric Robot State Estimation! cameronosmith.github.io/fidu… FidEx makes it fast + robust from a single image 🧵