Collaborative Robotics Lab

We also observe that, using L2D2, expert users were able to teach long-horizon tasks to the robot with fewer physical inputs and less teaching time as compared to teleoperation.

In our user study, novices end-users with no prior experience used L2D2 to convey the tasks to the robot correctly while spending significantly less time in teaching the robot as compared to teleoperation and achieved a better performance as compared to the drawing baselines.

With expert users, we compare L2D2 to state-of-the-art drawing approaches (RT-Trajectory and Sketch-to-Skill) and direct end-effector teleoperation. L2D2 matches teleoperation performance with fewer physical interactions, while outperforming existing drawing-based baselines.

We use Principal Component Analysis (PCA) to find the optimal camera view minimizing information loss from 3D to 2D. From sketches on these images, we learn mappings to 3D trajectories, then ground them with a small set of real-world corrective demos to close the sim-to-real gap.

To truly enable the users to teach the robot asynchronously, we leverage vision–language segmentation to synthetically generate diverse environments, so users can provide a large number of demonstrations without physically resetting the scene.

Users can leverage our proposed interface to sketch desired robot trajectories on 2D images of the robot’s workspace (e.g. on a tablet). This interface enables the users to specify grasping points and gripper orientation in addition to providing trajectory drawings.

Learning from demonstrations is key in robotics, but physical demonstration & teleoperation are slow and don’t scale well. We introduce L2D2, a sketch-based interface + algorithm that lets users teach robots by simply drawing trajectories on a 2D image of the robot environment.