sigrobotics

Day 28/365 of robot learning. Today I spent some more time experimenting with PPO and trained two policies from scratch in PyTorch using Isaac Lab: a drone learning to fly toward different targets, and another robot learning to defend against soccer shots. For the drone, I trained 4,096 environments in parallel on a gpu, reaching a 100% success rate. The reward shaping was pretty fun to debug because the policy would find ways to maximize rewards while completely ignoring some basic things I hadn't accounted for. For example, in one experiment, the drone would reach its target properly but start spinning uncontrollably because I hadn't penalized rotation. Adding a penalty for body rotation helped fix this. It also made me think more about how hard it is to design proper evals, benchmarks, and reward functions that capture the behaviors we want from robots. Just because a policy performs well on these metrics doesn't necessarily mean it'll generalize to different environments during deployment. I guess this is also why research on policy steerability is pretty important in robotics, since being able to correct a policy's behavior during deployment could help account for things we didn't anticipate during training. I'm also finally done with exams, so I want to take this a step further and try something more challenging. I'm planning to experiment with sim-to-real transfer and see if I can get my robot arm to perform some tasks zero-shot, without any additional real-world training.