A convincing simulation is a starting point. The useful question is how closely it matches the conditions you want to test.
A few checks worth building into the workflow.
1. Start with measurements. Record mass, friction, actuator response and latency. Even units of the same hardware can behave differently.
2. Compare the same motions. Replay them in simulation and on hardware. Measure the gap and use parameter fitting to reduce it.
3. Track what simulation misses. Cables, liquids, deformable objects and complex contact can be difficult to reproduce. Use real-world observations to check those gaps and guide randomization.
4. Keep variation physically meaningful. Parameters such as mass should usually stay consistent within a rollout. A policy with memory may help infer conditions from recent observations.
5. Inspect unusually good results. Check whether success depends on unrealistic contact, excessive torque or other simulator shortcuts.
For data collection, those gaps are useful. They help identify which conditions need more real-world coverage and which belong in evaluation.
🤖 Made with AI
5
47
A robot can ace pick-and-place and still struggle with the next task. Breaking evaluation down by skill and difficulty helps show where it gets stuck—and gives the next round of data collection a clearer target.
Tired of pick-and-place benchmarks?
Want to explore manipulation tasks of different difficulty levels?
Introducing ManiLadder 🪜, a simulation manipulation benchmark that organizes 112 simulated tasks into 4 calibrated difficulty levels × 8 suites. 🧵
maniladder-benchmark.github.…
2
48
Getting the parts right is a start. Making them work together is a different challenge.
Good to see an eval that checks both! A useful design has to do more than look close.
Today, we're releasing SolidWorks-Bench: can AI models do real mechanical engineering tasks in SolidWorks?
Models get close on individual parts but struggle to assemble them. No model completes a single machine.
Fable 5.2 leads with a 57% mean reward.
Full report below
1
40
As models improve, how should the data brief evolve?
At OrphLux, we work with frontier researchers and data experts to keep up with our clients’ changing needs.
Four priorities keep coming up — expert judgment, realistic tasks, real-world coverage, and clear evaluation criteria.
🤖 Made with AI
6
95
Higher scores at less than a third of the cost per task!
That makes testing on real workflows a lot more practical.
Next up is seeing how much expert review and rework the outputs still need.
Replying to @OpenAI
GPT-6.2 Astra—a frontier model built for your hardest projects. It sets new highs on evals across professional work and scientific research at less than a third of the cost per task of GPT‑6 Astra.
You’ll see the difference in things like slideshows, websites, and 3D designs.
55
🍖A wonderful BBQ night in Los Altos Hills with researchers from @OpenAI , @GoogleDeepMind , @xai , @MicrosoftAI, and @AMD.
Compute efficiency. Infrastructure for agents. AI helping advance AI research.
Happy to be a sponsor. Thanks to @BFC_Global for bringing us together!
Independent review is more than useful when experts can inspect the cases behind a score! For data providers like us, keeping test data linked to grading decisions and model versions gives reviewers a clearer basis for finding gaps.
We’re working with @ApolloResearch, @METR_Evals, @TransluceAI, and @GoodfireAI on embedded assessments: giving outside experts deep access to our internal models and systems to investigate alignment and test our safeguards.
These assessments build on our ongoing alignment and safety research, bringing independent scrutiny to how we identify and manage risks in our systems.
openai.com/index/our-first-p…
1
67
A real factory task is a useful starting point. Shift the part or test recovery from a missed grasp to see how well the approach holds up.
We gave Claude Fable 5.2 a real task from a precision-machining factory and a robot arm.
It got the job done, writing its own code. No teleop, no demos, no task-specific training.
Agentic robots are where physical AI starts doing real work !
48
Excited to see GR00T 2 learn from human demonstrations. Capturing how objects change after each action gives teams a clearer way to evaluate what the robot learned.
Meet GR00T 2, our most advanced, open System One model for robotics research. It learns skills from human demonstrations and predicts how its actions will change the physical world.
Those skills can carry across different robot bodies, reducing the need to teach each robot every task from scratch.
Watch GR00T 2 put that research into motion. 🧵
1
36
Excited to see Atlas open to everyone. We’d love to explore how well a scene holds together as the camera moves. Reference data showing the same space from multiple viewpoints could help evaluate that consistency.
Atlas is now available to everyone!! I’m very excited to see what users can build with our newest world model! 🤩🌎
40


















