Reran my previous tower stacking game with image/multimodal inputs.
Kept the text prompt and game-state info the same and tested on the same 10 seeds.
Added an image of the tower and an explanation of the image.
Same as before models can choose to either drop the piece or adjust it.
If the model chooses adjust, it chooses from options to rotate CW 5 degrees, rotate CCW 5 degrees, move left one position, or move right one position.
Average final tower heights across the 10 seeds, comparing our earlier text-only runs → latest image runs:
• TypeSafe Jev 1.14: 9.75 → 11.87
• Perplexity PPLX Decider V1.2: 14.74 → 13.76
• Liquid d1-ultra: 6.58 — a new addition
• Cloudflare Clef-flash: 8.58 → 10.21
Dropping every piece immediately, without adjustment, averaged 8.58.
The gold Drop-only line marks where the final height of the tower would be if the model decided to drop each piece immediately, without adjustment, on that specific seed.
Finishing above it means that run beat the drop-only approach.
The video shows each model’s best seed out of 10
I tested decision models against each other in a 2D physics based tower stacking game. Every 5 pieces, the platform base would tremor.
Each model received the same prompt and was tested on the same 10 seeds.
Models can choose to either drop the piece or adjust it.
If the model chooses adjust, it chooses from options to rotate CW 5 degrees, rotate CCW 5 degrees, move left one position, or move right one position.
This then repeats until the model decides to drop the piece or it hits a decision limit.
These were the best results from each.
The gold Drop-only line marks where the final height of the tower would be if the model decided to drop each piece immediately, without adjustment, on that specific seed.
i.e. If it finishes above that line the adjustment choices the model made benefit the tower compared to simply dropping the pieces
Despite trying many different types of prompts, Clef-Flash always decided to drop without adjustments no matter the prompt.
Is Claude research mode broken for anyone else? Just goes off researching for hours and never creates a report @claudeai