Sami Jawhar

ALT We measure average speedup achieved on an improved version of KernelBench, comparing our “KernelAgent” to the KernelBench leaderboard. The pink line represents KernelAgent using each model of 4o, sonnet, o1, and o3-mini, and the red line represents the performance of the best soluton found by models so far. Using the same models as the original KernelBench paper, KernelAgent achieves a speedup of 1.81x vs KernelBench's 1.05x. We attribute this to scaffolding improvements and higher test-time compute spend -- as these tasks are easily checkible, we do best-of-k across agent attempts and spend around $20 per task per model (over an estimated <$1 for results on the KernelBench leaderboard).