A team deleted about 168,000 lines of comments from its codebase, and the AI coding agent working on it scored the same either way. Roberto Aranda of OpenStation made a copy of the code with every comment stripped out. He then gave Claude Code the same two tasks on both copies, three runs each. Every run passed lint, type checks and the existing tests. A separate AI judge scored each result without being told which copy it came from. On the small task, scores out of 19 came in at 18.3 with comments and 19.0 without. Tokens were 2.18M against 1.71M. On the bigger task, scores out of 23 were 21.7 and 22.0, with tokens at 9.06M and 9.12M. Aranda is careful about what this shows. He calls the score differences noise and names the small task's 22 percent token saving as the one real effect. He also shows what cherry-picking does. Compare one pair of runs on the bigger task and removing comments looks 30 percent more expensive. Compare another pair and it looks 31 percent cheaper. He says three runs per branch rules out a big effect and not a small one. My read: put a number on how small. On the bigger task the three comment-free runs used 7.01M, 9.70M and 10.64M tokens, a range of 3.63M. The difference between the two branches was 0.06M. So the thing being measured is about one sixtieth of the scatter the measurement produced. That is not a weak result. It is a result this design could not separate from noise. openstation.blog/2026/10/13/…