Tobi

Congrats to @AtharvaRNaik !! Some of Atharva's co-authored work from past 2 years that I really like and encourage everyone to read: 1. arxiv.org/abs/2606.21811 Training Mini Critics for Fast Local Code Review 2. openreview.net/forum?id=VLgF… Evaluating Code Slop in Agentic Code 3. aclanthology.org/2026.findin… Programming Benchmark Inspired by Historical Linguistics, tests reasoning independent of domain knowledge 4. aclanthology.org/2025.emnlp-… Strong and Weak LLM Agents Collaborating to Write a Repo

Reviewing agentic coding traces late October from a model family's new release, @ArnhavD observed something that quickly kindled some discussion among us at @PatronusAI ... He noticed the model's reasoning had nearly no articles and dropped a good number of other "function words" or "stop words" or, to call it simply, filler words, like the "be" in "be doing" and the "to" in "wants to fix". The traces sounded, in his words, like Kevin Malone's lines from The Office: “Why waste time say lot word when few do trick?” (“Need inspect repo. Grep.”) Another parallel is that they felt like stuff people would type into telegrams historically e.g. "Arrived safe. Weather rainy." or "Child born, mom healthy." Intrigued now by the stark difference we had eyeballed b/w end of '26 and early '26 shortness of this kind in model reasoning, we sought to answer if this phenomenon held up more widely, and how to even compute it? Specifically, was there a clear -ve drop Δ b/w newer late '26 vs older early '26 models from the top #3 open source families by open source use = Deepseek 4* Flash, GLM-5* & Kimi K3* ?? We computed what we called filler word % (articles, auxiliaries, self-talk, pronouns etc) in reasoning coming out of newer vs older models, when solving the same 20 tasks from three popular code benches. In all the 3 families, filler share in reasoning basically halved, going from 25-30% to the low teens (10-14%)! To conclude, we had 2 notable side-observations: a) This didn't always cut total token usage (in some cases it rose) going from older --> newer model b) The terseness is not quite of the human kind: self-talk e.g. "Wait!", "Eureka!!", "Maybe I'm right!" persists even when other filler words are getting dropped ... Note that observations to this effect have already been made by many others, particularly in the case of Astra and Deepseek-Flash-4.1 traces by @scaling01 and GLM-5* by @Tim_Hua_ , as also posited by @karpathy as a possible endpoint of outcome driven reinforcement learning happening too well back in late '24 to early '25 itself Nonetheless, we felt an independent repro through controlled comparison based on some formulation of filler word or "shortness" b/w older and newer models from the 3 top model families would enrich the narrative! Full blog link in comments, glad to hear thoughts, questions and feedback!

These days, we increasingly both: a) generate policy documents 📃 using LLMs / agents e.g. Agents.md , Protocol.md working with Claude Code or Codex in your repo b) make agents act per policy 📃 taken from other sources, potentially themselves auto-crafted, e.g. Skills or Regulations or LLMs.txt from websites These policy docs 📃 could, however, be wildly ambiguous, and w/o a human 👨‍🦰 present proactively flagging these OR writing them in the first place, we do need automatic yet fair measurement of ambiguity! Exploring this direction, Sally Dong @dsallydt and co-authors @LukeThorburn_ @Werdnamai @bakkermichiel et al propose DELTA, a measure and framework of document ambiguity, accepted and presented today at #EMNLP2026 ! DELTA defines ambiguity as extent of disagreement b/w people on concrete cases or situations which invoke the policy This can be done with actual cases + human raters OR by synthesizing these with LLMs and marginalizing over the simulated decisions , a.k.a LLM-DELTA ... DELTA beats LLM-as-a-judge and word concreteness scores both at being discriminative b/w consensus-inducing and disagreement-causing policies Lastly, they also show that LLM-DELTA 🤖 and Human-DELTA 👨‍🦰 are positively correlated .. An Added + : DELTA is designed in a way that it can benefit from scaling in LLM ability as more representative cases + persona can be produced and hence aggregate i.e. DELTA improves too!