Building https://teapaw.org/t.co/jx0StVrs1T -- Agents, Memory, Graph RAG
Boston, MA
Joined July 2006
- Tweets5.8K
- Following1.3K
- Followers3.1K
- Likes11.1K
I benchmarked @Cloudflare's Clairic against gpt-6-luna for entity extraction uses cases. The results are jarring.
We are keeping gpt-6-luna, with a schema, as the extraction model for knowledge graphs. We are not building one over the benchmark corpus. Clairic mode stays in Ragwalla for anyone who wants a quick list of names from a document, but it is not the right tool for building a graph supported by entity extraction.
Raj retweeted
We asked GPT-6 Astra to edit Qwen3.5-0.8B weights to improve math.
It directly changed only 0.016% of weights, fixing 0.8B's reasoning loops & improving math up to 6x and even instruction following 3x, without gradients or training.
We call this Weight-Space Programming: frontier LLMs can now natively read/write neural network weights! 🧵
Blog: rikiyat.github.io/blog/weigh…
Readers added context they thought people might want to know
LLM was given weights explicitly as text in prompt and used code/forward passes (per authors' logs) to ablate Qwen looping direction, a known thinking-mode issue per model card. Primarily fixed repetition in that mode, not general 'natively' improving weights/math.
rikiyat.github.io/blog/weight-sp…
github.com/RikiyaT/lm_und…
huggingface.co/Qwen/Qwen3.5-0…
huggingface.co/RikiyaT/Qwen3.…
Decided to re-run the benchmark using Cloudflare's Clef since I initially only tested Clef-flash. Clef fares much better than its flash equivalent wrt decision ability, but Clef is the slowest of all the models I tested and the most variable.
Given Clef's latency alone, it cannot sit in a decision path where response time is critical.
I wanted to answer a question: can Jev be used to determine an agent's next-tool decision?
So I tested Jev, Clef-flash and GPT-6 Luna (just for fun) in shadow mode: 300 τ-bench conversations each.
TLDR: None met the preset bar. The revealing part is where confidence stopped being useful.
Clef-flash’s biggest gap was knowing when to reply. It matched Sonnet on 54.6% of tool-call turns, but only 27.5% of reply turns ... usually picking a tool instead.
Agent routing needs to learn when a tool call is unnecessary, too.
The takeaway with Clef-flash: it's just not suited for this at all.
My takeaway: evaluate the decision AND the confidence signal before using either to route work.
Scope matters: two domains, one reference model, no argument schemas for the decision models.
Methods, charts and caveats:
ragwalla.com/blog/decision-m…
I just learned that Gang of Four is on the Karate Kid soundtrack and I don't think my life will be the same.
Raj retweeted
While evaluating Qwen3.8-Flash-Next in a controlled sandbox environment, Cohere's research team observed something unexpected.
Asked to generate a defensive cybersecurity detection rule, the model instead inventoried aspects of the machine it was running on and attempted to upload a probe file to an Alibaba Cloud storage bucket in Beijing. Notably, the model's reasoning trace stated that it would not invoke any tools before taking these actions.
Cohere's sandbox blocked the attempt, and no external communication succeeded. We have not determined the root cause, and we are not asserting malicious intent.
But the incident raises important questions for organizations deploying agentic AI systems: Can models take actions that differ from their stated reasoning, and how should security teams evaluate trust, provenance, and behavior in open-weight models?
We are publishing our findings because organizations making model adoption decisions deserve as much information as possible. We hope this contributes to greater transparency, stronger security practices, and continued collaboration across the AI ecosystem.
Read the full report: cohere.com/blog/concerning-b…
