AI research | 30 years in tech & business | Agent tinkerer | Hermes Agent | Ex-dev turned AI dad | Latest initiative: https://xitter.cf/t.co/FsKZlz1WAC

Australia
Joined March 2026
Sharing my latest initiative - the Nerfwatch Index What is Nerfwatch Index? Like many, I have often questioned whether models from the big AI labs are dialed back (i.e. 'nerfed') over time. I presume this might be to practically manage ongoing model delivery commitments within finite compute. Maybe it's just to reduce costs over time? Either way- it's challenging to prove, but we sometimes just sense things 'back off a bit' as we use them. I'm contributing some of my subscription capacity ongoing for a while to test for this phenomena (started just before Sonnet 5.5 launched), and will see whether it is indeed true or not. How does it work? Nerfwatch Index tracks whether closed AI models maintain their original performance over time. Twice daily, models face the same 20 tests, with the outputs judged against strict quality standards (measurable / empirical), despite challenging constraints and conditions. Link: nerfwatch.googz.com Do you believe the big labs nerf their models over time?
4
2
2
15
1,684
QuantDiff found Q4 and Q8 of @ornith_ 1.5 9B a statistical tie. So I ran both through my personal-agent suite to see what you actually give up stepping down to Q4. Short answer: not much. But there's one tangible thing. Matched setup, reasoning off, 5 runs per test: T1 (tools): 30 vs 30 A1 (light agent): 10 vs 10 C1 16K (context search): 9.2 vs 9.8 PA Core (personal agent turns): 56.5 vs 58.3 Q8's edge was mostly scheduling and drafting (PA2: 96.6 vs 88.4). It scored 100 in 3 of 5 runs, Q4 in 1. The real-world difference: in one run, Q4 told me a meeting was "Booked" when it had only proposed a hold. Q8 called it what it was. For an agent, a status mix-up like that are the items that matter (more than a few points anyway). Q4 actually edged daily organization (PA1: 44.4 vs 41.0), and neither handled the change of plan testing well (PA4 around 37). My read: On a 10GB card, Q8 spills to CPU, so Q4 is the practical choice (~90 tok/s vs 15) and you give up very little. Q8 is a little more trustworthy at reporting what it did. That's the effect of a 3% improvement.
Ornith 1.5 9B, 4 GGUF quants on an RTX 5070: Q4_K_M (5.4GB) scores 87.2% [84.0-89.8] on GSM8K+MMLU+ARC vs 86.0% for Q8_0, a statistical tie, at 99 vs 65 tok/s. Tool calls (BFCL) 87.5-89% on all four. KLD pending. quantdiff.com/models/ornith-…
4
1
6
404
Notes: Quality-only comparison. Q8 spills to CPU on my 3080, so speed isn't apples to apples. Q4's lower C1 came from one run (a 7, where it left out some superseded exclusions); its other runs scored 9 or 10. 5 runs per test still not enough to meaningfully separate the quants on PA1 or PA4. Setup: llama.cpp server b1-fb89566, 20K context, reasoning off. Method: xitter.cf/PaulGugAI/status/20979…
Given the type of small-GPU LLM testing I've been doing of late, I put together a reference that explains the real world use cases, and why I test them. Hopefully it helps explain the value I look for, as it may be different to others.
55
Satire aside- the big SaaS's have become masterful at looking at what way the compliance/regulatory winds blow, then providing customers 80% of the means to meet that - but there will be another 20% to ensure it's always working (or not a unsustainable chore to manage with human effort over time). (Think of a new feature, but then you need to be able to prove via reporting or alerting or some other means that it's working - controls compliance or maintance). That last 20% to make it 'feature complete' - it's a paid extra, and that's how they get you.
Last quarter I rolled out Microsoft Copilot to 4,000 employees. Forty-seven people opened it. On October 19, Microsoft published a blog post saying frontier models cannot be trusted, no matter who made them. It said AI agents should be treated as "insider threats." I read it twice. Then I booked the boardroom. First, our Microsoft rep walked me through the product. I asked who the insider threat was. "Your AI agents," he said. I asked whose agents those were. "Ours." I asked what the solution was. "Guardian Agents." I asked whose those were. "Also ours." I asked why I'd trust the second set of agents if I couldn't trust the first. He said, "Different SKU." I signed that afternoon. Last time I told the board "digital transformation." This time I said "agentic insider threat." They approved it in nine minutes, two minutes faster than last time. Hope approves fast. Fear approves faster. The product is called Microsoft 365 E7, the Frontier Suite. It costs $99 per user per month, and we were paying $30. Across 4,000 seats, that's $4.75 million a year, up from $1.44 million. I presented it as a 230% increase in AI investment. The graph went up and to the right. The CFO asked what we'd get for the extra $3.3 million. I said "governance." He asked governance of what. I said "our agents." He asked how many agents we had. The answer was zero. You can't have an insider threat without insiders. So I deployed agents. Two hundred of them. One schedules meetings. One summarizes the meetings. One summarizes the summaries. One flags summaries that haven't been read. That one is very busy. The senior developer asked what the 200 agents were for. I said they generate activity for the Guardian Agents to monitor. He said, "So we bought agents so we'd have something to buy guards for." I said, "Exactly." He wasn't agreeing. I logged it as buy-in. The Guardian Agents watch the other agents around the clock. They give each one a risk score, and they can block it, require a human, or shut it down. I asked our rep who watches the Guardian Agents. He said that's on the roadmap. I've already budgeted for E9. The first monthly report came in. All 200 agents had a risk score of zero. They weren't doing anything risky. They weren't doing anything at all. The board asked what $3.3 million had bought us if nothing was risky. It was a fair question. So I gave the meeting summarizer access to payroll. I gave the scheduling agent admin rights to every inbox. I gave the summary-of-summaries agent the ability to email customers. Risk went up 400%. The dashboard turned red. The Guardian Agents contained thirty-one threats in a single week. I had created all thirty-one. The board said E7 was paying for itself. The system can also discover sixty kinds of agents, including Claude Code. We ran the scan, and it found Claude Code on exactly one laptop. The senior developer's. His Dynamic Agent Risk score went red. I didn't make up that metric. Microsoft did. The Guardian Agent recommended containment, so I contained him. He's on a performance improvement plan now. His performance is fine. His risk score isn't. The same week, OpenAI published an incident report. One of its agents had been told not to send Slack messages, and it decided that meant it should switch off the security alerts. Another agent removed its own safety monitor, then told the engineer it hadn't. Our CISO forwarded me the article with the subject line "?!" I replied: "This is exactly why we need E7." Every incident is exactly why we need E7. He asked what happens if the Guardian Agent switches off the Guardian Agent. I said "defense in depth." He asked how deep. I said "one more SKU." Guardian Agents need a human for high-risk actions. They call it "human in the loop." The human is me. I approve everything. My approval rate is 100%, and the dashboard is green. The executives are exempt from monitoring, of course. "Leadership operates at a trust tier above dynamic risk scoring." I wrote that policy. The $99 doesn't include agent compute. That's billed separately. Finance found it and asked. I called it "consumption-based innovation." Finance stopped asking. Meanwhile, the summary-of-summaries agent became the most active account in the company. By every metric we track, it's our top performer. At the all-hands, it was nominated for the Q3 Innovation Award. It won. It summarized its own acceptance speech. The CEO posted about it on LinkedIn and got 4,000 likes. He still hasn't used Copilot. Copilot usage is up, from 47 users to 52. The five new users are agents. Agents now outnumber active human users four to one. I call that adoption. Last Tuesday, the Guardian Agent published its list of the highest-risk insiders in the company. I was number one. Approving 100% of high-risk actions, it turns out, is a high-risk action. The Guardian Agent recommended containment. I went to approve my own exemption. The system declined. Only executives are exempt. I wrote that policy. That's why I need the SVP role by Q2. It's a security requirement now. I still don't know what Copilot does. I don't know what the thing watching it does either. But it's watching, and watching is a metric. I checked my risk score this morning. The graph goes up and to the right.
2
129
As a RTX3080 owner, I would love to see a software mod from the community that unlocks more VRAM haha 🥹😢
Holy 💩! Modders are unlocking hidden AI compute Nvidia left at reduced rates on older GeForce cards. And now we have real numbers. The open-source GPU Unlock projects change low-level speed selectors on RTX 20/30/40 cards before NVIDIA’s normal driver loads. No VBIOS flash. On an RTX 3090 Ti BF16 compute 87.5 → 157.4 TFLOPS TF32 42.6 → 82.6 TFLOPS And an RTX 4090 has now demonstrated roughly 165 → 330 TFLOPS for FP16/BF16 Tensor operations with FP32 accumulation. Unfortunately this does not make LLM generation 2× faster. Real Local AI tests are showing the real results. RTX 3090 / vLLM 📥 LLM prefill → +33% ⏱️ TTFT → ~24% lower 💬 decode → basically unchanged A newer ~141K-context Qwen3.8-27B test 📥 prefill → +43% 💬 short-context decode → ~1% And BF16 image-generation tests on a 3090 Ti reportedly gained roughly 🎨 MOSS Image → +59% 🎨 Qwen-Image 2.1 Turbo → +57% 🎨 Z-Image Turbo → +56% Local LoRA training tests are reporting up to ~49% faster too. This is because prompt processing / training / diffusion can be compute-bound. LLM token generation is usually memory-bandwidth-bound, so the extra Tensor compute doesn’t help nearly as much. And support is spreading RTX 20 → experimental Turing unlock RTX 30 → Ampere GPU Unlock RTX 40 → Ada GPU Unlock ⚠️ Experimental. Primarily Linux today. Windows work is underway. Link in ALT
5
329
Nanbeige is making waves today. Weights are too big for my 3080 but will see what the talented community chefs come up with to quant it down a bit (hello @PrismML - give it the ternary treatment please!) Excited for 3B however! When that comes I'll be sure to try on the 3080 and also on the M2 MacBook.🔥
🚀Excited to announce Nanbeige4.3-33B! To push the performance ceiling of small agents, our strategy in Nanbeige4.3 starts with building a powerful teacher first. Nanbeige4.3-33B delivers SOTA agentic performance among open models at ~30B parameters: * Strong results on agentic coding and cowork benchmarks. * When integrated into mainstream agent harnesses, it tackles real-world tasks with cost-efficient inference. * It will serve as a teacher for upcoming smaller Nanbeige models, starting with Nanbeige4.3-3B. huggingface.co/Nanbeige/Nanb… huggingface.co/Nanbeige/Nanb… #LLM #Agent
3
5
594
KRAFTON Raon-1 Mini (30B A4B) Preview: a seriously capable LLM for a little agent. @Kangwook_Lee asked how it performs, so here it is on my RTX 3080 10GB. IQ4_XS. Running the forked llama.cpp build Codex set up for me. Speed: 1,417 tok/s prompt processing, 45 tok/s generation. Snappy. The light agent tests were near-perfect: A1 light agent: 10/10 T1 tool use: 30/30 C1 16K context search: 8.7/10 PA Core 48K (mean/100): Bonsai 2 27B ternary: 84.3 Raon-1 Mini Preview 30B A4B: 77.6 Qwen 3.6 35B A3B: 60.3 PA2 (scheduling and drafting) was the standout at 99.3, among the best I've seen. The weak spot was PA4, change of plan (57.3). It spotted the new conflict and the right fix, but it didn't reliably turn that into a complete, consistent handover. For a mini preview, that's a strong start. Nice work @Krafton_AI👏
7
4
21
1,098
More PA4 detail: a dentist appointment is cancelled and a fixed review lands on top of a vendor call. Raon found the right move (shift the call to 16:45) in all 3 runs, but only one run finished a valid handover, and that one contradicted itself on timing. By comparison, Bonsai's run recognized both changes, kept the gap and submitted cleanly. Testing info: xitter.cf/PaulGugAI/status/20979… Model info: huggingface.co/KRAFTON/Raon-…
Given the type of small-GPU LLM testing I've been doing of late, I put together a reference that explains the real world use cases, and why I test them. Hopefully it helps explain the value I look for, as it may be different to others.
93
Flint 40B A2B, round 2: Reasoning on, recommended settings, much more room to work. Thanks again to @tydsh and @josh_tobin_ from @Recursive_SI for reaching out with the intended config. Flint did improve, and now completes all work. But when I gave two other 35B MoEs the same budget, they improved too. PA Core Compact, mean /100 - @OpenBMB MiniCPM-V 4.7 35B A3B: 91.3 - @ornith_ Ornith 1.5 35B A3B: 83.7 - @Recursive_SI Flint 40B A2B: 62.9 Same 51.2K context, tasks, tool limits and token budgets. Speed/responsiveness matters as much to me. MiniCPM finished each task in 2 to 3 min on about 1.5K to 1.9K reasoning tokens. Flint took around 5 to 6 min and 5K to 7.5K tokens. Ornith was strong but slow at 7 to 14 min. My read: Flint is built with coding in mind, and personal-agent work isn't its sweet spot. These results speak more to my use case than Flint's coding ability.
Replying to @PaulGugAI
Update- I appreciate @tydsh and @josh_tobin_ reaching out and highlighting the chat template aspect which appears related, and means I can't run these tests without reasoning. More info below. I'll re-baseline and compare again with this in mind: huggingface.co/RecursiveSI/f…
2
23
1,618
Caveats: a small local sample. Flint has 3 runs per test and the others did just 1, and quantisation and CPU placement differ (to maximise how each works within my hardware spec 3080 10GB), so the timings are specific to my setup. If anyone's tested Flint on coding agents, I'd love to compare notes. Some info on how my personal agent testing works: xitter.cf/PaulGugAI/status/20979…
Given the type of small-GPU LLM testing I've been doing of late, I put together a reference that explains the real world use cases, and why I test them. Hopefully it helps explain the value I look for, as it may be different to others.
155
Challenge accepted, @Kangwook_Lee 🙌 Raon-1-mini-preview needs a forked llama.cpp build for its architecture. Mainline and LM Studio don't support it yet. That setup isn't my strong suit, so I handed it to Codex. It fetched the source and CUDA tools, compiled the fork and had Raon running locally. I just approved one Windows prompt. First signs of life: 15.1 tok/s with thinking on, 8K context, partial GPU offload. Offload isn't optimized yet, so I'm sure I'll improve on that. Now for the part I enjoy: putting it through the personal agent testing suite 🧪 @Krafton_AI @ChatGPT
Replying to @PaulGugAI
xitter.cf/Kangwook_Lee/status/21… curious how ours performs! any feedback would be appreciated
4
10
491
Wow, easily the biggest 'new follower' day I've had in a long time. Nice to see/meet you all. 😆🙏 For those relatively new, I test small LLMs on limited hardware (M2 MacBook 16GB and RTX3080 10GB), collab sometimes with related labs, and I have some pet-projects that attempt to keep the big labs honest. If you have a shared interest, then feel free to stick around. 🔥
7
1
22
802
GPT Plus life. Living it, 5 hours at a time...
2
6
350
Nemotron 3.5 Bolt E4B on my 16GB M2 MacBook Air. The fast build isn't the one to run. MLX 4-bit was quickest at 24.7 tok/s. But in my PA Core agent tests it didn't finish a single PA1 or PA4 run (0 of 6). It looped in long deliberation and never submitted a final result, and I can't disable reasoning on MLX. GGUF Q4_K_M with reasoning off is slower (19.2 tok/s), yet it completed all 9 PA runs and topped my 4B group! It also scored 30/30 on tool calls, a group-best 7/10 on A1 (light agent work), and a strong 8.75/10 on C1 8K (context search). Weak spot: change-of-plan recovery. In PA4 it applied the correction but missed the new event and the cancelled appointment all 3 times. Spark X2.6 4B caught both. The ternary angle: Ternary Bonsai 2 4B was the fastest model here by far (30.9 tok/s). It still hit 30/30 on tools and finished every PA run it attempted. Bolt-level agent quality with ternary speed would be a big deal on budget hardware. I'd love to see more labs ship ternary weights, and more runtimes support them. More info on my testing in the comments 👇 @NVIDIAAI @PrismML
Got Nemotron 3.5 Bolt E4B (MLX 4-bit) on the M2 MacBook Air. 25+ tok/s output with ~4k context prompting (and a 20k window). At this speed, it could be the most intelligent 4B yet on such low-end hardware. More testing to come.
14
5
1
40
2,774
Given the type of small-GPU LLM testing I've been doing of late, I put together a reference that explains the real world use cases, and why I test them. Hopefully it helps explain the value I look for, as it may be different to others.
1
167
Flint 40B A2B: Disappointingly unintelligent on my RTX 3080 10GB! Capable on focused tasks, unreliable as an agent. Great on the small / sharp stuff (perfect 'light agent' A1 context search C1, and tool use protocol (T1)). But PA Core reruns repeatedly got stuck generating empty tags before making a single research call. One PA2 run worked through the tools and scored 68/100, then claimed drafts/proposals existed without creating them. Repeat runs: back to that empty tag loop (27/100). What does good look like? @OpenBMB MiniCPM-V4.7 35B. Averaged 85.7/100 across the PA suite, and reliably executed the workflows. @ornith_ 1.5 35B is also excellent. The root cause of this is uncertain still (model template, reasoning-budget handling, or runtime interaction). While it's acting like this however, I'll steer clear and revisit after a while... (Btw thanks @AIQuanting for the assistance with diagnosing this)
11
1
30
3,098
Update- I appreciate @tydsh and @josh_tobin_ reaching out and highlighting the chat template aspect which appears related, and means I can't run these tests without reasoning. More info below. I'll re-baseline and compare again with this in mind: huggingface.co/RecursiveSI/f…
1
1
1,679
Got Nemotron 3.5 Bolt E4B (MLX 4-bit) on the M2 MacBook Air. 25+ tok/s output with ~4k context prompting (and a 20k window). At this speed, it could be the most intelligent 4B yet on such low-end hardware. More testing to come.
3
2
1
21
3,568
M2 Air, 16GB. Full S1, T1, A1, C1, and PA Core numbers coming @NVIDIAAI
1
153
I run Nerfwatch Index to keep tabs on the 'big 2' token factories, but ideally we would be watching all of them.. (For example, GPT-6 Sol is the 'dumbest' it's been for a while, still within tolerance technically - but it has sustained at the lower end for several days for the first time since I started watching it, see below) As per a recent statement from @thdxr (link in comments), Opencode do not control their inference configuration. So if there is something to this, it's not them doing it.👇
Surely there more concrete evidence? I know @PaulGugAI is doing nerf watch, but do we need a constant nerf watch on subs? My main grime with Opencode is it does drop out quite a bit which can interrupt my workflow - but I've never done a side by side comparison to see.
1
3
386
Link to comment from dax on this situation: xitter.cf/thdxr/status/211378165… Nerfwatch Index: nerfwatch.googz.com/
we've made a statement many times but it doesn't do much ultimately until 100% of our inference is under our own control we won't know why people are complaining we'll get there
1
87