Founder / Member of Non Technical Staff @covenant_ai ( @tplr_ai : @basilic_ai : @grail_ai )

Joined April 2014
A small step for mankind, a massive leap for decentralised training... for agency. In the space of 9 months, @tplr_ai went from 1.2B -> 72B. It's never been easy, and has broken everyone on the team multiple times. But I speak for all of us when I say it is the most rewarding thing we have ever done. We have a fraction of the resources. We don't have the PhDs. But Bittensor shows you it doesn't matter. Innovation happens at the edge. We innovate through scarcity. The ones who rewrite the rules are never the ones with the most. They're the ones who refuse to accept the limits they were handed. Bittensor is prophecy. Subnets (@covenant_ai and others) are the tools through which that prophecy is manifested. Next stop: TRILLIONS.
We just completed the largest decentralised LLM pre-training run in history: Covenant-72B. Permissionless, on Bittensor subnet 3. 72B parameters. ~1.1T tokens. Commodity internet. No centralized cluster. No whitelist. Anyone with GPUs could join or leave freely. 1/n
38
33
6
272
75,362
Sam Dare retweeted
turns out that human slop is much dangerous than ai slop
1
1
19
1,200
Minimal harness + strong model = flexible code review agent. That’s the idea behind RevEngine, Laude’s open-source tool now topping @GitHub’s ReviewBench. @NickAJalbert shares how it works, which model the harness helped most, and what else the team learned in the process: laude.org/updates/lab-notes/…
3
3
3
32
3,531
Your neo-lab wants to pre-train a frontier model, but no provider will sell you enough co-located chips? We got you. Our new work shows that SOAP & SPlus achieve strong worker scaling to 96+ DiLoCo workers (e.g., 96 providers) by whitening the update’s covariance. 🧵1/9
6
25
5
98
7,455
Well said. “We got lucky with pretraining. The people who built Wikipedia and Stack Exchange weren’t making training data; they were making things they wanted to exist. But no one is going to inscribe the work of a tax attorney into a reproducible environment for fun.” As a field we’re essentially rapidly constructing the internet-sized corpus for post-training on the fly.
I've been thinking about where the data for the next generation of open models will come from. The internet gave us a remarkable commons for pretraining. Posttraining is different: it consumes tasks, not documents. And that commons doesn't exist. Not yet. openathena.ai/blog/there-is-…
1
2
125
14,140
Kill your priors
"My colleague pushed code that doesn't match what I had in mind." How bad is that really? Consider: it's now much faster and easier to change code. Years ago a wrong decision could cause days and weeks of refactoring. Now? Probably less. Adjust.
1
2
547
Sam Dare retweeted
Scout’s hardest eval: explain the Datadog bill
Meet Scout 1.0, our first AI model built for investigating production incidents. Scout matches frontier-model investigation quality at a fraction of the cost, which makes it practical to investigate more issues, earlier. We trained Scout on realistic simulations built from real-world incidents and production telemetry, using reinforcement learning to reward accurate root causes backed by evidence. Learn more and sign up for the preview: bit.ly/3TJknGO
5
2
1
19
1,126
Sam Dare retweeted
Congrats @bthecohen and @datadoghq on releasing #Scout 1.0! I've been waiting for a post-trained SRE model for a while and #Scout makes my day. Thanks for trying #SREGym ! and thanks for writing a very informative tech report! web-assets.dd-static.net/425…
To check that Scout does not just fit our harness, we also ran the public SREGym benchmark: 90 faults, with the supplied Stratus harness. Scout 1.0 xhigh resolves 80 of 90 faults, competitive with the best frontier models at least 2× cheaper. Scout 1.0 medium beats GLM-5.3 at more than 4× lower cost.
3
7
38
1,915
Worth checking out this model
Google's flagship model, now on Zro: Gemini 4 Argon. - 10% below Google's own API pricing. - $1.80 input / $9 output per 1M tokens. - Enable it once in your dashboard. - GPT 6 Astra level intelligence at 20% the price
1
5
474
Sam Dare retweeted
OpenAI's Navier-Stokes effort used ~10,000 concurrent agents and ~130B output tokens. It made us rethink the scale ahead for RL infrastructure. slime v0.4.0 is a first step toward greater scale, with four architectural changes: • Focus: synchronous training for algorithmic exploration and correctness; fully async training for throughput at scale. • Distributed rollout: spread orchestration across cluster CPUs, with independent generation pools and event loops. • Persistent storage: introduce straw, our new storage middleware for rollout and training data, providing durable queues and shared tensor storage. • Independent recovery: restart Megatron while healthy SGLang engines stay running, preserving accepted rollout work. More on the architecture below 👇 github.com/THUDM/slime/relea… 1/5
6
15
3
130
12,148
Sam Dare retweeted
In Mixture of Expert (MoE) models the cost of moving tokens to their experts can dominate total runtime. At Zyphra research, we use patterns in how tokens are routed to experts to make that communication faster by up to 2.63x on @AMD MI300X GPUs, with the model unchanged.
2
28
4
243
30,748
Order is Heaven's first law
9
480
Important lesson that every era of interpretability research often needs to relearn: you only get reliable understanding/control if you optimize for it. Classifier guidance works even better if you insert it inside the model :) It gives you control and understanding simultaneously.
💥 New paper: AI safety is full of "forbidden techniques" (using CoT to detect reward hacking, using model internals for training, etc). But do they really have a clear scientific basis? I'm not sure. In this paper, we directly optimize models against harmlessness and honesty probes, and it works just fine (if you continuously update the probe!). The models learn how to generate harmless responses to harmful queries and honest responses under pressure to lie. Figuring out how to correctly use interp techniques for training is becoming increasingly important: it's very likely that soon we won't be able to align models using output-based supervision. Future models will just max out all alignment training scenarios, but for the wrong reasons due to their general reward-seeking behavior. To have a chance of aligning future models, we need to do much more research on supervising model training using their internals *without losing monitorability*!
2
2
1
31
1,844
Sam Dare retweeted
Much faster inference means more than low latency. With V2 faster than conventional alternatives far below its size and intelligence class, you can often stop trading off reliability and time. Especially valuable for tasks which *cannot* be parallelized:
8
9
1
75
61,887
Sam Dare retweeted
Thread on why speed matters so much for general AI, why V2's speed can often fundamentally NOT be matched by e.g. (a) multiple parallel normal LLMs, (b) diffusion models, or (c) "decision" models (which is really almost a subset of (a)/(b)), and some caveats/nuance:
Much faster inference means more than low latency. With V2 faster than conventional alternatives far below its size and intelligence class, you can often stop trading off reliability and time. Especially valuable for tasks which *cannot* be parallelized:
1
1
30
2,188
After a long wait, releasing the SYNTH paper! It’s not pretraining, mid-training or post-training, it’s just training: a fully synthetic single-stage pipeline to train workable reasoning models with unprecedented data efficiency.
29
79
11
682
30,358
Sam Dare retweeted
The heads of the biggest AI labs want to agree among themselves on when everyone should slow down. @TheEconomist's piece on that push ends with Covenant-72B, the model we finished training in March on GPUs contributed over the internet, as a reason such agreements may be hard to enforce. What the piece doesn't say is that once training no longer requires one giant, tightly connected cluster, the power to build new models no longer has to sit with a few companies. Globally distributed training and open models are essential tools for keeping that power from concentrating in the frontier labs. They let many more participants, from nations to companies, own a share of the future of AI and take part in building it. economist.com/international/…
2
4
7
895
Sam Dare retweeted
1
3
8
983
Sam Dare retweeted
Good morning! ☀️
9
6
1
290
13,180
Sam Dare retweeted
This is an elitist statement. That there’s no reason to be this fast, and therefore we have to slow down! It’s truly remarkable the level of intelligence I have available on my phone with a $200/month pro plan. However, it’s way too expensive for this to be widely available to billions of people around the world. Tao is worried about the nonlinear nature of progress, except that technological breakthroughs, almost by definition are nonlinear. Otherwise, we would have run out of resources in the world long time ago. People forget how recently we figured out how to produce food at today’s scale. In the 1800s, an important source of fertilizer was an island off the coast of South America. Birds had spent centuries dropping their shit there, building up deposits that people mined for fertilizer. Then came a remarkable invention by the German scientist Haber, which Bosch helped scale industrially. By combining nitrogen from the air with hydrogen, using energy, we could manufacture ammonia for fertilizer. Fertilizer from thin air. There’s another lesson in there about the nature of technology. The same technology that helped create an abundance of food also enabled the production of explosives at scale.
Mathematician Terence Tao: "we have to slow down AI. the pace is insane, and there's no reason to be this fast — no reason at all" It's amazing how willing we are to change everything without any idea what happens afterward These are extremely nonlinear dynamics
13
11
2
168
17,263
i predict math ai progress will organically slow down millennium prize problems are non-renewable resources for pr, soon to be depleted then the big labs won't be incentivized to spend major effort on math ai similarly, deepmind moved on from go ai (alpha go/alpha zero) not long after beating lee sedol (world champ). they could have kept training stronger and stronger go ai, but who would appreciate it? and i think being even better at abstract math won't transfer to other ai priorities (rsi, knowledge work, robotics, etc.) regardless, the work of pro mathematicians will shift from solving problems to understanding ai solutions
Mathematician Terence Tao: "we have to slow down AI. the pace is insane, and there's no reason to be this fast — no reason at all" It's amazing how willing we are to change everything without any idea what happens afterward These are extremely nonlinear dynamics
46
6
5
154
49,023