Pinned Tweet
Since I started getting interested in ML I got it in my head that all I wanted to do was one smart thing that I could look back on and be satisfied that I did. Most papers are kinda bad even if they get accepted - the idea is very incremental, or it's just not that good an idea, or it doesn't really matter.
I never was able to do this all through PhD or my time at Amazon. All the papers I did there got into various places, but I never really thought they were actually that good. And I'd pretty much given up on this because Pluralis meant I couldn't really devote enough time to research myself. But in February I decided I didn't care and spend two months focused on a specific problem that had been going round in my head for about a year that I felt we needed to solve, and the solution came to me, and @ChaminHewa picked it up and generalised the approach and ran a bunch of novel experiments I hadn't thought of, and pulled everything together into an actual paper. And yesterday we presented this work at NeurIPS.
This is the first and probably only work I will ever do that for me feels like "ok that was GOOD". I don't care if it racks up a bunch of citations and disperses into the field or not, I don't care if someone repackages the ideas and takes all the credit for it, I don't care. For me there is an internal checkbox that just got ticked after more than ten years of trying. Anyone in ML will understand what I'm trying to say. Special day I'm going to remember for a long time.
"“What do you expect?" he said. "For generations they've been built up to worship competition and the market, productivity and economic usefulness, and the envy of their fellow men-and boom! it's all yanked out from under them. They can't participate, can't be useful any more. Their whole culture's been shot to hell.”"
Received a moving message from an aspiring mid-20s professional that, I think, captures an increasingly widespread despair among younger cohorts:
"It really saddens me when I hear about the work [senior people in the field] are doing. Their work seems interesting, and I'll never get to do it.
The people in their fifties and sixties, they're lucky. I'm even lucky. I feel bad for the kids who are fourteen. I at least got a chance to use my brain for one or two minutes, and that's still priceless. I got to be human. I got to be more than just some puppet who presses Enter so something can think for me.
It's very depressing looking at people who are sixty, who I would love to be in twenty years. But to know that I won't get to do that because Dario said so, right? And, like, that's actually heartbreaking.
It's heartbreaking to know that no matter how hard you try and no matter what you do, you cannot add value as a human being in a year or two's time.
It's like hitting a brick wall. You feel no motivation to build the skill, to get better, to attain wisdom and knowledge. It seems worthless to you. And all the older people are telling you all your efforts might be worthless, and they don't need you anymore. And they can just sit there and, you know, make money and leave you to drown."
Alexander Long retweeted
It's extremely rare to encounter such a thoughtful blog on pretraining and scaling. I really liked the profiling plots depicting a clear yet sufficient picture.
What is not covered in the report is how we scaled model training to be efficient on 768 B200 GPUs.
I explain it in detail here: aleph-alpha.com/en/blog/scal…
A how to guide to scale up model training without wasting your GPU hours. Even if you only have 1 or 2 GPUs and cannot scale, our optimisation philosophy can still make your GPUs go brrr.
I couldn't explain ALL the details of our method in the blogpost, since the big launch came after. You can connect the dots now, the 30B MoE architecture was in fact Kolibri Origin all along.
Alexander Long retweeted
1501: Pope Alexander VI criticizes Gutenberg’s printing press safety
“The art of printing can be of great service in so far as it furthers the circulation of useful & tested books; but it can bring about serious evils l..it will, therefore, be necessary to maintain full control”
Pope Leo criticises Nvidia’s Jensen Huang over AI safety ft.trib.al/azwdiGc
Alexander Long retweeted
We are fortunate to have 3 papers accepted to this years NeurIPS.
The first is AsyncMesh - a technique for asynchronous updates across both data and pipeline parallelism simultaneously.
Second is NuMuon, which shows the low-rank structure we observe in Subspace Networks is also present and can be exploited under Muon-style optimisers.
The third introduces a technique to finetune open-weight models in the Protocol Learning setup. This is significant as prior to this work, this kind of distributed training required architectural changes, and hence starting from scratch.
My position is typically to focus on getting things to work rather than making noise about what needs to be done, but enough people have messaged me in last 24 hours that felt I should write some brief thoughts. The tl;dr is I completely agree with the sentiment but think no-one aside from Pluralis is earnestly moving towards a realistic solution.
The reality is without some mechanism to pool resources for the creation of the models - with hardware that can't be taken away - the situation remains exactly the same, with a faint hope that Nvidia pours enough money into various challengers that the open <> closed model gap stays narrower for longer. I see very few scenarios where an individual group can pull ahead to the frontier (and even if they do, they will be subject to the same dystopian regulation the labs are now speedrunning towards). It must be a community driven effort to achieve this and hence we require a substrate to pool compute, data, and expertise effectively. This looks like p2p/distributed/multiparty training using hardware that has already been purchased to do local AI, thats sitting idle half the time, is physically controlled by individuals and can't be taken away. Repurposing this hardware for creation of the models and not just consumption is absolutely critical.
I don't view most of the other things in this list as meaningfully moving the needle towards a good outcome. RL post-trains as a service.. ok great you made it easier for enterprise clients to use open models. Inference providers serving openweights... ok you host models and make money I don't see what you've changed. Open harnesses... my concern was never that I'd be locked out of the harness layer. Local inference... I don't really care about qwen3.8 27b running at 200 tok/s I want deepseek v4 pro running locally, and I want a better model than deepseek v4 pro. I wanna own and impact the process not the output.
We don't have this yet because its hugely technically challenging - it requires solving fundamental research problems around low-bandwidth training and a re-write of most layers of the training stack. This is what we've been doing with extreme urgency for the last year and what we will continue to do until its working. Once such a system exists however, we will look back and find it strange that private companies ever felt they would be able to monopolize and control something this fundamental.
Alexander Long retweeted
I keep hearing the commercial agreement for Kimi is 30% take. Given how hard it is to host a model of this size you’ll almost certainly need a sophisticated commercial provider.
I totally understand Moonshot’s reasons for this. But clearly, open very much does not mean free.
Alexander Long retweeted
Mark Zuckerberg gets it right:
“The defining question of our age isn’t whether superintelligence will exist, but who will have access to it. Will it be centralized and restricted to a few institutions, or will it be a tool that empowers everyone?”
Concentration of power is the biggest risk of AI. When a small number of labs (working hand-in-glove with the administrative state) decide who has access to which model capabilities, they inevitably shape what can be said, known, and built. That’s not “safety.” It’s control.
As Mark points out, the history of open source shows that broad access and transparency are usually the best path to actual security and resilience. Decentralization creates checks and balances on power. By contrast, centralized alternatives, like bureaucratic approval regimes and mandatory gatekeeping, typically produce regulatory capture and reinforce cartels.
Personal superintelligence in everyone’s hands, with competing models and real data sovereignty, is a far better check on a dystopian future than self-appointed guardians who claim to be “aligned” with all of humanity.
Alexander Long retweeted
hello there the jacobian conjecture is false thanx to my close friend akhil for asking about it and my other close friend fable for working during the world cup final
((1+xy)^3 z + y^2 (1+xy) (4+3xy), y + 3 x (1+xy)^2 z + 3 x y^2 (4+3xy), 2 x - 3 x^2 y - x^3 z): \C^3\to \C^3, has jacobian determinant -2, and sends (0, 0, -1/4), (1, -3/2, 13/2), and (-1, 3/2, 13/2) to (-1/4, 0, 0)
Alexander Long retweeted
If you don’t have at least 1TB VRAM and a VPN to torrent model weights from China, you are asleep at the wheel, and ignoring the uncomfortable truth that open weight models may be banned this year.
Losing access to frontier models will be depressing and make a lot of people feel helpless - a few will have access to 100x productivity, and you will be stuck. Personally I cannot imagine doing work without frontier-level models right now.
Think of the narrative: Chinese model (GLM-6?) with Fable capabilities, no guardrails. It’s plausible.
Everyone is underpricing the chance that open models get banned. What are you going to do if it happens?
Alexander Long retweeted
RL post-training on Macs
14 Macs across 4 countries generate every rollout for the run. Everything's running over the internet. No wire between any of them.
As far as we can tell, this is the first RL post-training run with its whole rollout fleet on consumer Macs.
Alexander Long retweeted
We had an amazing time yesterday at the Protocol Learning Workshop at #ICML2026 in Seoul! It was a full day of learning from researchers working across distributed optimisation, decentralised training, and large-scale LLM training.
Here’s a brief recap 🧵
1/n
Alexander Long retweeted
Second Protocol Learning Workshop at ICML is a wrap.
A packed day of talks and posters covering large scale distributed training and open source AI.
A movement is growing.
Alexander Long retweeted
From Dylan Patel @ semianalysis "There are multiple Chinese model labs who are telling all the inference guys our next model is not going to be open source, we're going to license it to you. Open source is dying, quickly"
BREAKING: Dylan Patel (@dylan522p) of @SemiAnalysis_ says "chips in Europe have less seasoning than chips in America & Mexico."
Plus:
› Data centers & France’s nuclear power
› AI infrastructure over-optimization
› Flexibility vs. specialized infrastructure
› Software-hardware co-design
› Open vs. closed models
› Memory costs & token pricing
› Tokenmaxxing vs. token budgeting
› Haiku scandal
› Favorite chip: 👀
“I think most people don't know what they're doing. They're just buying NVIDIA stuff.”
“It feels like a lot of people are trying to optimize on the current rather than think about where the workload is heading... and that's gonna lead to a lot of wasted infra spend.”
“I think anyone who doesn't token max is gonna get left behind. I think all this token budgeting stuff is loser mentality.”
“Open is dying quickly, unfortunately.”
These never consider the possibility global, independent compute swarms being able to produce models. Completely alters all of this.
Alexander Long retweeted
open source ai dinner at my house with @AlexanderLong on the 14th. Have a couple more spots. dm if you'd want to join!