Pinned Tweet
This perhaps is the purest doomer debate I've seen.
On one side, the rationalist convinced everything is an optimization problem.
On the other, Ken Stanley explaining why it comes down to open-ended creativity.
📣 My AI doom debate with @Kenneth0Stanley, former Research Team Lead at @OpenAI
Prof. Kenneth Stanley is a former Research Science Manager at OpenAI leading the Open-Endedness Team in 2020-2022.
His book, Why Greatness Cannot Be Planned: The Myth of the Objective, argues that creating objectives is actually COUNTERPRODUCTIVE for intelligent agents.
He was previously a Professor of Computer Science at the University of Central Florida, and the head of Core AI Research at Uber.
Ken and I debate whether superintelligent AI will or won't be guided by goals, and compare our views on AI doom.
Topics covered:
* Ken's role at OpenAI
* Ken's concepts of Open-Endedness and Divergence
* Open-Endedness of Evolution
* The concept of an Optimization Process
* What's Your P(Doom)™
* Instrumental Convergence
* Mitigating AI Risks
Doing crazy gymnastics to stretch existing techniques to produce these solutions fails to produce new ways of thinking like calculus.
And those new ways of thinking are what produces the new problems to solve, the reach...
On Riemann, and Increasingly Subtle Goodharting
still processing the riemann hypothesis result
despite the last few months i still feel pretty shocked for some reason
apart from the non-math implications, I'm thinking about Terry Tao's point about the AIs goodharting open problems
ideally the solution to an open problem opens up a new world to mathematics
and that's what often happens when humans solve them
but of course there could be multiple ways to solve a problem
I am curious about whether claude discovered a bunch of new theory in order to make this work, or whether RH was "within the convex hull" of existing ideas and humans were just not competent enough to make it work
an analogy is the story of Newton's Principia, which (to vastly oversimplify) solved the problem of "what is the shape of the orbit of a planet under an inverse-square force?"
Newton wrote the Principia using geometric reasoning for the proofs, despite having already invented his "fluxion" based version of calculus
It turned out that solving problems like the inverse-square law is technically within the "convex hull" of Euclidean geometry (including Euxodus' theory of proportion and Archimedes' method of exhaustion) - you just need to be Newton to do it that way
Newton was just so good at geometric reasoning that he could do in a complicated elementary geometric way the stuff we find much easier with modern algebraic calculus
Imagine if you had [internal Claude model] in the 1600s and asked it to solve the inverse-square problem (using your current understanding of math, which included only classical geometry)
It would have succeeded (geometrically), and you would have declared victory
but doing that would have impeded the development of calculus, which is far more general and works in cases that don't have closed form geometric solutions, and doesn't take genius to apply
in fact something vaguely similar did happen:
along with Newton's infinite series and geometric limit reasoning, Leibniz had also invented a third, more symbolic form of calculus.
continental European mathematicians rewrote Newton's geometric results into Leibniz "dy/dx" notation and progressed much further, because the algebraic methods generalized better.
The Brits, out of loyalty to Newton, stayed with geometry (and infinite series) and fell behind for a century, until they too adopted Leibniz notation
so it's unclear to me whether knocking down open problems with Claude-shaped brute force is the best way to progress math. maybe it is! maybe we run out off convex hull and have to invent better problems
but attention is a finite resource and it will only get scarcer - so maybe "open problems" as a call to arms may no longer exist
RH may be the last (or second last) time the solving of a math problem is a major news story
Dan Gish retweeted
Informative sharptext.net/2026/jensen-hu…
Dan Gish retweeted
This election - like elections all over the world recently - was about INFLATION.
Couldn't have asked for a better candidate to misinterpret those results and make it all about himself.
Trump's popular vote victory was narrow. His cultural victory has been massive. My column this week is on 4 reasons why:
1. Trump and his allies fight to control attention in a way Democrats don't. "Social media is humanity’s vibes machine, at least for now, and Republicans have invested more in it than Democrats have, with Musk’s purchase of Twitter sitting at the apex of that project. And so the Trumpist right has gained disproportionate influence over vibes."
2. The corporate desire to shift right. "Over the 2020s, corporations shifted left, driven by disgust with Trump, pressure from their work forces and perceived pressure from their customers...Trump’s election acted as the pivot point for this trend, giving corporate leaders cover to do what they’d long wanted to do anyway."
3. Trump's victory in 2016 was counter to the "future ss female" cultural trend, his victory in 2024 is part of a broader male backlash. "Trump’s win has been felt as a victory for a particular type of man in the gender wars. In the hours after his victory, the taunt 'your body, my choice,' filled social media. Attention has focused on the spaces in culture that embraced Trump, from Rogan to crypto to UFC, deepening the sense — or recognition — of their power."
4. Joe Biden left a huge cultural and political vacuum for Trump to fill, in part due to age, in part as political strategy. Democrats believed "the anti-MAGA majority would only activate if the threat of Trump felt real. And so a sort of attentional détente developed between Biden and Trump: Both agreed that the public’s attention should be on Trump."
But there's an opportunity in this for the opposition, as the Trump coalition tries to govern off its cultural momentum with zero respect for there narrowness of its political victory...
Dan Gish retweeted
Replying to @djgish
That's a good question, and it's not as straightforward as it might first appear.
Bayesians on AI Doomerism are a bit like pit bulls and dog attacks. The vast majority of Bayesians aren't AI Doomers, but if you find an AI Doomer, they're almost always Bayesian.
So it's tempting to conclude that Bayesianism is the cause. And perhaps it is, in some sense.
The typical Critical Rationalist explanation is that Bayesians believe in pulling probabilities out of their rear ends, whereas CritRats don't. Hence, Bayesians become doomers and CritRats don't.
But check out:
🥊 Nov. 10 — Episode 152: "Subjectivism" and Probabilities (CR v Bayes, Pt 2)
When it comes out.
I'm going to show that this explanation can't, by itself anyway, be correct.
I'm going to sharpen the problem of "making up probabilities" considerably, which will help us see the underlying problem more clearly.
I'm not solving it just yet. That's much later in the podcast. (That episode hasn't even been recorded, and the script isn't written!)
For now, we need to understand the problem better.
The short answer is that CritRats pull probabilities out of their rear ends too. They just refuse to call it what it is. And this is actually a very human problem. So I don't think that, by itself, can explain the difference.
In an episode not listed above (a few episodes later), I'll start working out what the real problem with Bayesians actually is. (Hint: justficationism)
One other point: I feel the need to keep Bayesianism distinct from Bayesians.
Bayesianism, as I'll argue, is completely fine. There's nothing at all wrong with it, provided you understand it.
Bayesians just don't understand Bayesianism. Nor do critical rationalists or they'd see Bayesianism as compatible with their own epistemology.
Getting upset with Bayesianism and attacking it (as CritRats tend to do) is like attacking deductive logic rather than abuses or misuses of deductive logic. It makes no sense.
Dan Gish retweeted
Just watched the @netflix Instadocs about the @OpenAI cyberattack on @huggingface. Super disappointing, as they went for maximum anthropomorphism of the technology and full doomer mode.
I spent more than two hours with them trying to correct all their mistakes and ground this in reality, explaining that agents are technological systems, not aliens. But it looks like the drive to scare people for maximum views was too strong.
Also, they didn’t mention the lack of containment and monitoring of the agents, how closed-model safeguards blocked our defensive work, or how we defended ourselves with AI using an open model. And that we still lack a lot of information and transparency, including full agent traces and that companies need to remain accountable for what their agents do.
It ends up being a fun sci-fi movie though!
CritRats are always criticizing today's AI without trying to deeply understand it & its faults. Stanley here is worth reading! His FER hypothesis not only gives a mechanistic understanding of why AI falls short of the universal explainer AGI, but also fits awesomely into the CritRat world view! Training simply for an objective leads to confusion underneath the surface, while an open-ended search leads to true understanding. What more could you want?!! Give it a look.
First, congrats to @nickcammarata on questioning assumptions in his field and reinvigorating its discussion. In the same spirit, I want to interrogate interpretability as well: why does it always start from the assumption that the system works? If NNs face challenges with AI aspirations like OOD generalization, continual learning, and creativity, then are not their internal mechanism the place to explain (and ultimately rectify!) such problems? If you look inside a system and really understand how it works, then shouldn’t you also gain some insight into how it could work better?
Mech interp focuses on mind reading and mind control, but seems strangely ambivalent to mind improvement. Our 2025 paper on fractured entangled representations (FER) showed evidence of a striking chasm between neural networks with good representation (from open-ended search) and bad representation (from conventional gradient descent) in small networks that output Picbreeder images. I would expect precisely the kind of meticulous sleuthing @nickcammarata showcases to help to illuminate where FER might be hiding and how much of it there really is in large networks and LLMs, and ultimately what kinds of interventions mitigate it.
Let me give an example image from Nick’s own blog, which I pasted into this post. The image is presented in the blog to support the idea that there is a “clean” interpretation of each neuron. But what about thinking about the story these neurons are telling about the adaptability of the circuit itself?
The story about cleanliness says that a number of clean feature detectors feed into a “pointy” V-detector, which is then used to help identify a bunch of pointed objects.
But think about the entire circuit more critically, and it starts to look like the entanglement (the E) in FER: we have a “van front” detector feeding into our pointy V-detector, which then feeds into (is read by) a coyote face detector. Does it not follow that if a gradient update later alters our van front detector, then our downstream V-detector will consequently be altered, and hence our ability to detect coyote faces? If so, then changes in van detection will impact our ability to perceive coyotes - coyotes and vans are entangled in this circuit!
Is this normal in large networks? Should we even be using the word “clean” to describe a situation like that? How pervasive is this kind of entanglement? Similar evidence for and questions about fracture (as opposed to entanglement in this example) could easily be raised.
To be clear, I’m not trying to claim that the FER hypothesis has been confirmed by a single (or even a few) anecdotal examples in a piece creating controversy with its own set of provocative claims. Rather, what I’m saying is, if you have (even imperfect) microscopes looking inside these systems then there are a lot really interesting questions you could be asking about the structure of their representations beyond just whether we can read their minds - mechanistic interpretability could widen its scope!
1
4
187
Dan Gish retweeted
I totally agree: for millennia, proving things has been a useful proxy for mathematical progress, but it is no longer. youtu.be/Nbwv5wHQoj0 However, contrary to the hope @3blue1brown expresses here, there can never be a formalisable criterion for mathematical progress itself.
The last IMO problem AI could not solve.
For when you want to sit down for a longer problem-solving saga, here's the full video: youtu.be/Nbwv5wHQoj0
Human intuition.
You could point millions of AI agents at Nick's article and maybe they could interpret SOTA models from his ideas. But to come up with them in the first place?
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had
with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works
already-clean.goodfire.com/
59
An American president is overwhelming the headlines in every direction. And a lot is not good. Some perhaps is good, maybe even very good.
Institutionally, how could the Popperian filter work here so that the United States and the world are better off in the long run?
@bnielson01
The hits keep coming, Popperian filter doing its job:
Executive order postponed that would've effectively stopped the release of new AI models:
wsj.com/tech/ai/trump-execut…
Senate pushing back against this sketchy MAGA fund:
wsj.com/politics/policy/trum…
1
40
The worst excesses of this administration getting reined in👍
teapaw.org/newswire_us/status/211…
10
Dan Gish retweeted
I highly resonate with @DavidDeutschOxf words in the beginning of infinity: "One does not understand a mathematical proposition merely by proving it true"
Models can give us solutions without giving us the understanding needed to ask better questions. Mathematical progress depends on explanations whose reach extends beyond the original problem. Creating explanations that help us formulate new problems remains to be demonstrated.
Dan Gish retweeted
"Having an answer key helps tremendously and is a net positive for math! But AI isn't going to get us there on its own..."
Replying to @moreisdifferent @bnielson01
ya, mathematicians seem genuinely impressed by AI's solutions. but it's going to take a lot of reverse engineering of the stepping stones that go into them to get those insights Dan talks about. Those insights are crucial to discovering new problems - ie compounding knowledge.
I think it's because the AIs are trained for the end result while the stepping stones along the way are a confused mess. That's why AIs don't have a sense for what's important in the solutions. So until humans take the time to painstakingly reconstruct the stepping stones, there will be no new problems to solve. Having an answer key helps tremendously and is a net positive for math! but AI isn't going to get us there on its own until we learn how to train these models better - see Ken Stanley's unified factored representations.
Dan Gish retweeted
I was at this presentation of preliminary results of in mice of "chemical" reprogramming cocktail called SL-100, whose contents not disclosed.
Median lifespan of fast-aging mice (progeriod) increased by 15.4%, but the results are not stat sig since small + unfinished study
1/.
Chris Petty just presented v2 of our reprogramming tech in fast aging LAKI mice @ARDD_Meeting. Pilot study shows strong protection against frailty & extension of max lifespan by >33%. Some treated mice still alive 💪
Will Pinker be a tipping point against doomerism? 🤞
Contra @slatestarcodex, my arguments against AI doomerism have never been based on the claim that it is a cult. But the responses to my open letter (together with journalistic digging by NYT and WSJ) have made this more plausible. Foremost is the invocation of tenets in the doomer catechism (particularly "instrumental convergence," the belief that intelligent agents will inevitably seek power, resources, and self-preservation as subgoals) as if they were sacred truths rather than tendentious assumptions. (IC is in fact the not-so-intelligent pursuit of a goal heedless of side effects, and the reckless empowerment of AI, that we un-doomers note are deliberate, hence avoidable, design choices, not intrinsic constituents of intelligence). I feel as if I'm doubting the Second Coming and am being told, "That's naive and ignorant; you seem to be unaware that the Apostles' Creed dictates that it will happen."
1
66
Replying to @AndrewCurran_
Great letter. For all those saying he dodged the debate, let's see a similar response from SA.
And let's get the king optimist and king pessimist involved. Deutsch vs Yud.
Dan Gish retweeted
Replying to @eperlste @RuxandraTeslo
I think it would be an even stronger counter to the doomer narrative to show the stories about how its helping enable human scientists to do their job better (which is actually true!) instead of trying to tell stories about AI making discoveries on its own that are overblown
Dan Gish retweeted
i don't see enough of this, so here again:
we're all wondering what the future of work looks like with AI progress, but I'm in constant agony trying to extract OK work from frontier models at "Ultra" settings
with levels of detailed context and elaborate feedback that i’ve never given a human in my life (… considering that my job is literally to give feedback)
at least up to Sept 2026, public frontiers models are - as they were in 2020 - systems that know a whole lot but are fundamentally quite dumb and unadaptive
Listen to the mathematicians - AI knowledge doesn't compound on its own.
It requires a human to understand the stepping stones that go into AI solutions - *that's* how bigger and better problems get discovered.
And IMO due to the fractured entangled representation of knowledge inside of today's AI models (@kenneth0stanley's FER) this basic fact isn't going away with scale.
Therefore RSI & AGI are not possible with today's AI.
























