Pinned Tweet
Thanks. The fact that it works so well for ARC is also a clue to the extent of generalization in these models. It was not at all obvious that it would work originally because the model is never trained on the actual answer. @fchollet 's ARC-AGI benchmarks are an invitation to understand efficient generalization. I feel strongly that gradient updates at test time are the doorway to a new level of AI (smaller, more capable models with actual memory).
Test-time training was popularized during the ARC Prize 2024 competition, after being explored in particular by @MindsAI_Jack and team. To date, I believe ARC 1-2 are the only datasets where TTT strongly outperforms. It would be interesting if TTT started becoming more mainstream. I believe it has great potential.
Was I right?
Replying to @kimmonismus
All the labs are going to release extremely good models ahead of @AnthropicAI's IPO would be my guess. Ant is holding back Fable 5.5 for when that happens. Going to be an interesting 9 days.
Jack Cole retweeted
I'm calling it. Neon is better than Opus 5.5.
This model is absolutely brilliant.
I missed this. Thanks for this analysis. @mhmazur Is the human baseline an average or the best of all humans?
Across the 25 ARC-AGI-3 public games at max reasoning, GPT-6.2 Astra used 64.1% fewer actions than the human baseline and 5.2% fewer than GPT-6 Astra with the provider adapter harness:
1
7
579
Wow. It would be an extremely rare human who could do this (maybe none on ARC 3). It would be interesting to see the comparison against the human distribution. Good ideas for the future benchmarks Greg.
This is the first 100% we've seen on v3
The way we've measured models GPT-1 through GPT 6.2 won't hold up for GPT-7+. We need a categorically new benchmark
Three ways to create benchmarks:
1. Out-of-distribution tasks
A models knowledge coverage looks like swiss cheese (not exactly, but you get the idea). You can always find gaps in knowledge as you niche your topic. Ex: Animals identification > Mammals > Dogs > Herding dogs > Border collies
This is the *easiest* way to create a new benchmark, it's also the *least* robust. Depending how much the labs care, you're only 2-3 months of model releases away from your swiss cheese gap filling in.
2. Increase task difficulty
Take an existing domain of tasks which models saturate and make the problems harder.
Hire PhDs and experts to come up with problems. This is operationally intensive and harder to execute.
You want to avoid cheap difficulty (a larger maze) and lean into rich difficulty (more reasoning required).
3. Measure the frontier
This is the highest signal (imo) way to test models, it'll be the defining benchmark for post-AGI systems.
Instead of measuring performance on a static set of tasks, measure the rate at which models push the frontier of knowledge forward.
The ultimate environment is reality/physics. Net new knowledge here is proof of innovation. However, physics is hard to benchmark with, sythenetic environments will do ok here in the short term.
To be clear, this is an incredibly high bar to hold models to. Maybe the highest?
All RSI, prosperity, advanced capability questions are downstream of AI's ability to create net new knowledge.
Once you have a system that can autonomously create new knowledge in any domain (like human as a collective can) then every other domain is a harness/training session away.
TLDR: Static task completion is no longer an adequate way to measure frontier models. Rate of learning is the last hurdle. Net new knowledge (what humans don't know) is the most robust setting to measure this.
11
712
Jack Cole retweeted
“If you’re cramming all this stuff into your brain all the time, then you don’t have time for [new thoughts] to happen,” Earl Miller, a professor of neuroscience at MIT.
Headphones, podcasts, music, and streaming are distracting us from our own thoughts.
nypost.com/2026/11/16/tech/h… – at Somerville, MA
Strong results.
Inkling 2 Small from @thinkymachines on ARC-AGI (Verified):
- ARC-AGI-3: 2.6%, $704 total (provider adapter); 0.3%, $858 total (standard)
It beat DeepSeek V4.1 Flash 1017 on ARC-AGI-1 (96.5% vs 94.5%) and ARC-AGI-2 (81.7% vs 78.1%), but trailed on ARC-AGI-3 (2.6% vs 5.8%).
Jack Cole retweeted
Astera Institute Residency applications are open. Very unique opportunity to build your project in a stimulating and supportive environment! Check it out, and tell your friends about it!
Applications are open for Astera Residencies, now through January 7, 2027! Up to $2M in funding, based out of our Emeryville hub. This round, we’re sharing a few specific project requests. Read more and apply here: astera.org/residency/
Reshare with folks who should see this!
Jack Cole retweeted
Sufficiently advanced agentic coding is essentially machine learning: the engineer sets up the optimization goal as well as some constraints on the search space (the spec and its tests), then an optimization process (coding agents) iterates until the goal is reached.
The result is a blackbox model (the generated codebase): an artifact that performs the task, that you deploy without ever inspecting its internal logic, just as we ignore individual weights in a neural network.
This implies that all classic issues encountered in ML will soon become problems for agentic coding: overfitting to the spec, Clever Hans shortcuts that don't generalize outside the tests, data leakage, concept drift, etc.
I would also ask: what will be the Keras of agentic coding? What will be the optimal set of high-level abstractions that allow humans to steer codebase 'training' with minimal cognitive overhead?
Jack Cole retweeted
"Latent Feedback Is a Compute and Data Amplifier"
Repeatedly training LLMs on the same data gives diminishing returns, but latent feedback provides a more efficient way to use that compute.
This paper revisits Latent Feedback Transformers, which feed each token's final hidden state into the next token's input, letting the model to reuse information across consecutive positions.
Across models from 60M to 778M params, latent feedback saves up to 48% of training compute or 41% of unique training data in measured comparisons at matched validation loss. A new scaling law also predicts performance on a held-out 778M model within 0.3%.
alphaxiv.org/abs/2611.latent…
Antitrust suit against DRAM manufacturers.
wccftech.com/federal-judge-r…
Jack Cole retweeted
looks very interesting! Congrats @StphTphsn1
"A method to automatically discover symbolic local learning rules"
–– expertly led by Andrea Perin (@perina_ndrea), & in collaboration with Fabio Anselmi (@Fabio_Ans)
🔗 arxiv.org/abs/2611.12248
accepted at #NEURIPS2026 (Concept & Feasibility Track)
Interesting. It seems like Gemini Carbon might get better performance on ARC from additional test time compute rather than stronger skills.
Gemini 4 Carbon from @Google on ARC-AGI (Verified):
- ARC-AGI-3: 99.1%, $4.6K (provider adapter), 65.1%, $7.2K (standard harness)
- ARC-AGI-2: 94.2%, $0.38/task
- ARC-AGI-1: 98.5%, $0.16/task
Carbon beat its predecessor Argon by 18% points on ARC-AGI-3 in the standard harness.
4
9
818
I have been saying for a long time that Google will be back at the frontier. They have arrived with Carbon. Great tests from @bijanbowen.
youtu.be/0GSvcg9wncc?is=ScP-…
1
23
2,493
Jack Cole retweeted
I have never seen so many companies trying to kill their technology.
🤖 Made with AI
Jack Cole retweeted
Gemini 4 Carbon is rolling out today. It’s built for sustained, long-horizon tasks, reasoning deeply through complex software engineering, cybersecurity, and scientific discovery.
Coming globally via the Gemini API in @GoogleAIStudio and Gemini for business, and for Ultra subscribers in @antigravity, the @GeminiApp, and AI Mode in Search.
At AI Safety Co
- The only people who could be better than us at keeping us from killing you are the politicians.
- We will take all your jobs, but maybe the politicians will help with that?
- We might cure cancer and other diseases for a few years before we kill you all.
P.S., May we please build a data center in your back yard?
P.P.S., Yes, we know the politicians haven't been able to do incredibly simple things like get rid of the yearly time change, but they can probably cut our chances of killing you by some percentage, because they are great at AI research. Your jobs are toast though, so hopefully they pay you something. We know they care about you.
2
4
320























