Working on Gemini @GoogleDeepMind | formerly: PhD @MIT_CSAIL @MITNLP. Opinions my own
New York, NY
Joined September 2014
- Tweets514
- Following831
- Followers1.9K
- Likes1.8K
Pinned Tweet
Introducing our work @GoogleAI CALM: Confident Adaptive Language Modeling 🧘
Large Language Models don't need their full size for every generated token. We develop an Early Exit framework to significantly #accelerate decoding from #Transformers!
🔗: arxiv.org/abs/2207.07061
🧵1/
Tal Schuster retweeted
We’re rolling out Gemini 4 Argon in Antigravity to paid Google AI Pro subscribers today. Demand has been incredible and we've scaled capacity deliberately to get here. We’ve rolled out a separate weekly-only Argon quota for Pro on top of the existing Gemini bucket for 3.8 Flash so you have the best of Gemini for all your tasks.
Tal Schuster retweeted
Gemini 4 Argon from @Google on ARC-AGI (Verified):
- ARC-AGI-3: 97.5%, $4.8K (provider adapter harness), 49.8%, $5.8K (standard harness)
- ARC-AGI-2: 91.3%, $0.32/task
- ARC-AGI-1: 98.5%, $0.16/task
Google DeepMind is the second frontier AI lab to score above 95% on ARC-AGI-3.
Been using Gemini 4 Argon a lot internally and it's truly a game changer!
Research ideas to landings in no time...
Excited to share it externally. More to come
Tal Schuster retweeted
Introducing Gemini 4 Argon – our new frontier model.
It’s built for complex workflows across coding, enterprise knowledge work, and cybersecurity defense – rolling out today to a set of trusted testers through our Fairwind Program.
Let the LLM decide which part of the context to attend to in each decoding step.
The really cool thing here is the model dynamically modifies its own internal functionality real-time simply via the chain-of-thought!
Great paper led by @itsnamgyu and team
Banger paper from Google DeepMind and colleagues.
(bookmark it)
A model reads its entire KV cache on every generated token, even though it ends up attending to a tiny slice of it.
In other words, if you ask about one detail from a 1M-token conversation the global attention layers re-read all of it, per token.
The usual fix is to guess the relevant tokens first with cheap proxy scores, which still costs O(N) every step. Declarative Attention asks the model instead.
The model declares where it needs to look, inside its own chain-of-thought.
In this way, generation splits into three modes: global reads the full context, focus reads one specific region, and local reads only recent output.
The inference engine parses those declarations the same way it parses tool calls and skips most of the cache read.
On zero-shot on off-the-shelf weights across 15 long-context tasks, attended tokens during decoding drop 52.0% on Gemma-4-31B and 31.1% on Qwen-3.6-27B.
Paper: arxiv.org/abs/2609.02737
Chat with Paper: academy.dair.ai/papers/langu…
Tal Schuster retweeted
Banger paper from Google DeepMind and colleagues.
(bookmark it)
A model reads its entire KV cache on every generated token, even though it ends up attending to a tiny slice of it.
In other words, if you ask about one detail from a 1M-token conversation the global attention layers re-read all of it, per token.
The usual fix is to guess the relevant tokens first with cheap proxy scores, which still costs O(N) every step. Declarative Attention asks the model instead.
The model declares where it needs to look, inside its own chain-of-thought.
In this way, generation splits into three modes: global reads the full context, focus reads one specific region, and local reads only recent output.
The inference engine parses those declarations the same way it parses tool calls and skips most of the cache read.
On zero-shot on off-the-shelf weights across 15 long-context tasks, attended tokens during decoding drop 52.0% on Gemma-4-31B and 31.1% on Qwen-3.6-27B.
Paper: arxiv.org/abs/2609.02737
Chat with Paper: academy.dair.ai/papers/langu…
Nice summary by @rasbt!
In Mixture of Recursions the model can decide how many loops to apply per token.
And was really nice working with @raymin0223 @yujin301300 @reza_byt @kim_sungnyun @jenhriver @TalSchuster @adamjfisch @harhrayr Ziwei Ji @AaronCourville Se-Young Yun
A lot of hype around OpenAI's Astra model here on my timeline today. Apparently, this goes back to a new article from The Information, which said Astra is a "recurrent depth or looped transformer".
It's always interesting to read about new or different approaches (including rumors about what the closed labs may be up to), but let's debunk this a bit.
About 2 months ago, I shared the architecture details of Nanbeige, for example, where "Nanbeige4.2-3B is pretrained from scratch on 28T tokens with a Looped Transformer that reuses the layer stack to increase capacity without adding parameters."
Yes, that's it. The looped transformer idea is just reusing layers in the transformer block.
In the case of Nanbeige, the main idea is to reuse the same 22-layer stack (=transformer block) twice instead of once. So, effectively it extends the 22-layer architecture to 44 layers, but without duplicating the weights.
In simple terms, this roughly doubles the size of the model (if we ignore the embedding and output layers for a second). But instead of requiring 2x the storage and RAM to host this model, it stays at the same size since we reuse the components. However, it's almost 2x as expensive in terms of compute, because we run the embedded text through almost 2x as many layers.
Why? In the Nanbeige 4.2 technical report, the researchers found that two passes gave the best trade-off and retained about 75% of the token efficiency of a standard architecture. (More passes gave barely any gains but made the training much slower and much more expensive.)
While, as far as I know, Nanbeige 4.2 is the first notable open-weight model that adopted this approach, the idea goes back to the NeurIPS paper "Mixture-of-recursions: Learning dynamic recursive depths for adaptive token-level computation". Actually, this paper proposes a mechanism that is a bit more sophisticated by adding a learned router that determines whether each token receives one, two, or more passes. So, easy tokens can exit early while harder tokens receive additional computation.
In sum, Astra may be a really good model, but this shouldn't be about this "looped transformer aspect," which is just a tiny architectural tweak.
Also, the statement "the new technique works in a way that obscures some or all of the AI's reasoning, otherwise known as 'chain-of-thought'" is not necessarily true with respect to the looped transformer method. It's possible that The Information journalist refers to some other technique or misunderstood the looped transformer method.
Reusing layers does not by itself suppress visible chain of thought. It adds computation in hidden states before the next token is emitted, just as ordinary transformer layers do.
But based on the information we have, the only plausible interpretation here is that if a model uses more of these recurrent passes, it may need to generate fewer intermediate reasoning tokens. So then more of its computation happens in latent activations that cannot be read as text. But we would get the same effect if we were scaling up the model size, like GPT 5.6 Luna -> GPT 5.6 Sol.
Tal Schuster retweeted
A lot of hype around OpenAI's Astra model here on my timeline today. Apparently, this goes back to a new article from The Information, which said Astra is a "recurrent depth or looped transformer".
It's always interesting to read about new or different approaches (including rumors about what the closed labs may be up to), but let's debunk this a bit.
About 2 months ago, I shared the architecture details of Nanbeige, for example, where "Nanbeige4.2-3B is pretrained from scratch on 28T tokens with a Looped Transformer that reuses the layer stack to increase capacity without adding parameters."
Yes, that's it. The looped transformer idea is just reusing layers in the transformer block.
In the case of Nanbeige, the main idea is to reuse the same 22-layer stack (=transformer block) twice instead of once. So, effectively it extends the 22-layer architecture to 44 layers, but without duplicating the weights.
In simple terms, this roughly doubles the size of the model (if we ignore the embedding and output layers for a second). But instead of requiring 2x the storage and RAM to host this model, it stays at the same size since we reuse the components. However, it's almost 2x as expensive in terms of compute, because we run the embedded text through almost 2x as many layers.
Why? In the Nanbeige 4.2 technical report, the researchers found that two passes gave the best trade-off and retained about 75% of the token efficiency of a standard architecture. (More passes gave barely any gains but made the training much slower and much more expensive.)
While, as far as I know, Nanbeige 4.2 is the first notable open-weight model that adopted this approach, the idea goes back to the NeurIPS paper "Mixture-of-recursions: Learning dynamic recursive depths for adaptive token-level computation". Actually, this paper proposes a mechanism that is a bit more sophisticated by adding a learned router that determines whether each token receives one, two, or more passes. So, easy tokens can exit early while harder tokens receive additional computation.
In sum, Astra may be a really good model, but this shouldn't be about this "looped transformer aspect," which is just a tiny architectural tweak.
Also, the statement "the new technique works in a way that obscures some or all of the AI's reasoning, otherwise known as 'chain-of-thought'" is not necessarily true with respect to the looped transformer method. It's possible that The Information journalist refers to some other technique or misunderstood the looped transformer method.
Reusing layers does not by itself suppress visible chain of thought. It adds computation in hidden states before the next token is emitted, just as ordinary transformer layers do.
But based on the information we have, the only plausible interpretation here is that if a model uses more of these recurrent passes, it may need to generate fewer intermediate reasoning tokens. So then more of its computation happens in latent activations that cannot be read as text. But we would get the same effect if we were scaling up the model size, like GPT 5.6 Luna -> GPT 5.6 Sol.
Tal Schuster retweeted
This is a big one. Full thread coming soon.
huggingface.co/papers/2609.0…
This loops me back to the really nice research on Recurrent Transformers that @raymin0223 did a few years ago during his internship in my team :)
Nice to see this idea getting attention!
This is essentially what we called "depth-wise batching": overlapping requests at different recurrent depths (much higher utilization with adaptive compute).
Recursive Models: arxiv.org/abs/2410.20672
MoR: arxiv.org/abs/2507.10524
Tal Schuster retweeted
Nice to see this idea getting attention!
This is essentially what we called "depth-wise batching": overlapping requests at different recurrent depths (much higher utilization with adaptive compute).
Recursive Models: arxiv.org/abs/2410.20672
MoR: arxiv.org/abs/2507.10524
a few thoughts on "recurrent depth" transformers
the main question: why recurrent depth instead of just scaling depth? it's not faster at inference or training* since you still go through the full "effective depth", the advantage is storage (for kv cache storage btw you could do kv sharing) and a few other things i tried to list below
1) could be better in undertrained regimes (too few data or compute constraints) but i don't think any frontier model is in this regime? could also help convergence somehow, maybe it's just not possible to train a 10T effective param model (so maybe 2Tx5) without tricks like this, but i doubt it..
2) one positive aspect could be better adaptive depth/compute (which is a good direction imo): depending on the token, you skip some recurrent blocks. but you could also just skip layers (or skip moe routing like in longcat flash) and get something quite similar
you could also imagine co-training models this way, sol with 8 blocks, terra with 4, luna with 2 or something like that
3) another interesting thing (to verify?): you could simulate higher batch size at inference by overlapping the computation of depth_block=1 of request 1 and depth_block=t of request 2 on the same gpus, so effective batch size = batch size * number of recurrent blocks (and you could even overlap across models if you do the co-training i mentioned above!), see the picture attached for an illustration of what i mean
still quite skeptical overall, i think scaling depth is just more expressive when you are not compute/data bound but excited to see a potential new direction
*on "faster to train", could be if you only backprop through the last few loops or something like that
Tal Schuster retweeted
Recursive or Looped Transformers are powerful tools, especially when initializing and training them correctly, or when slightly relaxing the layer tying constraint with layer-wise LoRAs.
Check out @raymin0223 student research project @GoogleDeepMind which found the right recipe
🚀 Excited to share our latest research @GoogleDeepMind on ♻️Recursive Transformers!
We make smaller LMs by "sharing parameters" across layers. A novel serving paradigm, ✨Continuous Depth-wise Batching, with 🏃Early-Exiting could significantly boost their decoding speed!
🧵👇
⚡♊
Great milestone. Continuing to cook smart and fast models :)
Gemini 3.7 Flash smashed previous Gemini growth records in its first week, making it our fastest growing model yet. Great to see the huge excitement from our developer community! Now running in Search and @Geminiapp too.
Tal Schuster retweeted
Google has released Gemini 3.7 Flash, improving 4 points over Gemini 3.6 Flash and reaching the Intelligence vs. Time per Task Pareto frontier
@GoogleDeepMind has released its third new Gemini Flash model in three months. Gemini 3.7 Flash (high) scores 56 on the Artificial Analysis Intelligence Index, just behind GPT-5.6 Terra (max, 57) and Muse Spark 1.2 (xhigh, 57)
We benchmarked Gemini 3.7 Flash across all three reasoning levels (high, medium, low) ahead of release. With high reasoning, Gemini 3.7 Flash is a 4 point improvement over 3.6 Flash, while achieving an average Time per Task of 1.7, 40% faster than GPT-5.6 Terra (max). This places Gemini 3.7 Flash on the Intelligence vs. Time per Task Pareto frontier, reinforcing Google’s focus on speed across the Gemini Flash family of models
Key benchmarking results across Gemini 3.7 Flash’s three reasoning levels:
➤ 4 point Intelligence Index improvement: Gemini 3.7 Flash (high) scores 56 on the Artificial Analysis Intelligence Index, up 4 points from Gemini 3.6 Flash. The improvement is driven primarily by gains on agentic evaluations, including Tau3 Banking (+3 points), Terminal-Bench v2.1 (+8 points), and GDPval-AA v2 (+103 Elo). With medium reasoning, Gemini 3.7 Flash scores 53, matching DeepSeek V4 Pro 0813 (max, 53) and GLM-5.2 (max, 53). With low reasoning, it scores 51, just behind DeepSeek V4 Flash 0731 (max, 52)
➤ Pareto frontier on Intelligence vs. Time per Task: Gemini 3.7 Flash produces ~340 output tokens per second, nearly 3x the output speed of GPT-5.6 Terra and GLM-5.2. With high reasoning, this translates to an average Time per Task of 1.7 minutes, placing Gemini 3.7 Flash on the Intelligence vs. Time per Task Pareto frontier
➤ 30% lower Cost per Task than Gemini 3.6 Flash: Gemini 3.7 Flash retains Gemini 3.6 Flash’s standard pricing of $1.50/$7.50 per 1M input/output tokens, however Google is offering discounted pricing through the end of the year at $0.75/$3.75 per 1M tokens. At this discounted price, Gemini 3.7 Flash (high) costs $0.40 per Intelligence Index task, 30% less than Gemini 3.6 Flash and matching Muse Spark 1.2 (xhigh, $0.40). With medium reasoning, Cost per Task falls to $0.26, placing the model on the Intelligence vs. Cost per Task Pareto frontier
➤ Leading performance on AutomationBench-AA and AA-AnalystAgent: On AA-AnalystAgent, our recently released benchmark measuring models’ ability to answer complex questions about spreadsheets and documents, Gemini 3.7 Flash (high) achieves the highest pass^5 score at 60%, ahead of Claude Opus 5 (max, 54%) and Fable 5 (49%). Gemini 3.7 Flash also leads AutomationBench-AA, our benchmark of agentic capabilities in simulated SaaS environments, with a score of 62.7%, ahead of Kimi K3 (max, 53%) and GPT-5.6 Sol (max, 51.2%)
Key model details:
➤ Context Window: 1M tokens, unchanged from Gemini 3.6 Flash
➤ Multimodality: Text, image, video, and speech input, with text output
➤ Pricing: $1.50/$7.50 per 1M input/output tokens at standard pricing. Google is offering discounted pricing of $0.75/$3.75 per 1M tokens through the end of the year. Cached input tokens retain the same 90% discount
Tal Schuster retweeted
Introducing Gemini 3.7 Flash : )
- it is fast!
- 50% lower price than 3.6 flash (through end of year)
- strong intelligence increase in only ~3 weeks (thanks to some awesome algorithmic improvements)
- available in the API, AI Studio, Antigravity, and more!
Tal Schuster retweeted
Gemma 4 MTP just got officially merged into llama.cpp
This means you can use Gemma 4 QAT + MTP for a lightweight + super fast setup. Excited to see what the community builds with it
github.com/ggml-org/llama.cp…
Tal Schuster retweeted
Gemma 4 12B + MTP speculative decoding on mlx-vlm 🚀
We benchmarked MTP on Gemma 4 12B across all 4 modalities in mlx-vlm — and it speeds up everything:
text, image, audio, and combined audio+image, up to 1.72× and 80 tok/s on a single M3 Ultra.
Get started today:
> uv pip install -U mlx-vlm
github.com/Blaizzy/mlx-vlm
🚀 Gemma 4 12B is here!
We partnered with @GoogleDeepMind to bring and optimize their new dense and unifed multimodal model for Apple Silicon.
◈ 12B dense · 256K context
◈ Thinking mode (built-in reasoning)
◈ Vision: dynamic res, OCR, UI + charts
◈ Native audio: ASR + speech translation
◈ Function calling for agents
◈ Text + image + audio, interleaved
Runs local. Get started now ⚡
> uv pip install -U mlx-vlm
github.com/Blaizzy/mlx-vlm
