research scientist @SakanaAILabs | views are my own
Joined February 2020
- Tweets597
- Following478
- Followers1.7K
- Likes11.3K
Pinned Tweet
My latest work got accepted to #ICLR2026! I am really grateful for my co-authors!!
We propose an alternative approach to end-to-end backpropagation!
1/
Makoto Shing retweeted
Introducing the Sakana AI Frontier Intelligence Group 🪷
sakana.ai/frontier-intellige…
Current AI systems are incredibly capable, but is intelligence “solved”? And if not, what’s missing?
At Sakana AI’s Frontier Intelligence Group (FIG), we believe that there are still breakthroughs to be made in AI. The Transformer and language modeling may be incredibly powerful, but it doesn’t mean that better alternatives don’t exist.
Natural intelligence still beats artificial intelligence across many dimensions. Agents lack the deep insights and creativity of humans. Individual models require far more data than the brain to learn robustly, and require far more energy to run. If we set these as targets, what kinds of AI systems could we develop?
Research at FIG has sought to address the gaps between natural and artificial intelligence. Here are some of our works, and the fundamental research questions that motivated them:
• Continuous Thought Machines: How can we improve information processing by leveraging temporal dynamics?
• Augmented Lagrangian Predictive Coding: How can local learning solve multilayer credit assignment?
• Sparser, Faster, Lighter Transformer Models: How can we massively increase data efficiency and generalization?
• The AI Picbreeder Experiment: How can we make artificial open-ended systems?
• Smart Cellular Bricks: How can physical systems achieve collective intelligence and self-repair without a central brain?
We hope that this encourages other researchers to also explore different paradigms, and take a leap of faith with us. After all, in the words of a dear friend of ours, “greatness cannot be planned”.
Makoto Shing retweeted
I continue to put my money where my mouth is when it comes to research freedom, this is me doubling down.
Come join us and you'll have the time, resources and talented collaboratiors to work on that mad idea that's being rattling around the back of your head for years!!!
Introducing the Sakana AI Frontier Intelligence Group 🪷
sakana.ai/frontier-intellige…
Current AI systems are incredibly capable, but is intelligence “solved”? And if not, what’s missing?
At Sakana AI’s Frontier Intelligence Group (FIG), we believe that there are still breakthroughs to be made in AI. The Transformer and language modeling may be incredibly powerful, but it doesn’t mean that better alternatives don’t exist.
Natural intelligence still beats artificial intelligence across many dimensions. Agents lack the deep insights and creativity of humans. Individual models require far more data than the brain to learn robustly, and require far more energy to run. If we set these as targets, what kinds of AI systems could we develop?
Research at FIG has sought to address the gaps between natural and artificial intelligence. Here are some of our works, and the fundamental research questions that motivated them:
• Continuous Thought Machines: How can we improve information processing by leveraging temporal dynamics?
• Augmented Lagrangian Predictive Coding: How can local learning solve multilayer credit assignment?
• Sparser, Faster, Lighter Transformer Models: How can we massively increase data efficiency and generalization?
• The AI Picbreeder Experiment: How can we make artificial open-ended systems?
• Smart Cellular Bricks: How can physical systems achieve collective intelligence and self-repair without a central brain?
We hope that this encourages other researchers to also explore different paradigms, and take a leap of faith with us. After all, in the words of a dear friend of ours, “greatness cannot be planned”.
Really interesting work from @jeffreyseely on training neural networks without backpropagation using predictive coding!!
Check it out! 🎉
Makoto Shing retweeted
sharing a new paper!
Augmented Lagrangian Predictive Coding
I think one of the coolest unsolved problems is how the brain does credit assignment.
thread 🧵
Introducing PC-ALM, a local-learning alternative to backpropagation.
Our method trains 1000-layer neural nets using only local dynamics, and without backprop.
Blog: pub.sakana.ai/pc-alm/
Standard deep learning relies on backpropagation. The brain, however, cannot implement backpropagation, at least not exactly. How can a physical system, such as the brain, solve multilayer credit assignment without explicit use of backprop?
We look for inspiration in two related fields: distributed optimization and NeuroAI.
In NeuroAI, predictive coding asks each neuron activation to solve an energy-based inference problem instead of using a standard forward pass. That inference step can be implemented as energy-minimization dynamics on local prediction errors.
This perspective -- each layer as a dynamical system -- has proven promising, but performance of predictive coding hasn't scaled well with depth. Credit signals at far ends of the network struggle to diffuse into internal layers.
We turn to distributed optimization, generalizing predictive coding to use an augmented Lagrangian instead of energy. This motivation stems back to a classic 1988 paper by LeCun, showing that the Lagrange multipliers of a deep network can be identified with gradients of a supervised loss. The augmented Lagrangian then bridges LeCun's perspective to the standard predictive coding that is used in NeuroAI.
We find that this new perspective yields a natural PC-like alternative to backpropagation, resulting in a method we call PC-ALM. PC-ALM differs from PC in that it introduces dual neurons (Lagrange multipliers) as part of the layer-local dynamics, resulting in each layer acting as a PI feedback control system to minimize local prediction errors.
We find that PC-ALM is capable of propagating signals to seemingly arbitrary depth, especially in deep narrow networks where standard PC struggles to learn.
Ultimately, our motivation here is to understand how distributed physical systems, such as the brain, can compute credit signals using only local coupling and local dynamics.
PC-ALM may also inform deep learning in neuromorphic hardware, where dynamics are cheaper than on GPUs.
Paper: arxiv.org/abs/2605.31022
Code: github.com/SakanaAI/pc-alm
Makoto Shing retweeted
創業以来、オープンソースコミュニティから多くを学び、また研究成果の公開を通じてそこに貢献してきました。オープンなエコシステムが健全なAI産業と技術主権を支える重要な基盤の一つであると考えており、その発展を支持します。
このたび、Sakana AIは、オープンウェイトAIモデルに関する公開書簡 「Open Weights and American AI Leadership」に署名しました。
書簡は、オープンウェイトモデルがAIへのアクセス拡大、競争の促進、ユーザーによるコントロール、そして安全性の向上に寄与することを訴えるものです。
PDF：aka.ms/OpenLetter
全文：microsoft.com/en-us/corporat… 🐟
For my first post, I’m sharing a letter @NVIDIA signed on why open models matter.
AI will transform every industry, power every company, and be built by every country.
Open models strengthen safety and cybersecurity, accelerate innovation and diffusion, and enable sovereignty.
The world needs both frontier closed models and frontier open models.
images.nvidia.com/pdf/Open-W…
Makoto Shing retweeted
We're building AI that people and organizations can shape and make their own. AI should extend our will and judgment instead of neglecting it; enabling that is the technical challenge we are working to solve.
thinkingmachines.ai/blog/the…
Makoto Shing retweeted
Today, we are officially launching the Sakana AI RSI Lab in Tokyo to build open-ended, adaptive AI systems that collectively self-improve. I am incredibly proud of our team’s work over the past 2 years, shipping the breakthrough research that laid the foundations for this moment.
Building in Japan provides us with the ultimate design constraint. Just like Japan’s historical dominance in manufacturing was achieved by fundamentally redesigning the factory floor to do more with less, we are focused on compute-efficiency.
We are not building the most compute-hungry self-improvement engine. We are building the most sample-efficient one.
If you are entirely unsatisfied with the brute-force status quo and ready to build the self-improving future in Japan, come join us.
Building AI that Builds AI: Introducing the Sakana AI RSI Lab 🚀
sakana.ai/rsi-lab
Today, we are announcing the Sakana AI Recursive Self-Improvement (RSI) Lab: a dedicated research group in Tokyo tasked with redesigning the AI development process itself using AI.
While the industry increasingly speculates about the theoretical potential of self-improving AI, we’ve spent the last two years actively laying the foundations to make it a reality:
▪ LLM²: AI models automating research to invent better preference optimization algorithms.
▪ Darwin Gödel Machine: Agents autonomously rewriting their own codebase to double software-engineering performance.
▪ ShinkaEvolve: Hyper-sample-efficient program evolution that builds novel loss functions for MoE models.
▪ ALE-Agent: Reinforcement agents outperforming hundreds of human experts via self-learning.
▪ Digital Red Queen: Open-ended adversarial coevolution laying the groundwork for RSI in cybersecurity.
▪ The AI Scientist: Towards end-to-end automation of AI research, recently published in Nature.
Now, we are unifying these breakthroughs. The Sakana AI RSI Lab is officially tasked with building open-ended, adaptive architectures that collectively self-improve.
Human intelligence did not emerge from limitless resources; it was forged through the open-ended, compounding process of evolution operating under strict constraints. We are applying this exact principle to AI.
We believe recursive self-improvement is achievable on modest, sample-efficient compute. It shouldn’t be a winner-take-all asset locked inside hyperscale clusters, but a democratized public good.
We’re scaling our team to execute this mission. We are looking for frontier scientists and engineers who are entirely unsatisfied with the brute-force status quo. If you are ready to break away from standard benchmarking and build the self-improving future in Japan, come build with us.
Makoto Shing retweeted
Diffusion models are recurrent neural networks🧐
sander.ai/2023/07/20/perspec…
Introducing DiffusionBlocks: Block-wise Neural Network Training via Diffusion Interpretation
pub.sakana.ai/diffusionblock…
What if we didn’t have to hold an entire neural network in memory to train it?
Standard neural net training optimizes all parameters jointly. As a result, the memory required during training grows linearly with the depth of the network.
In our #ICLR2026 paper, we propose DiffusionBlocks, a principled framework to train networks one block at a time, drastically reducing memory requirements while matching end-to-end performance.
With DiffusionBlocks, we split the network into blocks and train them one at a time, so you only need memory for a single block.
How? We explicitly assign each block a role: to move the representation a little closer to the target than the block before it did. That role turns out to be precisely what a diffusion model does, step by step. Each block only needs to optimize its own objective and can be trained independently.
We validated this across five different architectures:
• ViT
• DiT
• Masked diffusion
• Autoregressive transformers
• Recurrent-depth transformers
In each case, performance is competitive with end-to-end training while using a fraction of the memory.
This perspective also extends naturally to recurrent-depth (Looped) transformers, which apply the same network iteratively and normally require expensive backpropagation through time (BPTT). Viewed through DiffusionBlocks, we can replace those multiple iterations with a single forward pass during training.
Read our paper and code, to learn more.
Paper: arxiv.org/abs/2506.14202
GitHub: github.com/SakanaAI/Diffusio…
🐟
Makoto Shing retweeted
ニューラルネットワークをブロックごとに学習する枠組みを開発
ブログ: pub.sakana.ai/diffusionblock…
ニューラルネットワークの学習は通常、ネットワーク全体を一度に扱う必要があり、深いモデルほど多くのメモリを必要とします。このメモリ消費は、近年のAIモデルの大規模化を支える上で大きな制約と なってきました。
Sakana AIは、この制約を緩和する学習フレームワーク「DiffusionBlocks」を提案します（#ICLR2026 採択）。ネットワークをブロックに分割し、それぞれを独立に学習できるようにすることで、学習時に必要なメモリを1ブロック分にまで削減することができます。
中心となる発想は、各ブロックに「ひとつ前のブロックよりも、表現を少しだけターゲットに近づける」という明示的な役割を与えることです。この役割は、近年大きな成功を収めている拡散モデルが時間方向に段階的に行っている処理に対応しており、この対応関係を踏まえることで、各ブロックを原理的な目的関数のもとで独立に学習することができるようになります。
画像分類、画像生成、テキスト生成にまたがる5つのアーキテクチャ（ViT、DiT、Masked Diffusion、AR Transformer、Recurrent-depth Transformer）で検証を行い、いずれにおいてもエンドツーエンド学習に匹敵する性能を確認しました。
この枠組みは、同じ層を繰り返し適用するRecurrent-depth Transformerにも自然に拡張でき、通常必要とされるbackpropagation through timeを経ずに、1回のフォワードパスで効率的に学習できることも示しています。
大規模AIモデルの学習を、より少ない計算資源でも進められるようにすること——これはSakana AIが継続して取り組んでいるテーマのひとつです。本研究が、その一歩となることを期待しています。
論文: arxiv.org/abs/2506.14202
本研究は、東京大学の小山雅典氏との共同で行われました。
Introducing DiffusionBlocks: Block-wise Neural Network Training via Diffusion Interpretation
pub.sakana.ai/diffusionblock…
What if we didn’t have to hold an entire neural network in memory to train it?
Standard neural net training optimizes all parameters jointly. As a result, the memory required during training grows linearly with the depth of the network.
In our #ICLR2026 paper, we propose DiffusionBlocks, a principled framework to train networks one block at a time, drastically reducing memory requirements while matching end-to-end performance.
With DiffusionBlocks, we split the network into blocks and train them one at a time, so you only need memory for a single block.
How? We explicitly assign each block a role: to move the representation a little closer to the target than the block before it did. That role turns out to be precisely what a diffusion model does, step by step. Each block only needs to optimize its own objective and can be trained independently.
We validated this across five different architectures:
• ViT
• DiT
• Masked diffusion
• Autoregressive transformers
• Recurrent-depth transformers
In each case, performance is competitive with end-to-end training while using a fraction of the memory.
This perspective also extends naturally to recurrent-depth (Looped) transformers, which apply the same network iteratively and normally require expensive backpropagation through time (BPTT). Viewed through DiffusionBlocks, we can replace those multiple iterations with a single forward pass during training.
Read our paper and code, to learn more.
Paper: arxiv.org/abs/2506.14202
GitHub: github.com/SakanaAI/Diffusio…
🐟
Makoto Shing retweeted
"DiffusionBlocks: Block-wise Neural Network Training via Diffusion Interpretation"
End to end backprop stores activations through every layer, which is why deep Transformers is expensive to train.
This paper reinterprets residual blocks as diffusion denoising steps, so each block trains independently on its own noise range.
Only one block needs gradients at a time, giving roughly Bx memory savings while matching end to end training across ViTs, DiTs, diffusion LMs, AR Transformers, and recurrent depth models.
Makoto Shing retweeted
Introducing DiffusionBlocks: Block-wise Neural Network Training via Diffusion Interpretation
pub.sakana.ai/diffusionblock…
What if we didn’t have to hold an entire neural network in memory to train it?
Standard neural net training optimizes all parameters jointly. As a result, the memory required during training grows linearly with the depth of the network.
In our #ICLR2026 paper, we propose DiffusionBlocks, a principled framework to train networks one block at a time, drastically reducing memory requirements while matching end-to-end performance.
With DiffusionBlocks, we split the network into blocks and train them one at a time, so you only need memory for a single block.
How? We explicitly assign each block a role: to move the representation a little closer to the target than the block before it did. That role turns out to be precisely what a diffusion model does, step by step. Each block only needs to optimize its own objective and can be trained independently.
We validated this across five different architectures:
• ViT
• DiT
• Masked diffusion
• Autoregressive transformers
• Recurrent-depth transformers
In each case, performance is competitive with end-to-end training while using a fraction of the memory.
This perspective also extends naturally to recurrent-depth (Looped) transformers, which apply the same network iteratively and normally require expensive backpropagation through time (BPTT). Viewed through DiffusionBlocks, we can replace those multiple iterations with a single forward pass during training.
Read our paper and code, to learn more.
Paper: arxiv.org/abs/2506.14202
GitHub: github.com/SakanaAI/Diffusio…
🐟
Makoto Shing retweeted
For over a decade, we’ve accepted that end-to-end backprop is the only way to train deep networks. But holding the entire network in memory all at once is why AI training is hitting a resource wall.
We found a new way to break the network into blocks and train them independently. The trick? Treating the network’s forward pass like a diffusion model denoising a signal.
This reinterpretation slashes the memory needed to train deep models. In our #ICLR2026 paper (arxiv.org/abs/2506.14202), we matched end-to-end performance across ViTs, DiTs, and LLMs. We did this while training just one isolated block at a time.
Introducing DiffusionBlocks: Block-wise Neural Network Training via Diffusion Interpretation
pub.sakana.ai/diffusionblock…
What if we didn’t have to hold an entire neural network in memory to train it?
Standard neural net training optimizes all parameters jointly. As a result, the memory required during training grows linearly with the depth of the network.
In our #ICLR2026 paper, we propose DiffusionBlocks, a principled framework to train networks one block at a time, drastically reducing memory requirements while matching end-to-end performance.
With DiffusionBlocks, we split the network into blocks and train them one at a time, so you only need memory for a single block.
How? We explicitly assign each block a role: to move the representation a little closer to the target than the block before it did. That role turns out to be precisely what a diffusion model does, step by step. Each block only needs to optimize its own objective and can be trained independently.
We validated this across five different architectures:
• ViT
• DiT
• Masked diffusion
• Autoregressive transformers
• Recurrent-depth transformers
In each case, performance is competitive with end-to-end training while using a fraction of the memory.
This perspective also extends naturally to recurrent-depth (Looped) transformers, which apply the same network iteratively and normally require expensive backpropagation through time (BPTT). Viewed through DiffusionBlocks, we can replace those multiple iterations with a single forward pass during training.
Read our paper and code, to learn more.
Paper: arxiv.org/abs/2506.14202
GitHub: github.com/SakanaAI/Diffusio…
🐟
I'll be presenting DiffusionBlocks at #ICLR2026!
⏰ Thu Apr 23, 3:15–5:45 PM
📍 Pavilion 3 P3-#506
📄 arxiv.org/abs/2506.14202
Please stop by if you're interested in efficient training or principled alternatives to backprop!
My latest work got accepted to #ICLR2026! I am really grateful for my co-authors!!
We propose an alternative approach to end-to-end backpropagation!
1/
Makoto Shing retweeted
Highly recommend this for a read!
My latest work got accepted to #ICLR2026! I am really grateful for my co-authors!!
We propose an alternative approach to end-to-end backpropagation!
1/
Makoto Shing retweeted
Thrilled to share that three of our papers have been accepted to #ICLR2026. Huge thanks to my co-authors!
Specifically, we propose a block-wise training framework that achieves performance comparable to end-to-end backpropagation while requiring less memory!
By interpreting transformer networks as diffusion processes, each block can be independently trained.
For example, if 6 layers are split into 3 blocks as shown in the figure, training requires memory for only 2 layers (6/3) at a time!
We demonstrated the effectiveness across various tasks:
* ViT on image classification
* DiT on image generation
* Autoregressive LMs on text generation
* Discrete Language Diffusion on text generation
As a bonus, our DiffusionBlocks naturally extends to recurrent-depth models (also known as recursive transformers) and eliminates the cost of BPTT during training!
2/
Paper (camera-ready version coming soon):
arxiv.org/abs/2506.14202