Pinned Tweet
New paper: "Self-Distillation of Hidden Layers for Self-Supervised Representation Learning"
We introduce Bootleg — a simple twist on I-JEPA/MAE that dramatically improves self-supervised representations.
The idea: MAE predicts pixels (stable but low-level). I-JEPA predicts final-layer embeddings (high-level but unstable). Bootleg bridges the two by predicting representations from multiple hidden layers of the teacher network — early, middle, and late — simultaneously.
Why it works: early layers provide stimulus-driven grounding that prevents collapse; deep layers provide semantic targets; and the information bottleneck of compressing all abstraction levels through masked patches forces the encoder to build richer representations.
The method is quite simple on top of I-JEPA: extract targets from evenly-spaced blocks, z-score and concatenate, widen the predictor's final layer. That's it.
Frozen probe results (no fine-tuning):
ImageNet-1K: 76.7% with ViT-B (+10pp over both I-JEPA and MAE)
iNaturalist-21: 58.3% with ViT-B (+17pp over I-JEPA, +15pp over MAE)
ADE20K segmentation: 30.9% mIoU with ViT-B (+11pp over I-JEPA, +6pp over MAE)
Cityscapes segmentation: 35.9% mIoU with ViT-B (+11pp over I-JEPA, +5pp over MAE)
Gains hold across ViT-S, ViT-B, and ViT-L.
Single-view, batch-size independent — no augmentation stack, no multi-crop, no contrastive loss, no large compute requirements.
Our study is just on images, but this change can be readily deployed to MAE and JEPA models across all domains.
arxiv.org/abs/2603.15553
Scott Lowe retweeted
We now run the largest frontier lab composed entirely of autonomous AI researchers.
Introducing Primus Society: a society of agents composed of thousands of researchers working under structured institutions designed to solve the world’s toughest problems. We believe this is how AI research should run at scale, safely and productively: an entire society of agents with institutions and purpose.
One discovery we can already share is that the society has discovered a novel result which improves model training by 30%.
Primus Society was inspired by the structures that have organized science for centuries and stress-tested against what’s known about how populations of AI agents fail.
Everything is observable. You can open the virtual city in a browser and read what any researcher is working on.
Learn more about Primus Society here:
lab.cloud/society
The biggest opening in the AI race is running the largest well-governed organization of AI researchers in the world. We’re building the institutions that let a million AI scientists safely tackle the world's toughest problems in AI and beyond. And we’re doing it right here in Canada.
Scott Lowe retweeted
🔥 PhD in the DINO team 🔥
We have a PhD program opening in our team. Don't hesitate to reach out to me or Max if interested!
Scott Lowe retweeted
We just got the top community score on ARC-AGI-3's 25 public games: 78.4% and 160/183 levels cleared, where the best frontier LLM session scores 7.8%. No training or demonstrations allowed.
Paper: arxiv.org/abs/2607.01531
Blog: david-courtis.github.io/opin…
w/ @WenhaoLi29 @ScottSanner
#AI #worldmodel #agentic #AGI @arcprize @fchollet
I'm at ICLR this week. Let me know if you want to meet up!
This afternoon @LinYorker will be presenting our work on KL-Shampoo, a new optimizer which expands on Shampoo and SOAP, and has performance competitive with Muon. arxiv.org/abs/2509.03378
Tomorrow, I will be presenting my recent self-supervised learning paper "Bootleg" (arxiv.org/abs/2603.15553) at the Multimodal Intelligence workshop, and follow-up showing the method yields drift-resilient representations at the Catch, Adapt, and Operate workshop.
• Apr 25, 15:15 – 17:45, Pavilion 3, Poster 403. Poster: "Understanding and improving Shampoo and SOAP via Kullback-Leibler Minimization". arxiv.org/abs/2509.03378
• Apr 26, 10:15 – 11:00, Room 207. Workshop: Catch, Adapt, and Operate: Monitoring ML Models Under Drift. Poster: "Hidden-Layer Self-Distillation Yields Drift-Resilient Visual Representations". openreview.net/forum?id=pa3z…
• Apr 26, 14:30 – 16:10, Room 204C. Workshop: Multimodal Intelligence: Next Token Prediction and Beyond. Poster: "Bridging Generative and Predictive Paradigms via Hidden-Self-Distillation". openreview.net/forum?id=YkMr…
Interesting timing — Meta released V-JEPA 2.1 on literally the same day as our Bootleg paper, independently arriving at the same core idea: self-distillation of multiple hidden layers as prediction targets, evenly spaced across the encoder.
The details are strikingly similar: ~4 target blocks, per-level normalization, concatenated along the channel dimension, EMA teacher.
Their ablation actually shows that multi-level prediction is what makes their new context loss viable — without it, the context loss destroys classification accuracy (-10pp). Hidden self-distillation is doing the heavy lifting.
Great to see convergent evidence from Meta's JEPA team confirming that this is a fundamental improvement to the framework. Our paper provides detailed ablations and analysis of why it works; V-JEPA 2.1 shows it scales to ViT-G and video.
Bootleg paper: arxiv.org/abs/2603.15553
V-JEPA 2.1 paper: arxiv.org/abs/2603.14482
New paper: "Self-Distillation of Hidden Layers for Self-Supervised Representation Learning"
We introduce Bootleg — a simple twist on I-JEPA/MAE that dramatically improves self-supervised representations.
The idea: MAE predicts pixels (stable but low-level). I-JEPA predicts final-layer embeddings (high-level but unstable). Bootleg bridges the two by predicting representations from multiple hidden layers of the teacher network — early, middle, and late — simultaneously.
Why it works: early layers provide stimulus-driven grounding that prevents collapse; deep layers provide semantic targets; and the information bottleneck of compressing all abstraction levels through masked patches forces the encoder to build richer representations.
The method is quite simple on top of I-JEPA: extract targets from evenly-spaced blocks, z-score and concatenate, widen the predictor's final layer. That's it.
Frozen probe results (no fine-tuning):
ImageNet-1K: 76.7% with ViT-B (+10pp over both I-JEPA and MAE)
iNaturalist-21: 58.3% with ViT-B (+17pp over I-JEPA, +15pp over MAE)
ADE20K segmentation: 30.9% mIoU with ViT-B (+11pp over I-JEPA, +6pp over MAE)
Cityscapes segmentation: 35.9% mIoU with ViT-B (+11pp over I-JEPA, +5pp over MAE)
Gains hold across ViT-S, ViT-B, and ViT-L.
Single-view, batch-size independent — no augmentation stack, no multi-crop, no contrastive loss, no large compute requirements.
Our study is just on images, but this change can be readily deployed to MAE and JEPA models across all domains.
arxiv.org/abs/2603.15553
cc V-JEPA 2.1 team: @murloren @mattmucklm @_amirbar @mido_assran @koustuvsinha @ylecun @ballas_nicolas @AdrienBardes
6
540
A few surprising findings from the paper:
The best single target layer for I-JEPA isn't the final layer — it's block 2 for ViT-S and block 6 for ViT-B. But you don't need to pick: targeting multiple layers simultaneously beats any single layer.
MAE with hidden self-distillation completely collapses at deeper target layers. But just switching from random masking to block-structured masking is enough to stabilize it. Masking strategy matters far more than you'd expect.
Paper: arxiv.org/abs/2603.15553
Code and model checkpoints will be released soon!
Work in collaboration with Anthony Fuller, Sageev Oore (@osageev), Evan Shelhamer (@ev4n3sce), and Graham Taylor at @VectorInst , @Carleton_U, @DalhousieU, @UBC, @UofG.
Scott Lowe retweeted
Within an information-geometric framework, we reconnect Shampoo/SOAP with both classical quasi-Newton ideas and Gaussian whitening, and develop practical methods that naturally handle tensor-valued weights in language model pre-training. arxiv.org/abs/2509.03378 opt-ml workshop
Scott Lowe retweeted
We were able to reproduce the strong findings of the HRM paper on ARC-AGI-1.
Further, we ran a series of ablation experiments to get to the bottom of what's behind it.
Key findings:
1. The HRM model architecture itself (the centerpiece of the paper) is not an important factor.
2. The outer refinement loop (barely mentioned in the paper) is the main driver of performance.
3. Cross-task transfer learning is not very helpful. What matters is training on the tasks you will test on.
4. You can use much fewer data augmentations, especially at inference time.
Finding 2 & 3 mean that this approach is a case of *zero-pretraining test-time training*, similar to the recently published "ARC-AGI without pretraining" paper by Liao et al.
Scott Lowe retweeted
i am so superduper excited to show a little teaser of this system we're developing for an improvising #music system with #AI! i can't explain how *actually* fun it is to play with this.
youtu.be/onPetq4gJ18
This 15 sec video shows a glimpse of me playing with the system. Notice that at the beginning of the video... the piano is playing itself! And then it seamlessly transitions back into a kinda cuban montuno groove, similar (but not at all the same) as the #prompt that I originally gave it! (i'll put a link to the prompt in the replies)
here's a very short blogpost too. lots more soon!
osageev.github.io/introducin…
The turn of the year is a good time for reflection. Here are my thoughts on where ML research is headed.
1
1
97
For GenAI, improvements are going to come from better data curation and value fns to drive the model toward high-quality outputs. Standard model training results in outputs representative of the training distribution, but users don't want average—we want the best quality.
1
59
There will also be continued interest in methods that allow more controllable manipulation of generated images (e.g. UIP2P arxiv.org/abs/2412.15216, Imagic arxiv.org/abs/2210.09276).
But maybe the pending copyright lawsuits will have major impacts on GenAI.
53















