PhD student at the University of Maryland
Joined November 2023
- Tweets230
- Following170
- Followers722
- Likes665
Pinned Tweet
Library of Layers: models which adapt layer order per input, routing over whole transformer blocks.
This opens up a new avenue to control: the execution path the model uses. We use this to control the algorithm used to solve mazes and sudoku by only changing the layer order.
🧵 1/n
Sean McLeish retweeted
After reading the MEMENTO paper (@vkontonis @DimitrisPapail) I wondered if compactions needs explicit text at all. Turns out latent compaction is possible!
AbridgeKV trains the model to compress its own output. Keeping 1/4 of the generated cache, it matches full-KV accuracy on competition math problem suite and finishes 3.2× faster on one H100. 🧵
Sean McLeish retweeted
SphereEncoder2 is out! Our 2nd-gen ImageNet model achieves SOTA image quality (FDr6 score) among all known pixel-space image generators, including diffusions.
The Sphere Encoder is a faster alternative to diffusion that produces images in a single forward pass. It's basically an autoencoder on steroids that maps 100k-dimensional random latent vectors directly to natural images. Here's some single-pass images from a model trained only on ImageNet-1K.
Sean McLeish retweeted
Came across this interesting work by Chen et al. on loss spikes in LM pretraining. They trace this late emerging spike (⌛️💣 :) to rounding errors in the output layer. Interesting part is they show why three fixes (z-loss, μ-centering and momentum SGD) help in the same way.
1/12
ALT Chen et al., The Role of Quantization in Loss Spikes, Figure 3.
Sean McLeish retweeted
📄 Library of Layers (LoL): Adaptively Ordering Transformer Layers at Runtime
by Abdellah Rahmani (@arahmani_AR), Sean McLeish (@SeanMcleish) et al.
A router selects the next block from a shared layer library at each depth, so execution order changes per input instead of staying fixed.
Key points:
• Safety probe on sequence-level routing features reaches 0.95 AUROC
• Maze strategy control hits 99.79-100% correct forced-strategy traces
• Sudoku forced routing yields 99.5-100% requested-strategy accuracy
Shows that layer order can be a learned, controllable part of transformer computation rather than a fixed architectural choice.
ALT Detailed method diagram showing layer architecture, strided attention, and router mechanism
Thanks to my amazing collaborators @arahmani_AR, @iamleonli, @LotfiSanae, @pafrossard, and @micahgoldblum
📝 arxiv.org/abs/2611.03526
8/8
Also check out Abdellah’s and Micah’s tweets!
teapaw.org/arahmani_AR/status/211…
teapaw.org/micahgoldblum/status/2…
Sean McLeish retweeted
Library of Layers (LoL): Adaptively Ordering Transformer Layers at Runtime
Library of Layers lets models dynamically reorder a shared pool of transformer blocks during inference, achieving competitive results while enabling controllable problem-solving.
arxivb.org/abs/2611.03526
Sean McLeish retweeted
Programmers build wildly different algorithms by chaining the same few primitives in different orders. LLMs can’t: every input runs through the same layers in the same order. So we built a model that re-orders its own forward pass at runtime. 🧵 1/11
Sean McLeish retweeted
What if an LLM could adapt its layer order to each input?
What if it learned a different layer order for each classic algorithm that solves Sudoku or mazes, and you could fix that order at inference to choose which algorithm it runs?
Meet LoL in under 4 minutes 🎥 (1/3)
Sean McLeish retweeted
What if we thought of layers as flexible building blocks that can be rearranged and parsed in an optimal order conditioned on the input itself? Our new work explores this question👇
Big congrats to @arahmani_AR, @SeanMcleish, @micahgoldblum on this great work!
Sean McLeish retweeted
As someone who's bitter lesson pilled, I've always been fascinated by the idea of learning model arch through gradient descent rather than relying on inductive biases.
MoE brought sparsity and dynamic routing to FFNs within individual transformer layers. Can we do the same across layers themselves?
Congrats to @arahmani_AR and @SeanMcleish on this great work!
Sean McLeish retweeted
We wanted to know whether fixing Softmax Collapse also prevents the delayed, unrecoverable “time-bomb” loss spikes we study in LLM pretraining. TLDR: in our setting the fix delays the bomb, but doesn’t defuse it. 🧵1/n
Loss spikes appear to be coming from the growing lm head norm. We trace it further, finding a growing lm head norm causes increased quantisation error and a snowball effect that culminates in an irreversible loss spike.
The fix: regularise to stay where the finite precision is high