🪗 Your Transformer Is Secretly Elastic 🧵👇 Transformer families come in a few fixed sizes, so you commit to one for your hardware and latency constraints, even as query loads vary. What if a single pre-trained model could serve every budget? We learn one importance ranking over every MLP neuron and attention head, once, with frozen weights. At inference time, the model can then be pruned to any computational budget by dropping its least important units, without retraining. Across 27 vision and text encoders spanning 22M to 8B parameters, performance degrades smoothly up to 60% sparsity. For example, pruning DINOv3 ViT-H+/16 to 50% sparsity removes 427M parameters, for only a 0.8% drop in ImageNet-1k linear probing. Super fun project in collaboration with Dheeraj Varghese (@dhevarghese), supervised by Gertjan J. Burghouts (@gjburghouts) and Cees Snoek (@cgmsnoek) 🌐 Project page: secretly-elastic.ashita.nl