Gallil Maimon

New preprint: 🔥 Latent Information Feedback Transformers (LIFT) 🔥 Information in LM generation propagates downward (high to low layers) only through the decoded token, creating a bottleneck. Removing it via state propagation makes the model recurrent, which is not scalable for training. Can we teach LMs to propagate state while keeping pretraining fully parallel? YES! LIFT is a Transformer-based architecture + teacher-supervised training approach that enables Transformer LMs to exploit deep-to-shallow feedback, while keeping training parallel. LIFT models consistently outperform standard Transformers and baselines on language modeling, downstream reasoning tasks, and procedural tasks under a token-matched budget, while being on par with or ahead of compute-matched Transformers. LIFT also outperforms Transformers trained on 8x more data on a state-tracking task, even when trained with the states of a Transformer that fails it! Work by the amazing @TiroshDor and @AmosaurusRex Paper: arxiv.org/abs/2609.38149 Detailed post + visualizations + code + models coming soon!