Abhinav Moudgil (@amoudgl) profile picture

Research Scientist at Meta Superintelligence Labs

London, UK
Joined July 2014
Photo by Abhinav Moudgil (@amoudgl)Photo by Abhinav Moudgil (@amoudgl)Photo by Abhinav Moudgil (@amoudgl)Photo by Abhinav Moudgil (@amoudgl)Photo by Abhinav Moudgil (@amoudgl)Photo by Abhinav Moudgil (@amoudgl)Photo by Abhinav Moudgil (@amoudgl)Photo by Abhinav Moudgil (@amoudgl)Photo by Abhinav Moudgil (@amoudgl)Photo by Abhinav Moudgil (@amoudgl)
Abhinav Moudgil retweeted
Today we're also opening the weights for Muse Glimmer, a great 30B parameter dense model that can run locally. Soon we'll also release the weights for Muse Spark 1.2, our latest foundation model. Meta is a strong supporter of open source and I'm proud of these releases. Congrats to @alexandr_wang and the MSL team for all your great work on these models.
1,846
2,507
1,247
30,207
5,301,716
Abhinav Moudgil retweeted
PyLO is accepted to MLSys 2026! 🎉🚀 A PyTorch-native library bringing SOTA learned optimizers to the codebases most of us actually use — with fast CUDA kernels and real speedups on large-scale training. Drop-in ready, no more JAX-only barriers. Library: github.com/Belilovsky-Lab/py…
Have you ever trained a neural network using a learned optimizer instead of AdamW? Doubt it: you're probably coding in Pytorch! Excited to introduce PyLO: Towards Accessible Learned Optimizers in Pytorch! . Accepted at @icmlconf ICML 2025 CODEML workshop 🧵1/N
1
8
11
1,778
Abhinav Moudgil retweeted
NEW paper from Apple. Interesting idea: "Attention to Mamba". The paper introduces a two-stage recipe for cross-architecture distillation from Transformers into Mamba. Naive distillation collapses teacher performance. Their trick: first distill the transformer into a linearized-attention student using a kernel adaptation, then transfer that student into a pure Mamba with no attention blocks. On a 1B model trained on 10B tokens, the Mamba student hits 14.11 perplexity against a 13.86 Pythia-1B teacher, nearly matching quality at linear-time inference cost. If you can reliably convert trained transformers into state-space models without retraining from scratch, the entire open-weights ecosystem becomes cheaper to serve at long context. This is the kind of quiet infrastructure work that decides which architectures actually get deployed in agent stacks. Paper: arxiv.org/abs/2604.14191 Learn to build effective AI agents in our academy: academy.dair.ai/
14
136
14
958
78,299
Abhinav Moudgil retweeted
Masked Diffusion LMs (MDLMs) are the most exciting paradigm shift in AR generation because they can decode in parallel, infill, and self-correct. But they are bottlenecked by the transformer's quadratic attention, making throughput fall apart for long contexts. We offer a simple solution. Introducing DiffuMamba: first diffusion LM with a bidirectional Mamba backbone. Better quality. Up to 8.2x faster. 🧵1/N
4
27
3
162
15,661
Abhinav Moudgil retweeted
Are frontier LLMs trained across datacenters? One thing is certain: if the pre-training optimizer’s critical batch size is too small, they are NOT! Excited to announce MuLoCo, a pre-training optimizer that can efficiently pre-train across datacenters while having large enough batch sizes to warrant doing so. 🧵1/N
3
33
2
96
20,736
Abhinav Moudgil retweeted
We’re excited to share our new work : “Stabilizing Native Low-Rank LLM Pretraining.” (arxiv.org/abs/2602.12429) 🚀 Can we train LLMs from scratch using only low-rank factorized weights and still match dense performance? Short answer: yes (with care).
4
43
6
207
34,780
Abhinav Moudgil retweeted
Can AI agents beat humans at frontier ML research? We are introducing AIRS-bench, asking agents to beat human SOTA on 20 research tasks from recent ML papers. Check out the results: arxiv.org/abs/2602.06855
5
19
2
111
9,289
Abhinav Moudgil retweeted
🚨TL;DR: Benchmarking for AI Scientists just got better!🚨 Everyone is excited about AI Scientists, but we don't have a large scale benchmark that evaluates automated (or augmented) AI research systems on the home turf of the machine learning community: Machine Learning benchmarks. Meet AIRS-Bench, our attempt at filling this gap. We hope AIRS-Bench will help the community to improve the signal-to-noise ratio in the era of research agents and is an important step towards turning ML benchmarks into standardised tasks for AI research agents. This has implications beyond AI scientists and will also help address the replication crisis in ML. The team has invested countless hours (human and GPU) selecting/constructing the tasks, running baseline agents, analysing the outcomes, and hardening the benchmark. We are excited for the community to both expand on our initial task set and benchmark new agentic systems!
2
9
68
12,288
Abhinav Moudgil retweeted
Introducing - AIRS Bench, a benchmark for “AI Researcher Agent”. Agents attempt 20 open ML problems starting from zero code (full research loop). And yes, they beat SOTA in few cases (read more below!) arxiv.org/abs/2602.06855
4
12
2
72
16,631
Abhinav Moudgil retweeted
Recently, there has been a lot of talk of LLM agents automating ML research itself. If Llama 5 can create Llama 6, then surely the singularity is just around the corner. How can we get a pulse check on whether current LLMs are capable of driving this kind of total self-improvement? Well, we know humans are pretty good at improving LLMs. In the NanoGPT speedrun challenge, created by @kellerjordan0, human researchers iteratively improved @karpathy's GPT-2 replication, slashing the training time (to the same target validation loss) from 45 minutes to under 3 minutes in just under a year (!). Surely, a necessary (but not sufficient) ability for an LLM that can automatically improve frontier techniques is the ability to *reproduce* known innovations on GPT-2, a tiny language model from over 5 years ago. 🤔 So we took several of the top models and combined them with various search scaffolds to create *LLM speedrunner agents*. We then asked these agents to reproduce each of the NanoGPT speedrun records, starting from the previous record, while providing them access to different forms of hints that revealed the exact changes needed to reach the next record. The results were surprising—not because we thought these agents would ace the benchmark, but because even the best agent failed to recover even half of the speed-up of human innovators on average in the easiest hint mode, where we show the agent the full pseudocode of the changes to the next record. We believe The Automated LLM Speedrunning Benchmark provides a simple eval for measuring the lower bound of LLM agents’ ability to reproduce scientific findings close to the frontier of ML. Beyond scientific reproducibility, this benchmark can also be run without hints, transforming into an automated *scientific innovation* benchmark. When run in "innovation mode," this benchmark effectively extends the NanoGPT speedrun to AI participants! While initial results here indicate that current agents seriously struggle to match human innovators beyond just a couple of records, benchmarks have a tendency to fall. This one is particularly exciting to watch, as new state-of-the-art here by definition implies a form of *superhuman innovation*.
40
193
36
1,189
572,527
Abhinav Moudgil retweeted
Is the mystery behind the performance of Mamba🐍  keeping you awake at night? We got you covered! Our ICML2025 paper demystifies input selectivity in Mamba from the lens of approximation power, long-term memory, and associative recall capacity. arxiv.org/abs/2506.11891
1
17
4
51
6,575
Abhinav Moudgil retweeted
Our work on Asynchronous RLHF was accepted to #ICLR2025 ! (I was so excited to announce it, I forgot to say I was excited) Used by @allen_ai for OLMo-2 32B 🔥 New results show ~70% speedups for LLM + RL math and reasoning 🧠 🧵below or hear my DLCT talk online on March 28!
Is your RLHF training too slow? 🦥 Does inefficient LLM generation get you down? We propose a new paradigm: Asynchronous RLHF! It's faster, more efficient, and achieves the same perf as SOTA methods only improving with scale! arxiv.org/abs/2410.18252 and we release code too
4
19
87
10,276
Abhinav Moudgil retweeted
1/N🧵 Have you been using cosine decay to continually pre-train your foundation models? 💭 Excited to share our new paper, Beyond Cosine Decay, where we explore infinite LR schedulers ♾️. Check it out! arxiv.org/abs/2503.02844. #MachineLearning #AI
2
5
14
829
Abhinav Moudgil retweeted
NiNo meets Nano! Added neural graphs for the nanoGPT benchmark from @kellerjordan0. My fork github.com/bknyaz/modded-nan…. Our pretrained NiNo doesn't yet generalize well in this case, which makes it an interesting benchmark. NiNo will be presented at #ICLR2025!
Optimization can be sped up by 50% using our NiNo model! It takes a history of parameter values and predicts future parameters leveraging "neural graphs". Accelerating Training with Neuron Interaction and Nowcasting Networks: arxiv.org/abs/2409.04434 code: github.com/SamsungSAILMontre…
1
2
1
13
2,122
How do MoE transformers, like DeepSeek, behave under distribution shifts? Do their routers collapse? Can they still match full re-training performance? Excited to present “Continual Pre-training of MoEs: How robust is your router?”!🧵arxiv.org/abs/2503.05029 1/N
1
21
1
39
3,778
Parameterized Transforms 🚀 Here is a new tool that provides a modular and extendable implementation of torchvision-based image augmentations that provides access to their parameterization. [1/5]
1
9
2
15
2,144
Abhinav Moudgil retweeted
Reading "Distilling Knowledge in a Neural Network" left me fascinated and wondering: "If I want a small, capable model, should I distill from a more powerful model, or train from scratch?" Our distillation scaling law shows, well, it's complicated... 🧵 arxiv.org/abs/2502.08606
10
146
14
1,036
123,076
Abhinav Moudgil retweeted
Enjoy attention? Want to make it ~18% faster? Try out Sigmoid Attention. We replace the traditional softmax in attention with a sigmoid and a constant (not learned) scalar bias based on the sequence length. Paper: arxiv.org/abs/2409.04431 Code: github.com/apple/ml-sigmoid-… This was an amazing collaboration with some great researchers at Apple: Federico Danieli, @EeshanDhekane, @FlorisWeers, @danbusbridge, @PierreAblin, Tatiana Likhomanenko, Jagrit Digani, Zijin Gu, @AmitisShidani1 and Russ Webb. More details in 🧵below.
15
163
16
824
128,538
Abhinav Moudgil retweeted
Optimization can be sped up by 50% using our NiNo model! It takes a history of parameter values and predicts future parameters leveraging "neural graphs". Accelerating Training with Neuron Interaction and Nowcasting Networks: arxiv.org/abs/2409.04434 code: github.com/SamsungSAILMontre…
🔍How can we design neural networks that take neural network parameters as input? 🧪Our #ICLR2024 oral on "Graph Neural Networks for Learning Equivariant Representations of Neural Networks" answers this question! 📜: arxiv.org/abs/2403.12143 💻: github.com/mkofinas/neural-g… 🧵 [1/9]
6
16
4
61
11,808