Research scientist at Meta (MSL FAIR). PhD in Computer Science. Working on AI theory+algorithms in free time.
Philadelphia, PA
Joined March 2018
- Tweets398
- Following159
- Followers247
- Likes2.9K
This suggests two things clearly to me (with caveat at end):
1. Most low likelihood tokens from pre and post training don’t really help benchmark scores (~useful tasks proxy). ➞ you can train on less data and get a better result cheaper
2. The knowledge distillation that happens during training works, such that training on the result of the distillation creates a better model for benchmarks. ➞ the underlying mental model approximated is good
Caveat: assuming the benchmarks contain enough diversity to be reliable and representative.
Apple Research just published something really interesting about post-training of coding models.
You don't need a better teacher. You don't need a verifier. You don't need RL.
A model can just… train on its own outputs. And get dramatically better.
Simple Self-Distillation (SSD): sample solutions from your model, don't filter them for correctness at all, fine-tune on the raw outputs. That's it.
Qwen3-30B-Instruct: 42.4% → 55.3% pass@1 on LiveCodeBench. +30% relative. On hard problems specifically, pass@5 goes from 31.1% → 54.1%.
Works across Qwen and Llama, at 4B, 8B, and 30B. One sample per prompt is enough. No execution environment. No reward model. No labels.
SSD sidesteps this by reshaping distributions in a context-dependent way — suppressing distractors at locks while keeping diversity alive at forks. The capability was already in the model. Fixed decoding just couldn't access it.
The implication: a lot of coding models are underperforming their own weights. Post-training on self-generated data isn't just a cheap trick — it's recovering latent capacity that greedy decoding leaves on the table.
paper: arxiv.org/abs/2604.01193
code: github.com/apple/ml-ssd
Thomas Lux retweeted
Step 3: Do a rep of gratitude every day.
No matter who you are, this is proven to help your mind and your body. And when it feels like there is absolutely nothing you can do, you can do this one thing. Write down one thing you’re grateful for, every day.
Fun weekend project, made an easy-access dedicated overlay + launcher for Grok on macOS that can be summoned or dismissed with Option+Space. I tried building it both with Grok-3 Think and o3-mini-high, and Grok-3 was the first one to give me a usable application. Now I can easily use Grok without moving focus away from whatever I'm actively doing on my laptop.
I'm pretty happy with the result! I made it a python package so it's easy for anyone to install. It's named `macos-grok-overlay` and after using pip to download it, you can install the launcher by running `macos-grok-overlay --install-startup`. The core logic is really simple, it's just a dedicated window with a webview locked to the main website. Most of the code that was needed was to make interacting with the custom window clean and getting Accessibility access to listen for the Option+Space keyboard command.
The quick mini dialogue is my primary way of using ChatGPT on my computer (via the desktop app), and I know that if I really want to give any other chatbots a chance (including Grok), I need something like this. Since it's just a webview, this should stay up-to-date with the newest release without needing explicit updates to the underlying launcher.🤞
Some caveats - in order to listen for the "Option+Space" keyboard sequence, the application needs Accessibility permissions to be granted (to python3); for some reason login via X does not work through a webview so I had to login through google; in case people enjoy using this I set the webview to include `referrer=macos-grok-overlay` so the xAI team can easily gauge interest. Happy building!
😁 @ibab @jondotvio #Grok3
Now login via X works with the newest release! Also have the ability to customize the keyboard shortcut and a DMG easy-installer for a dedicated application so you don't have to use python. github.com/tchlux/macos-grok…
github.com/tchlux/macos-grok…
☝️ here's the repo for anyone interested! 🤓
And below are the conversations that I used to build it all (normally I'd start separate conversations, but wanted to keep it in fewer threads for posterity here).
first (got too long) grok.com/share/bGVnYWN5_cd69…
second (and final) grok.com/share/bGVnYWN5_9e45…
LLM agents that can modify their own context while “working” is necessary, but instead of classic RAG and copy-paste, research should pursue how to do it in token vector space directly! A better model would be able to create new tokens on the fly to represent “new thoughts”.
Few things pretty obvious to a few AI researchers but that most don't want to believe:
1. 90% of the most impactful AI research is already on arxiv, x, or company blog posts
2. q* aka strawberry = STaR (self-taught reasoners) with dynamic self-discover + something like DSPy for RAG optimization
3. State of the art search = GoT(graph of thoughts) + MCTS (monte-carlo tree search) + DSPy / CLIN inspired tooling for "self-play" like optimization at build-time and runtime + graph-based knowledge bases
4. Continuous learning agents is clever graph retrieval inspired by GraphRAG, CLIN, DSPy and dynamic reasoning modules
5. Multiple OOM improvement to be realized when those design patterns are baked into models, fine-tuned on and prompt-optimized at build-time + runtime with human in the loop.
Although new foundation models around these patterns may not be needed to achieve significant improvement on existing open-source models...
Read for yourself and thank @aidan_mclau and @apples_jimmy
GOT: arxiv.org/abs/2308.09687
MCTS: arxiv.org/abs/2406.07394
STaR: arxiv.org/pdf/2203.14465
DSPy: github.com/stanfordnlp/dspy
CLIN: arxiv.org/abs/2310.10134
Self Discover: arxiv.org/pdf/2402.03620
GraphRAG: github.com/microsoft/graphra…
Path to AGI: situational-awareness.ai/fro…
Q* / Strawberry: reuters.com/technology/artif…
Context windows could be so much smaller (and models so much more compute efficient and fast), if instead of packing more tokens in, we knew how to pack more information into fewer tokens.
The amount of “knowledge” in a context window grows linearly with more tokens. But …
More than two years later, the last chapter of my dissertation is finally published! 😍 dl.acm.org/doi/10.1145/35701… As far as I know, this is now the state-of-the-art for monotonicity preserving spline interpolation software (1D). Fast, reliable, and twice differentiable.😎
Was recently asked what are some important theoretical research questions that can be addressed in ML right now, "fundamental" / generally applicable ones. I'll focus in on differentiable compute graphs & optimizers here (neural network training) here are 6 directions:
[1/7]
Prove that projecting data onto its principal components and scaling to componentwise unit 2-norm (or unit variance) as a preconditioner increases the initial diversity (spacing) of basis functions (when they are uniformly randomly initialized). [6/7]
Just finished implementing a truncated linear regression (reduces outputs with SVD then regresses). Will be working this into my neural network "conditioning" function to improve fit performance next!
Promising theory I'm working on: the best preconditioner for MLP's isn't just zero mean unit variance. It's projecting onto the principal components and *then* unit variance. This maximizes the radial symmetry of the data in the 2-norm, minimizing global curvature of MSE loss.