Research @Microsoft - Frontier Tuning | Scalable & Efficient RL for Frontier LLMs | PhD @ KAIST | ex-@mila_quebec (Bengio Lab) | Views are my own.
Redmond, Washington, USA
Joined March 2025
- Tweets118
- Following162
- Followers279
- Likes456
Pinned Tweet
TSP is one of the cleanest RL proxies for LLM reasoning and agents: combinatorial decisions, long horizons, and sparse terminal rewards.
People trained Transformers to generate near-optimal tours for this NP-hard problem. RLOO (a popular LLM RL method) was introduced by Wouter Kool et al. on TSP in 2019.
In my past work called GFACS, off-policy GFlowNets + metaheuristic search outperformed policy-gradient baselines on TSP. I expect this to carry over to LLM agents.
Recent work on GFlowRL (applied at 235B LLM) from Microsoft Research is encouraging in this regard.
GFACS: proceedings.mlr.press/v258/k…
GFlowRL: arxiv.org/abs/2607.13394
Minsu Kim retweeted
Decided to revive the blog and write a short post about embracing the agents and increasing the scope of what kind of research is possible. Very much in line with the general experience but a good excuse to get back to writing blog posts (more soon!)
mj10.github.io/blog/2026/fem…
Minsu Kim retweeted
There is some very nice non-LLM literature on this topic (e.g. arxiv.org/abs/2003.06350 and arxiv.org/abs/2106.03955 by @folinoid ), but the findings need to be revisited to account for the effects of starting a strong pretrained policy
Minsu Kim retweeted
At least some settings (model-data combinations) benefit from IS on top of this kind of regression objective, but in others IS isn't needed.
(IIUC, the TBA objective below matches CORO when you set the reference to be the inference policy and drop IS).
xitter.cf/bartoldson/status/1996…
Minsu Kim retweeted
Highly recommended read from Google DeepMind and colleagues.
There is a lot of interest in self-evolving agents, but they're hard to get right.
This work highlights how important and difficult memory and retrieval are in agents.
The big question in this paper is whether self-evolving agents keep learning without forgetting.
They tested ReasoningBank, GEPA, and Meta-Harness on four task streams that drift in topic, workflow, or domain, built from BIRD, BFCL, and ScienceWorld.
The weights stay frozen, and the agents adapt by rewriting prompts, memory, and skills.
Self-evolution improves overall scores on most streams. But more adaptation consistently means agents forget more.
Methods that rewrite a shared prompt or workspace make global edits, so they interfere with unseen tasks and overwrite older knowledge when the stream shifts.
Memory retrieval keeps updates more local, but it still pulls in memories from other domains and collides when similar queries need different workflows.
Static benchmarks hide both problems, so test self-evolving agents on a stream that changes over time.
arxiv.org/abs/2610.25858
Chat with Paper: academy.dair.ai/papers/are-s…
Minsu Kim retweeted
“All roads to off-policy learning go through path consistency”
The CORO objective used here is equivalent to the TBA objective proposed in our paper from last year. All of these are variants of PCL and in the on-policy trajectory case yields same gradient as REINFORCE
arxiv.org/abs/2503.18929
Minsu Kim retweeted
New Google DeepMind paper finds that self-improving agents forget as they learn, and the method that learned new tasks fastest broke old ones the most.
Self-evolving agents improve without retraining by editing their system prompt, memory, or skill files. In real use tasks keep shifting, and each edit made for 1 task also applies to every task that follows.
The team ran ReasoningBank, GEPA, and Meta-Harness on task streams that changed topic, workflow, or domain. They tracked how much each method learned on the current task and lost on the others.
GEPA learned the most, 6.2 points per block on average, and also lost the most elsewhere, 4.3 points. After it learned to ask users for missing details, a later block rewrote that rule and its 15.5-point gain fell to 2.5.
If your agent edits its own prompt or memory, keep task-specific rules separate and retest old tasks after every update.
Minsu Kim retweeted
The CORO objective used here is equivalent to the TBA objective proposed in our paper from last year. All of these are variants of PCL and in the on-policy trajectory case yields same gradient as REINFORCE
arxiv.org/abs/2503.18929
No one changes the world alone
Introducing Ampersand - the first model built for us. Ampersand is a collaborator trained to help people achieve their individual and collective goals. In multi-month simulations, users collaborating with Ampersand accomplished more than with Fable and other frontier models
Available to everyone through our API
Minsu Kim retweeted
context-beyond-window.github…
If you're attending COLM, do not miss @Yoshua_Bengio's 9am talk at our workshop on Friday!
Recursive self-collapse of AI academia: Many great papers offer deeper insights into paths to superintelligence than industry does. But curation has become sampling from a slop-dominated distribution. AI-assisted curation feeds slop back to researchers, generating even more slop. Industry could be next.
Minsu Kim retweeted
In LLM post-training, policy gradient is "the RL loss".
When it fails, we blame the RL: exploration, credit assignment, sampling noise.
Image classification has none of these, and the policy gradient is exact. Following it is terrible...
ImageNet, 4% vs 62% for cross-entropy.
Recursive reasoning adds a new scaling axis: loop depth. Small models can solve hard problems, with more headroom as models scale.
GRAM turns recursive reasoning model into a sampler, adding width scaling through diverse parallel reasoning trajectories. #NeurIPS2026
GRAM is accepted to #NeurIPS2026! 🎉
Turning recursive reasoning into a stochastic latent trajectory — multiple hypotheses, and inference-time scaling in both depth and width.
A revised version of the paper and the code will be released soon!
🌐 ahn-ml.github.io/gram-websit…
General intelligence is compute-inefficient.
At the same compute budget, specialized intelligence will almost always beat general intelligence on the task it specializes in.
Or put differently: a small specialist can often match a much larger generalist.
And a mixture of specialists may be a much better scaling path than one monolithic intelligence.
There is no free lunch for general intelligence. Generality has a compute tax.
Value functions are severely underestimated in LLM post-training. People often count on compute scaling to compensate for the high variance of Monte Carlo estimators without credit assignment—broadcasting the same reward to all tokens.
Value functions should be revisited to push the frontier toward longer-horizon agentic tasks. We need to make value functions work, not simply throw them away.
Nice work from @MistralAI and @Siddarth on privileged value functions and TETHER.
LE CRITIQUE: Privileged Value Functions for LLM Reinforcement Learning
Value functions for LLMs have been grossly underutilized. I'm excited to finally share my work on Privileged Value Functions, done @MistralAI !
Our paper includes 2 massive improvements to value functions:
1) Privileged Value Functions: Turns out value functions can use hidden context unavailable to the policy for improved token-level RL signal! Condition them on reference solutions, verifier rubrics, or even other group samples and rewards. No more hacky self-distillation necessary! (being a tad hyperbolic)
2) TETHER: Do not trade off group and value baselines, you can just do both! We introduce a new baseline that adaptively trades off between the group mean and value baselines adaptively based on value function reliability.
🌐 privileged-value-functions.g…
📄 arxiv.org/pdf/2608.16739
Code: github.com/HyperPotatoNeo/pr…
🧵below
Minsu Kim retweeted
Let's bootstrap reasoning models by training on tasks with expert-level solutions (e.g., game solvers, robotic planners, theorem provers).
But, where do we get the CoT labels? Just asking another LM to impute the CoT doesn't work.
Solution: variational inference over CoT!
We've been thinking can we distill the knowledge humans encoded in expert systems into LLMs, and make it general?
Decades of human engineering live inside expert systems. Solvers, planners, schedulers: each one encodes real decision-making knowledge, but as a program that acts in a single domain and explains nothing.
Paper: arxiv.org/abs/2607.21856
More👇:
Minsu Kim retweeted
Can LLM agents figure out that their "private" chain of thought is being monitored, with no hints, no training, just from getting blocked?
Yes. And some then explicitly reason to hide their reasoning.
New paper with @FlemmingKondrup (co-first authors) & @Yoshua_Bengio: "Noticing the Watcher" 🧵
Minsu Kim retweeted
New work led by @FlemmingKondrup and @tomjiralerspong highlights an important vulnerability in chain-of-thought monitoring for agents, I highly recommend giving it a read.
Link to the paper: arxiv.org/pdf/2603.16928
Can LLM agents figure out that their "private" chain of thought is being monitored, with no hints, no training, just from getting blocked?
Yes. And some then explicitly reason to hide their reasoning.
New paper with @FlemmingKondrup (co-first authors) & @Yoshua_Bengio: "Noticing the Watcher" 🧵