Research Intern @Bytedance (San Jose) & Ph.D. student on LLM @UChicagoCS Prev. intern @UWNLP @BerkeleyNLP, BS @XJTU1896 24’
San Jose, CA
Joined February 2023
- Tweets148
- Following392
- Followers362
- Likes848
[5/6] Open questions: Can we improve perceived creativity without an explicit novelty objective? How can current training recipes be adapted to improve tail performance?
Curious to hear your thoughts and welcome any feedback!
[6/6] With @TenghaoHuang45 (Microsoft), @vishakh_pk (Stanford) & @MinaLee__ (UChicago). Many thanks to them!
Code/data: github.com/YichenZW/longtail…
Paper: arxiv.org/abs/2610.12998
Explore our taxonomy and reading list of LLM creativity research: github.com/YichenZW/awesome_…
When a new model climbs a creativity leaderboard (e.g., LM Arena’s creative-writing ranking), what has actually been measured - and what has improved?
Our new paper examines this and points to one shared target: long-tail performance.
15 benchmarks. 26 open-weight models. 🧵
ALT First page of “Evaluating LLM Creativity through Long-Tail Performance” by Yichen Wang, Tenghao Huang, Vishakh Padmakumar, and Mina Lee. Includes the abstract and a diagram showing creativity benchmarks in the long tail of a reference input–output distribution. arXiv:2610.12998.
[4/6] Applying q@Tail reveals a gap: most creativity-promoting methods we test increase output novelty without consistently improving quality in the tail.
This points to a concrete target for improvement, such as taking better actions in less familiar situations.
ALT Table 1 compares ten creativity-promoting methods with a vanilla model. Columns show overall quality, quality at tail (q@Tail, alpha = 25%), overall novelty, and novelty at peak. Most methods increase novelty but reduce tail quality. Min-p raises tail quality by 17.7% while reducing both novelty measures. Subscripts give relative percentage changes; green means increase and red means decrease. Results are averaged across four representative models and the evaluated benchmarks.
[3/6] Given these findings, we introduce q@Tail(α):
1. Select the most novel α fraction of an existing benchmark using input-output novelty.
2. Measure each model’s average task performance on that same subset.
A practical creativity diagnostic using benchmarks you already have
[2/6] Finding 2: On each benchmark, model rankings on higher-tail subsets align more closely with human creativity judgments in LM Arena’s creative-writing category.
This relationship holds not only on creativity benchmarks but also on standard benchmarks (e.g., GPQA).
ALT Original Figure 5: GPQA-Diamond examples divided into five bins from peak to tail using input–output novelty. The table reports agreement with human creative-writing rankings: Kendall’s tau is 0.099, 0.154, 0.175, 0.396, and 0.222 for bins 1–5. The two highest-tail bins show stronger agreement than bins 1–3.
[1/6] Finding 1: Creativity benchmarks sit further in the long-tail region than standard benchmarks, across writing, math, and QA.
Their inputs, outputs, or both are less common in a reference corpus (we estimate novelty using n-grams absent from the Pile).
ALT Figure 2 from the paper. Input novelty is on the horizontal axis and output novelty on the vertical axis. Distributions are shown for writing, mathematics, and question answering benchmarks. Creativity benchmarks shift toward higher input novelty, output novelty, or both compared with standard benchmarks. Novelty is the fraction of 4-grams absent from the Pile; output novelty is averaged across evaluated models.
[5/6] Open questions: How can we improve perceived creativity without an explicit novelty objective? Can current training recipes be adapted to improve tail performance?
Curious to hear your thoughts and welcome any feedback!
[6/6] With @TenghaoHuang45 (Microsoft), @vishakh_pk (Stanford) & @MinaLee__ (UChicago). Many thanks to them!
Code/data: github.com/YichenZW/longtail…
Paper: arxiv.org/abs/2610.12998
Explore our taxonomy and reading list of LLM creativity research: github.com/YichenZW/awesome_…
Yichen (Zach) Wang retweeted
Every KV cache evictor for efficient long reasoning scores cached tokens to decide which ones to keep. We find the score buys almost nothing: keep the prompt, evict at random per head, and you match the strongest evictor at 32–43% higher throughput.
Paper: arxiv.org/abs/2609.03430
Page: arthur-heng.github.io/Random…
Yichen (Zach) Wang retweeted
Amazing to see all the excitement around RSI, which finally feels within reach.
I pitched RSI back when I was at FAIR in 2023, right after we did Toolformer, which was arguably too early. But we got to see the first signs it may be possible while working on MLGym with @deepaknathani11.
I shared my vision for how we can achieve RSI and Superhuman Scientific Discovery in my @raais talk earlier this year.
Spoiler alert: open-endedness is a key piece of the puzzle.
and full talk here! youtu.be/Tek-FwtEwTk?si=WJwR…
Yichen (Zach) Wang retweeted
Definitely not on my bingo card for 2026, but we got to analyze human-AI collaboration patterns from in-the-wild Claude traffic!
For the first time, we’ve given external researchers a way to study AI’s impacts using real, privacy-preserved Claude usage data. To date, this work has only been possible within AI labs. We can’t tell the whole story alone, so we opened up our tools.
anthropic.com/research/enabl…
Yichen (Zach) Wang retweeted
🎉 Our paper “LLM Probability Concentration: How Alignment Shrinks the Generative Horizon” is accepted by TMLR!
The short version:
📉 LLM generation usually self-narrows
✂️ Alignment compresses the horizon further
⚡ Unexpected context can reopen it locally
We study these dynamics with Branching Factor (BF)—exponentiated length-averaged entropy, interpreted as the effective number of plausible next steps.
Direct evidence first: across tasks, BF typically declines as more tokens are generated, in both base and aligned models.
Then we intervene. Replacing the model’s own prefix with equally long random tokens makes BF jump; continued autoregressive generation narrows it again. Self-narrowing is therefore a robust aggregate tendency, not a monotonic token-wise law—and not an artifact caused only by alignment.
Alignment is a separate force: it lowers BF by 2–5× overall and up to ~10× at early positions. Reasoning models also remain lower-BF than direct-answer aligned models at matched output lengths, so their concentration is not merely because they generate longer.
Most importantly, BF can rise in a structured setting. In synthetic agentic tasks, we hold the task, environment state, and current plan fixed, then change only the next environment feedback. A plan-invalidating surprise raises BF relative to normal progress; continued generation subsequently smooths the local spike.
So the dynamics are not simply “entropy collapse.” The model narrows, surprise can reopen its consideration set, and self-conditioning narrows it again.
Practical implication: when exploration matters, branch early—or introduce genuinely NEW information before the model has fully committed. 🧵
Learn more from our Branching Factor v1.1 tweet: xitter.cf/chrome1996/statu…
BranchingFactor v1.1 just dropped! 🚀
(Yes — it’s an actively updated paper.) (arxiv.org/abs/2506.17871)
As models rely more on post-training, understanding the synergy between pre-training and alignment becomes crucial. Branching Factor (BF) offers a simple way to track the remaining generative potential of a model — since entropy inevitably decreases during generation, BF measures that process.
What’s new in v1.1:
1️⃣ Major rewrite
We now introduce BF directly — much clearer and easier to read.
2️⃣ Theorem correction + extension
Thanks to @StarLi27496427 and Yuwei for catching my misunderstanding of the AEP theorem! We fixed the derivation and extended it to variable-length LLM outputs.
The good news: the main result still holds — length-avg log-likelihood can estimate length-avg entropy for sufficiently long generations, in a memory-efficient way. Useful if you want to monitor entropy during training or inference.
3️⃣ Broader evaluation
Added experiments on OLMo2 and Qwen3, plus multilingual and long-context tasks.
Key findings so far still holds often:
📉 BF decreases during generation
✂️ Alignment significantly reduces BF
⚖️ Interestingly, OLMo2 appears less aggressively shrunk by alignment than Qwen3/Llama3 (preliminary observation).
4️⃣ SFT vs RL analysis
We started dissecting how SFT and RL affect BF. Early signals from OLMo2:
🧠 Smaller models: BF shrink mostly happens during SFT (possible memorization effect).
🏗️ Larger models: SFT and RL have comparable impact.
Still very preliminary — but it raises interesting questions about how post-training should scale with model size.
Yichen (Zach) Wang retweeted
Automatic harness evolution appears to be a promising path toward AI self-improvement, but we find that its gains still largely come from repeated sampling and show limited generalization.
Blog post: yikee.github.io/harnessevolu…
Code: github.com/rethinking-harnes…
Yichen (Zach) Wang retweeted
Excited to give a talk at the @icmlconf RLxF Workshop today at 3:30pm. I’ll be talking about the role of open-ended esss in achieving superhuman scientific discovery.
Thank you @shaohua0116 @shaneguML et al. for organizing this!
We're excited to welcome an outstanding lineup of speakers at the RLxF Workshop: Benjamin Eysenbach @ben_eysenbach, Chelsea Finn @chelseabfinn, Jesse Zhang @Jesse_Y_Zhang, Roberta Raileanu @robertarail, Jerry Tworek @MillionInt, and Brian Zhan @brianzhan.
[🛫ICML] We will be presenting our paper about *base-aligned model collaboration* on Tuesday morning and also Saturday (HAI Co-Creativity workshop). Happy to chat!
PS: We've updated our paper, code, and reading list to include the latest results and resources. Check it out!
Lack of diversity in your LLM generation?
(also noted by Artificial Hivemind, best paper @NeurIPSConf)
Time to bring your base model back!
An inference-time, token-level collaboration between a base and an aligned model can optimize and control diversity and quality!
