Yining Chen (@cynnjjs) profile picture

Researcher@OpenAI

Joined October 2015
Photo by Yining Chen (@cynnjjs)Photo by Yining Chen (@cynnjjs)
Yining Chen retweeted
From various reports OpenAI really did bolt a “Universal Verifier” onto the GPT-5 training loop. And here's that paper, that OpenAI published earlier. "Prover-Verifier Games Improve Legibility of LLM", showing a production-ready pipeline where a verifier model scores each reasoning chain and feeds that reward back into policy updates. The paper is explicit that the verifier is small enough for large-scale rollout and is “designed for future GPT deployments”. 🔄 How the prover-verifier game works Think of two personalities living in one model. The “helpful” persona solves a problem and tries to convince a lightweight verifier network that the answer is sound. The “sneaky” persona deliberately sneaks in wrong conclusions yet still aims to fool the same verifier. By alternating roles, the big model learns to write solutions that are harder to fake, while the small verifier sharpens its ability to flag errors. --- An Aug-2024 Wired article explained how OpenAI swapped some human feedback with model-based critics when fine-tuning GPT-4 code helpers, noting the system “will be folded into RLHF for future mainline models” GPT-5 is that next mainline model.
📐 OpenAI GPT-5 will be a steady step that lifts coding, math, and agent control, not another giant jump from GPT-3 to GPT-4. According theinformation report. OpenAI hit three big snags at once: fresh data dried up, reinforcement learning runs kept wobbling, and the Orion model never lived up to its hype. To keep things on track, the team built a universal verifier, an extra model that grades every answer during reinforcement learning and only lets the solid ones loop back into training so the next model starts from cleaner, more reliable examples. OpenAI spent early 2024 training a bigger model called Orion to replace GPT-4, but the tweaks that helped small test runs failed to scale and clean new data was scarce, so results stayed close to GPT-4 while costs kept rising. Because of that, the company rebranded Orion as GPT-4.5 rather than GPT-5 and shifted its focus to other training tricks Teams pivoted to o-series reasoning models, adding more NVIDIA GPUs and code search; raw problem solving rose but quality dipped once the model had to chat in plain English. GPT-5 folds those lessons together: it scales compute per query, writes cleaner interfaces, handles tricky refunds, and lets the universal verifier grade thousands of synthetic answers.
7
46
10
253
34,167
Yining Chen retweeted
Another Superalignment paper from my time at OpenAI: We train large models to write solutions such that smaller models can better check them. This makes them easier to check for humans, too. openai.com/index/prover-veri…
10
82
9
663
78,797
Yining Chen retweeted
We trained advanced language models to generate text that weaker models can easily verify, and found it also made these texts easier for human evaluation. This research could help AI systems be more verifiable and trustworthy in the real world. openai.com/index/prover-veri…
206
535
111
3,305
703,300
1/ Excited to share our new paper: "Prover-Verifier Games Improve Legibility of LLM Outputs"! We explore how to make large language model (LLM) outputs easier to check and thereby more legible. 🧵
1
11
1
69
24,020
Excited to share our new work “Prover-Verifier Games improve legibility of language model outputs”! We trained strong language models to produce text that is more checkable by weak language models, and found that this also made it more legible to humans. openai.com/index/prover-veri…
1
13
1
131
25,551
We found that optimizing for correctness on math problems leads to solutions that are less legible. Initialization pretrained on human-written math derivations is more legible, but has low accuracy. Our algorithm retains legibility, while partially recovering the accuracy.
1
6
500
I’d like to thank my amazing collaborators @janhkirchner @burdayur @HarriLEdwards @__nmca__ and @janleike as well as the platform, human data, and comms team at @OpenAI for their support to make this work happen!
1
11
424
Super excited about our new research direction for aligning smarter-than-human AI: We finetune large models to generalize from weak supervision—using small models instead of humans as weak supervisors. Check out our new paper: openai.com/research/weak-to-…
5
38
16,975
Yining Chen retweeted
In the future, humans will need to supervise AI systems much smarter than them. We study an analogy: small models supervising large models. Read the Superalignment team's first paper showing progress on a new approach, weak-to-strong generalization: openai.com/research/weak-to-…
501
1,235
326
6,475
2,515,599
Yining Chen retweeted
Super excited about our new research direction for aligning smarter-than-human AI: We finetune large models to generalize from weak supervision—using small models instead of humans as weak supervisors. Check out our new paper: openai.com/research/weak-to-…
66
301
107
1,842
1,382,451
Will be at NeuRIPS Dec 10-16! Let me know if you want to discuss anything about LLM / Alignment.
1
11
1,009
Yining Chen retweeted
📢 Exciting new research: CaML is a causal meta-learning framework for personalized prediction of the effects of novel interventions. 🧪🔍 From personalized medicine to public policy and online marketing, CaML outperforms traditional methods by leveraging intervention's and individual's features. 🚀 #CausalMetaLearning #PersonalizedPrediction #InnovationInTheMaking Great work by @h_nilforoshan @Michael_D_Moor @yusufroohani @cynnjjs Anja Surina n and Sara Oblak.
The same drug 💊 can harm one patient and save another’s life. Can we predict what will happen (AI for personalized medicine), without years of trial & error for each drug? Our work on zero-shot causal learning takes an important step towards this goal. arxiv.org/abs/2301.12292
1
12
1
86
19,387
Many invariance-based approaches to domain generalization suffer from linear "environment complexity" (# environments needed to identify inv. features). In our #NeurIPS2022 paper, we show a *non end-to-end* feature matching algorithm can achieve logarithmic dependence!
In the invariant feature approach to domain generalization, the goal is to identify invariant features after seeing a small # of environments. How many envs are needed? We study this q in a variant of a toy data model we introduced w @ElanRosenfeld in arxiv.org/abs/2106.09913.
1
2
11
Yining Chen retweeted
RNNs and transformers with *infinite*-precision are universal approximators---but are they statistically learnable? We show that even finite-precision transformers can express any Turing machine, and they are learnable with polynomial samples. arxiv.org/abs/2107.13163
4
39
240
Yining Chen retweeted
Past theory for domain generalization often suggests the number of training domains has to scale linearly in dimension. Any hope to work with a more realistic # of domains? We give an algorithm that needs log-in-dim # of domains on a toy data model arxiv.org/abs/2106.09913. 1/6
2
32
2
118