Yichen Sheng (@Coding_Black) profile picture

Research scientist in NVIDIA. Working in creating a world you cannot distinguish from reality. DLSS5 is my first step. Opinions are my own.

Joined July 2015
Photo by Yichen Sheng (@Coding_Black)Photo by Yichen Sheng (@Coding_Black)Photo by Yichen Sheng (@Coding_Black)Photo by Yichen Sheng (@Coding_Black)Photo by Yichen Sheng (@Coding_Black)Photo by Yichen Sheng (@Coding_Black)Photo by Yichen Sheng (@Coding_Black)Photo by Yichen Sheng (@Coding_Black)Photo by Yichen Sheng (@Coding_Black)Photo by Yichen Sheng (@Coding_Black)
Yichen Sheng retweeted
This Saturday I'm giving a talk at @saturdayrobotic in SF. Mostly about FLORA, our Data Engine used to train GR00T, and how I think about robot data going forward. Everyone is welcome. Come say hi! luma.com/ub5ask7s
🚀 @saturdayrobotic Reading Club #39: @nvidia Flora — Scaling Robot Learning to 200,000 Hours of Data Per Day! 🎤 Speaker: @yunzeman (Research Scientist, @nvidia GEAR) Keynote: NVIDIA Flora — The Open Data Engine for NVIDIA GR00T 📍 San Francisco 📅 Saturday, November 21 | 2–5 PM 🎟️ RSVP: luma.com/ub5ask7s Host: @junfanzhu98 Robot foundation models need data labels at every stage. Pretraining needs timestamped action captions, both to condition on language and to filter and rebalance large, mixed datasets. RL post-training needs process rewards that show where an episode goes wrong. Deployment needs outcome rewards to monitor fleets and evaluate policies. Today these labels come either from human annotators, which doesn't scale, or from frontier API models, which are too expensive at robot-data scale. Most existing tools also produce only one kind of label. This talk presents Flora, the open data engine used to label training data for NVIDIA's GR00T models. Flora consists of three models built on open VLMs. A temporal segmenter and a contextual captioner turn human and robot videos into timestamped action instructions. These range from single-camera egocentric clips to multi-view recordings of bimanual and dexterous-hand robots. A single reward model gives episode outcome scores and dense per-step labels that flag suboptimal actions and failures. It can also grade episodes against rubrics written at test time. 🔥 In production, it labels up to 200,000 hours of data per day. The talk will close with what an open data engine makes possible next: closing the loop between robot deployment and learning, human-to-robot transfer, and physical auto-research. See you at Saturday Robotics! 🤖 🎟️ RSVP: luma.com/ub5ask7s #NVIDIA #GR00T #Flora #PhysicalAI #Robotics #WorldModels #SaturdayRobotics
2
7
35
1,122
Yichen Sheng retweeted
Over the past week, I’ve been overwhelmed by the love and support of friends, industry colleagues, and people I've never met. Thank you to everyone who has shown kindness to my family and me. It means the world to me. I came to the United States 53 years ago, and becoming an American citizen is one of the greatest gifts and proudest moments of my life. NVIDIA has been my life’s work. Building it into the company it is today alongside my colleagues has been a privilege and an honor. Nowhere but in America could a company like NVIDIA be possible. I love our country, and I love NVIDIA. I did not live up to the standards expected of everyone at NVIDIA, and I have made serious mistakes in my personal life. I created problems for the Board that I deeply regret. I own my mistakes completely, and I am committed to cooperating with the Board’s independent investigation. Since my departure, I’ve seen an outpouring of calls from NVIDIA’s employees, investors, business partners, and customers for me to return as CEO. Many continue to reach out to me directly as the company navigates complex technology, market, competitive, and geopolitical dynamics. NVIDIA is a once-in-a-generation company. If the Board of Directors decides that my return is in the best interest of our shareholders, our employees, the company, and America, I would be honored to answer these calls and return as CEO. Thank you again to everyone who has shared their good wishes. I am profoundly humbled by your support. To my family: I love you, and I will spend the rest of my life being worthy of yours. Jensen
61,373
121,791
24,525
1,175,292
59,442,345
Yichen Sheng retweeted
Totally agreed with @TimSweeneyEpic Where is the journalism in @WSJ?
Replying to @WSJ
I can't believe this garbage. The article starts out by saying that NVIDIA's export strategy was lawful, then ignores that and tears into the company and its lawful export strategy with unsourced unsubstantiated suggestions that other companies might have done bad things.
8
25
337
21,907
Yichen Sheng retweeted
Replying to @GregoryDiamos
The media said it, and implied they were the source. Every time their claimed source denies it, they make up a new one. And now they are trying to redefine the English language to make their past false statements seem true-ish. I have never seen anything like this in industry.
48
94
8
1,357
21,216
Yichen Sheng retweeted
A heartbreaking day. I came to @nvidia as an intern. It was my first job, and I never left. For my whole career, @JensenHuang has been the leader I looked up to. He taught us to reason from first principles, that the mission is the boss, and that greatness comes from suffering. And he lived it every single day, working harder than any of us. I've never known a more inspiring leader. Thank you, Jensen. We'll keep building on what you started.
1,247
2,834
186
53,773
2,182,840
Yichen Sheng retweeted
Replying to @WSJ @WSJopinion
With fake news this good @TheEconomist @nytimes @WSJ , china’s plan is simple: Do nothing. Win.
4
36
1,951
Yichen Sheng retweeted
I’m deeply saddened that @JensenHuang is no longer leading NVIDIA. I once had the opportunity to meet Jensen alongside a few teammates. He was genuinely enthusiastic about our work and warmly shook hands with each of us. It was my first time shaking hands with him. More than the gesture itself, what meant the most to me was knowing that our hard work was recognized and valued by the leader of NVIDIA. I’ve always admired Jensen’s vision, conviction, and willingness to make bold decisions. He has consistently pushed NVIDIA to think big and do what truly matters. Thank you, Jensen, for your extraordinary leadership and for inspiring us to keep pushing forward. In the harshest winter, the plum blossom still blooms. To Jensen.
🤖 Made with AI
5
15
108
25,239
Yichen Sheng retweeted
Very inspired way to make a good use of data! Happy to see #DL3DV is contributing to industry level world models!
Excited to finally share C2R#! 🚀 Coarse-to-Real (C2R)# turns rough 3D and anime videos into photorealistic videos, without requiring paired video training data, by learning from much cheaper image pairs instead. Last year, we introduced C2R, using video generation models to turn rough 3D blockouts into photorealistic videos. But making this work reliably in a game engine is another story. Even models as powerful as Seedance 2.5 often require multiple attempts to get the desired result. For real applications, we need stable and controllable generation. This typically requires large amounts of paired coarse/real videos, which are extremely expensive to collect. So how can we get around this data bottleneck? One evening over dinner, while I was struggling with this problem, our intern Daohan proposed an interesting idea: Could we learn video translation from image pairs instead? After all, image pairs are much easier and cheaper to obtain. Over the following months, we worked hard to turn this idea into reality. We developed a lightweight Real-to-Coarse model trained only on image pairs. By applying it frame by frame to real videos, we can synthesize pseudo-coarse videos and create large amounts of pseudo-paired video training data. We then use these pairs to train a large Coarse-to-Real video generation model, without needing actual paired coarse/real videos! The resulting C2R# technology was eventually used at Roblox RDC. C2R# can also handle other input proxies, including anime-style videos, as long as you have 10K image pairs. 🌐 Project: gonzalognogales.github.io/co… 📄 Paper: arxiv.org/abs/2610.28954
1
5
615
Yichen Sheng retweeted
Dream Labs is hiring! Check out our open positions: dreamlabs.ai/careers/ Referrals are very welcome — please share with your amazing friends! I’ll also be at CoRL (Nov 9–12). If you’re interested, let’s chat in person!
14
15
1
257
17,470
Yichen Sheng retweeted
Introducing Flora, an open model suite for scaling robotics data engines. It watches robot and human video and annotate what happened (timestamped instructions) and how well it went (rewards and failure reasons). 🔗 research.nvidia.com/labs/gea… We use it to train NVIDIA GR00T. 🧵
21
32
15
191
24,784
Yichen Sheng retweeted
Today, Dream Labs is officially out of stealth! 🚀 What an incredible journey these past six months have been. I’ve always believed in two things: scaling and generalization. That belief led us to introduce World Action Models (WAMs), which learn from videos of the world and use that understanding to act. With enough scale, the right architecture, and the right data, I believe we can build a general-purpose robot brain that works across different robots, tasks, and environments. Yet despite incredible progress in robotics, most breakthroughs have remained in research labs, out of reach for the broader public. Six months ago, @jang_yoel , @scott_e_reed , @youliangtan , and I co-founded Dream Labs with a simple dream: What if anyone could give their robot a general-purpose brain, just through an API call? Today, we’re taking that first step. Dream-1 is now live for everyone Dream-1 controls bimanual robots with parallel grippers it has never seen before, zero-shot, on unseen tasks in unseen environments. But what excites me most isn’t just what Dream-1 can do today. It’s what we might discover together. With our public robot brain API, we want to put general-purpose robot intelligence into everyone’s hands. Try it on new robots, tasks, and environments we’ve never imagined. We want to empower everyone to discover their own “wow” moments with robots and explore what’s truly possible. ChatGPT wasn’t perfect when it first launched, but it opened up AI to the world. I hope today marks a similar beginning for robotics. Dream-1 is still a research beta. It will make mistakes, and there’s a long way to go. But someone has to take the first step. None of this would have been possible without our amazing team. I’m incredibly proud of what we’ve accomplished in just six months and grateful to be building alongside such talented people. ❤️ And we’re just getting started. If you share our dream, come build the future of robotics with us! 🚀 Everyday capabilities for everyday robots @dreamlabs_ai
Today, Dream Labs is coming out of stealth with Dream-1, a robot foundation model that controls bimanual robots it has never seen, on tasks it was never shown, in places it has never been, from $3/hour. Dream API is live for everyone, today. 🧵
15
14
2
166
11,710
Yichen Sheng retweeted
Please help spread the word! We’re hiring Research Scientists and Research Interns at NVIDIA Research to work on some of the most exciting problems in generative AI: 🌎 World Models & Video Diffusion 🧠 Diffusion Language Models 🧬 Biomolecule Design If your research aligns with any of these areas, please apply directly through the links below. Open positions + application links: research.nvidia.com/labs/gen…
5
58
9
488
68,661
Yichen Sheng retweeted
“5 years of graphics progress in one toggle” Now that people have had a chance to play with DLSS5, we can see that neural rendering has redefined real-time graphics. Ten years of dedicated research and development paying off. youtu.be/WY5u1-4h48Q
35
19
2
224
22,746
Yichen Sheng retweeted
Seeing two female scientists Dr.@LisaSu and Dr.@drfeifei shaping the future of AI together moved me deeply. Hoping more female see a place for themselves in that future and find the courage to follow their curiosity, pursue their own North Star, and take on the hardest scientific questions.
1
11
1,120
Yichen Sheng retweeted
Can't wait to see some of you in Berlin!
Open models are how developers build AI they can own. Not just weights. Model families, post-training recipes, evaluation tools, and supporting data give teams the ability to inspect, adapt, and deploy AI for real applications. At GTC Berlin, Bryan Catanzaro, VP of Applied Deep Learning Research at NVIDIA, will show developers what that looks like in practice. 🎙️ Building Open Models That Developers Can Inspect, Adapt, and Deploy: nvda.ws/4ivv0qH 📍 #NVIDIAGTC Berlin | October 20–22, 2026 🔗 Register with code GTC-NVP8A3H2 for 25% off: nvda.ws/4j297iN
5
8
119
10,017
Yichen Sheng retweeted
Hello everyone! Me and @davnords will be presenting RoMa v2 and LoMa back-to-back at 9AM in the Geometry, Localization, Matching & Motion spotlight session at #ECCV2026
1
2
1
28
4,391
Yichen Sheng retweeted
Impressed by GPT-Astra’s CoT ! This Kung Fu demo is built from scrach in Blender and takes the reference video + simple prompt as input. The CoT reasoning really show a grasp of kinematics, joint coordination, and body motion ! Reference video comes from PhysAvater @qingqing_zhao_ @yang_zheng18
5
3
42
4,442
Yichen Sheng retweeted
GPT-6 Astra feels deeply personal. In summer 2024, during my internship with #NVIDIACosmos team, we started Scenethesis(research.nvidia.com/labs/cos…) with GPT-4o: an agent that could imagine a 3D scene, ground it through promt/vision, align it with physics, judge the result before replaning. Those were good old days. Two years later, the dream has grown far beyond a single scene. Astra brings much stronger multimodal reasoning and long-horizon tool use; Blender and Unreal are becoming canvases for intelligence. Watching the base model advance this quickly is astonishing—and incredibly exciting. The mission ahead excites me most: agents that create, simulate, and explore worlds; accumulate knowledge from experience; and keep evolving—for games, robotics, and embodied AI. Huge congratulations to the brilliant @OpenAI team. What a beautiful moment for the agentic world!
Replying to @OpenAI
GPT-6 Astra is the most intelligent and aligned model in the world, and sets a new state of the art for computer use, browsing, software engineering, cybersecurity, science, and professional work. openai.com/index/gpt-6-astra…
1
1
20
2,801
Yichen Sheng retweeted
🚀🚀 We officially release DLSS 5, a real-time, 3D-guided neural rendering stage that transforms rendered frames into vivid, realistic scenes while preserving developer-authored content and design intent. To achieve both enhanced realism and real-time rendering at up to 4K resolution, we develop a one-step pixel-space diffusion model. Compared with offline foundation models such as GPT Image 2 and Gemini 3 Pro, DLSS 5 delivers strong realism while much more faithfully preserving the input content, identity, and scene structure. Looking ahead, we’re continuing to push on realism, controllability, temporal stability, and efficiency. Stay tuned! Check out our technical report for more on how we think about generative modeling for gaming: research.nvidia.com/labs/adl… I feel excited to finally see DLSS 5 released! As a researcher working on visual generative modeling, it’s especially meaningful to see our fundamental research make its way into a real product. This has actually been a very special journey for me—from exploring research ideas, to building our first working prototype, and then working closely with the team to turn the research, step by step, into DLSS 5. Proud of what our team has accomplished! We’d love to hear your thoughts on how we can continue improving the technology and where generative rendering can take the future of gaming. Discussions are very welcome!
We’re publishing a deeper technical report on why generation is needed for the next leap toward photorealism. VRAM and compute limit both the scene abstraction and the number of rays we can afford. Previous DLSS reconstructs from that representation—so even perfect reconstruction is still bounded by what the scene contains. DLSS 5 is the first DLSS to generate beyond that abstraction and break through that ceiling, while staying grounded in the game and controlled by the artist. We also compare it with offline generative models and renderers at similar scaling cost. research.nvidia.com/labs/adl…
11
5
53
25,316