Andrew Tao (@drewtao) profile picture

VP of Applied Research, NVIDIA Nemotron Multimodal models, DLSS

Joined March 2012
Photo by Andrew Tao (@drewtao)Photo by Andrew Tao (@drewtao)Photo by Andrew Tao (@drewtao)Photo by Andrew Tao (@drewtao)Photo by Andrew Tao (@drewtao)Photo by Andrew Tao (@drewtao)Photo by Andrew Tao (@drewtao)Photo by Andrew Tao (@drewtao)
🚀 Nemotron 3.5 Super VL is here! Double AA index with no architectural changes! 💚 Same architecture as Super-v3 (120B total / 12B active hybrid Mamba-Transformer MoE). AA Intelligence Index doubles (13 → 26) with the new training recipe: What we added: • Joint SFT on vision + language data from the first update • Logit distillation as continuous pretraining from Ultra for 1T tokens. • Agentic mid-training (boosts software-engineering / long-horizon work) • Improved C-RADIO vision encoder for better document analysis. 📃 All details in the tech report: research.nvidia.com/labs/nem… 🤗 Model weights on HF (NVFP4): huggingface.co/nvidia/NVIDIA… Blog: developer.nvidia.com/blog/nv…
3
7
1
60
5,504
Nemotron Super 3.5 … Smarter and less verbose than ultra. Also our best vision model by far. This was a good tight sprint. Hope it’s useful to you!
Nvidia’s Nemotron 3.5 Super VL is now the leading US open weight model, scoring 26 on the Artificial Analysis Intelligence Index Nemotron 3.5 Super VL achieves twice the intelligence of its predecessor, Nemotron 3 Super (13), and does so using ~60% fewer output tokens per task It launches alongside Nemotron 3.5 Bolt 74B-A9B 24GB, the first model in its new Bolt family, which scores 21 on the Intelligence Index Congratulations to @NVIDIAAI on the releases! Key Takeaways: ➤ Leading US open weight model: Nemotron 3.5 Super VL scores 26.5, edging out Thinking Machines' Inkling Small (25.7) and ahead of NVIDIA's Nemotron 3 Ultra (23) and Meta's Muse Glimmer (17) ➤ Bolt comes within 2 points of Ultra: Nemotron 3.5 Bolt 74B-A9B 24GB scores 21, 1.5 points behind Nemotron 3 Ultra 550B A55B (23). The 2 bit memory has about a seventh of Ultra's parameters (74B vs 550B) and a sixth of its active parameters (9B vs 55B) ➤ Super 3.5 VL near lossless in NVFP4: Nemotron 3.5 Super VL releases in NVFP4, scoring within 0.1 points of its BF16 weights on the Intelligence Index, with no component moving by more than 2.6 points Nemotron 3.5 Super VL details Context window: 1M tokens Modalities: Vision and text input; text output Parameters: 120B / 12B Nemotron 3.5 Bolt 74B-A9B 24GB details Context window: 256k tokens Modalities: Vision and text input; text output Parameters: 74B / 9B
2
2
327
We've been cooking! 👨‍🍳 DLSS 5 with 3D-Guided Neural Rendering is being upgraded with a 2X faster model for up to 1.4X faster framerates - available first in NBA 2K27 Season 2 on October 16. Learn More ➡️ nvda.ws/4dUwbwP
9
12
2
88
45,237
Games are entertaining not just because they’re pretty but moreso because a team of developers has built an amazing experience and world/story. Agents and NR can help realize a vision but you truly need devs to craft an experience.
4
274
Andrew Tao retweeted
We’re publishing a deeper technical report on why generation is needed for the next leap toward photorealism. VRAM and compute limit both the scene abstraction and the number of rays we can afford. Previous DLSS reconstructs from that representation—so even perfect reconstruction is still bounded by what the scene contains. DLSS 5 is the first DLSS to generate beyond that abstraction and break through that ceiling, while staying grounded in the game and controlled by the artist. We also compare it with offline generative models and renderers at similar scaling cost. research.nvidia.com/labs/adl…
28
41
16
400
131,408
A NVIDIA diz que as otimizações do DLSS5 serão lançadas de maneira bem mais rápidas, fazendo o modelo ficar mais leve e maior qualidade a cada update. Inclusive teremos um novo update para o DLSS5 até o final do ano.
3
15
2
157
7,785
Andrew Tao retweeted
DLSS 5 Neural Rendering is a huge step towards the future of real-time graphics and the realization of 10 years of research. It is controllable, consistent, and fast, and I can’t wait to see what game developers accomplish with this new tool. youtu.be/58FagrSqC4M
67
34
10
381
37,062
What we're trying to do with DLSS5/NR is challenging: to provide uplift while preserving artistic intent. Ultimately, the goal is to give the dev more power to achieve their vision, not less.
2
1
296
It will be up to the dev to tune how it's applied, and up to the user whether to enable it. Either way, we're vry excited about the future of realtime gfx!
189
Andrew Tao retweeted
Replying to @NVIDIAAI
@NVIDIAAI Nemotron 3 Nano Omni on your DGX Sparks is something wonderful to experience 🤩
My second recipe, this is a quick win for all your DGX Spark and local AI folks I just replaced gemini-flash, openai, and claude with my own model running on a single DGX spark to give my entire agent fleet vision, audio understanding, and video understanding A lot of LLM models lack vision, and other modalities, you can solve it super quick by just spinning this up. @nvidia 's Nemotron-3-nano-omni-30B is a multimodal perception engine: Computer-use, vision, audio, and video built into one MoE checkpoint. 31.6B params, 3B active. It takes 21GB on the spark. you still have 86GB free — run it alongside your main model. i have it serving as the vision sub-agent for every agent in my fleet, replacing three separate cloud API calls. Here's what I tested to verify it can replace my cloud AI endpoints: → sends it a news screenshot, it reads the headline, identifies the UI, reads the branding. 7s. → sends it a photo of toys on a shelf, it catalogs 12+ objects. 9s. → sends it a full 2:46 song as raw audio, it transcribes the lyrics, identifies the key (C#m), tempo (~91 BPM), instruments, and song structure. 17s. → sends video, it identifies the show (Arcane), reads Netflix branding, classifies the genre. 9s. The thing I found out: vLLM already supports audio input through the standard API. you just need pip install "vllm[audio]" — the parakeet ASR encoder was sitting there the whole time, nobody turned it on. One serve command, all modalities: vllm serve nemotron-omni-nvfp4 \ --limit-mm-per-prompt '{"image":4,"video":1,"audio":4}' 21GB checkpoint. 86GB VRAM still free. 62 tok/s decode. zero cloud. zero API costs. zero rate limits. full recipe + benchmarks + findings → [reply link]
1
1
1
6
1,405
Andrew Tao retweeted
For my first post, I’m sharing a letter @NVIDIA signed on why open models matter. AI will transform every industry, power every company, and be built by every country. Open models strengthen safety and cybersecurity, accelerate innovation and diffusion, and enable sovereignty. The world needs both frontier closed models and frontier open models. images.nvidia.com/pdf/Open-W…
16,548
34,449
12,082
193,762
69,458,367
Andrew Tao retweeted
Why are open technologies for AI so important? How and why is NVIDIA building Nemotron? What can we learn from China’s AI efforts? Great conversation with Matt Turck. youtu.be/Oojrfdl42LI
9
23
4
87
21,342
Andrew Tao retweeted
Selected as a best paper finalist at #CVPR2026: PixelDiT from NVIDIA Research In most image generation models, a pretrained autoencoder compresses the image before any diffusion happens, causing quality loss that accumulates across the entire pipeline. PixelDiT, or Pixel Diffusion Transformers, removes this step entirely. It's a single-stage model that learns the diffusion process directly in pixel space, end-to-end.
20
35
4
253
30,692
Replying to @nvidia
@NVIDIA introduces Nemotron 3 Nano Omni LLMs. A year ago, we started research on omni-modal LLMs—covering everything from architecture to data—and released OmniVinci (ICLR 2026). We’ve received some feedback since then, but two clear themes emerged: 9B parameters weren't quite enough for many applications, and restrictive licensing was holding back commercial use. Therefore, we’ve teamed up with a larger group of amazing NVIDIA researchers/engineers to introduce NVIDIA's first commercial omni-LLM Nemotron 3 Nano Omni. This new 30B-A3B model is fully open-source and commercially usable. It also consistently outperforms similar-sized models across image, video, audio, and omni benchmarks. My primary focus on this project was Omni-RL infra development and model training, besides architecture/hyper-parameter exploration and verification. Here are a few key takeaways from the process: • Cross-modal synergy is real: In multimodal RL, improvements in one modality reinforce the others, translating directly into stronger overall benchmark scores. • Data quality beats quantity: When it comes to RL, a steep reward curve slope is often much more critical than sheer training duration. • Untapped potential in RL infra/algorithm: Especially for multimodal setups, there is still so much room to explore and optimize underlying architectures and scheduling systems. To better support both the research community and industry developers, we aren't just releasing the model weights—we are also open-sourcing the training datasets and training code. All LLMs should be omni-modality. We are looking forward to hearing your thoughts and feedback! Blog: nvda.ws/4tCmhFS Hugging Face: nvda.ws/4u79ue9 Tech Report: nvda.ws/4tDzHBq #AI #MultimodalAI #OpenSource #ReinforcementLearning
4
4
1
30
4,566
Today we released Nemotron-3-Nano-Omni-30B-A3B - our first Omni model, with speech and audio understanding capabilities powered by parakeet-tdt-0.6b-v2 encoder. 🫡1st position on VoiceBench 🌏English only 🎙️5.95% WER on Open ASR Leaderboard 📽️Video+audio understanding
18
47
5
497
28,987
Andrew Tao retweeted
Replying to @ctnzr
This looks incredible! Can’t wait to see it in action on my new Alienware Area 51 👽 😎
13
1
1
30
4,778
Andrew Tao retweeted
So proud of DLSS5: Fully generative neural rendering, in real-time, in real games. Mind-blowing realism. A whole new generation of real-time graphics. A decade of continuous research and development. Coming soon to PCs everywhere. 💚
3,820
211
1,059
3,668
1,966,299