Speech + LLMs @nvidia | Previously: @Meta MSL, @jhuclsp, @IITGuwahati

New York, NY
Joined September 2009
I’m happy to share that I’m starting a new position as Senior Research Scientist at @nvidia! Looking forward to open science for speech full-duplex models :)
After 2 wonderful years, I left Meta this week. During this time, I worked on several projects related to speech and LLMs: - Built the first multi-channel audio foundation model with M-BEST-RQ (arxiv.org/abs/2409.11494) - Made ASR with SpeechLLMs faster (arxiv.org/abs/2409.08148) and more accurate (ieeexplore.ieee.org/document…) - Shipped the first production-ready full-duplex voice assistant (about.fb.com/news/2025/04/in…) - Improved Moshi’s reasoning capability with chain-of-thought (arxiv.org/abs/2510.07497) I am grateful to my managers for having my back on critical projects, and fortunate to have collaborated with several brilliant researchers and engineers during this time. As to what's next, I am still in NYC and continuing to do speech research. More on that later!
52
8
517
38,746
Saturday is for time with family.
Saturday is for fixing bugs while sipping coffee. What are some great places to hangout in SF?
45
61
13
2,531
167,699
A lot is being made of the fact that Dessa was trained only on human conversations, without any text or pretrained models. First, dGSLM (arxiv.org/abs/2203.16502) already did this in 2022 with just 2k hours of Fisher data. Vendors now provide thousands of hours of high-quality 2-channel dialog off-the-shelf. Second, why do you even want to do this? Humans spent hundreds of thousands of years before inventing writing systems, and then optimized it over 5k years to store knowledge very efficiently. dGSLM had to train only on dialogs because at the time, we didn't know have unified models that could train jointly on text and speech, but this problem was solved by autoregressive transformers. You can cut your training ~10x by letting the model learn from text first, then spoken dialog, like Moshi (and basically every other modern S2S model). Different != smart. Making very natural chit-chat bots is not where the frontier lies. Demos are cool, but the real challenge is intelligence, agentic behavior, and delegation. I would have been blown away if the naturalness was accompanied by GPT-Live style function calling, but alas!
14
9
4
164
7,719
When we unveiled Moshi two years ago, we also showed an early checkpoint we'd trained just for fun, directly on Fisher (phone calls from the late 90s and early 2000s). As a result, every time you called, you'd get to chat with a random person straight out of that era. That was trained on just 2k hours of data. Today, you can get 100s of thousands of hours of full-duplex conversational data. There's an obvious scaling law at play here!
We believe voice is the most natural way to interact with intelligence, so we created Dessa, the most authentic full-duplex speech model ever built. Talking with it feels like talking with a person…full of emotion, nuance, pauses, and a natural cadence. block.xyz/dessa
14
19
202
12,698
Do people not know that Moshi has been around for over 2 years?
We believe voice is the most natural way to interact with intelligence, so we created Dessa, the most authentic full-duplex speech model ever built. Talking with it feels like talking with a person…full of emotion, nuance, pauses, and a natural cadence. block.xyz/dessa
4
4
1
85
9,252
Desh Raj retweeted
idk I try to help youngsters high schoolers etc in my DMs if I can be helpful this is the reason why I don't like SF, feels like it's full of social hierarchies where numbers/markers matter not as much in Europe
people need to status max so much more high status people saying "chasing status is bad" is such an effective way of keeping everyone below them literally the only reason why you care about a person you have not met is because they've done something publicly approved rant:
7
7
140
9,177
Desh Raj retweeted
Meet NVIDIA Nemotron 3.5 Super VL Our new open model combines reasoning and tool use with text, image, document, and video understanding for long-running AI agents. 120B total parameters with 12B active 🧵
🤖 Made with AI
289
553
365
4,455
601,568
I love full-duplex models and GPT-Live is awesome, but this claim is dubious at best. From Josh's post, most of the models hold up under "street noise" and "cafe chatter", but fail under "background TV", despite all of them being added at low energy levels. The only conclusion that we can draw from this is that most models are trained with noise augmentation resembling ambient noises or on in-the-wild data where it is unlikely to find TV noise. On the TV noise scenario itself, GPT-Live degraded by ~39% while Grok degraded only by 15%, despite not being a full-duplex model. From xAI's blog post: "We've focused on making Grok Voice Think Fast 2.0 perform exceptionally well in real-world settings, with substantial background noise and telephony compression." In short, they trained with a lot of real-world noise augmentation. Speech folks have worked on noise robustness and target-speaker recognition for a decade now, and came up with several cool architectural tricks. But if you look at ASR recipes from Kaldi/ESPNet, the only constant is noise augmentation and far-field simulation.
AFAIK gpt-live-1 is the only model here where turn detection is built into the audio model itself rather than separate, which is why it’s so much better perceiving turns in noisy environments. Enabled by the full duplex architecture
7
5
1
59
3,900
How to kill a voice agent? Just turn on the TV ☠️ We benchmarked 14 voice agents with a TV playing in the background. Task completion fell from 38.7% to 8.6% 🧵/
29
19
13
192
33,052
Desh Raj retweeted
Introducing ChatGPT for Dishwashing GPT-6 achieves state-of-the-art results on dishwashing benchmarks for long-running jobs and difficult stains. The Navier–Stokes Millennium Prize Problem is a $1 million challenge about whether the equations describing fluid motion can break down. In this video, I accidentally found a counter-example while trying to optimize my dishwasher. But in reality, OpenAI used 10,000 AI agents to find a proof in 88 hours: openai.com/index/navier-stok… Parody of the original GPT-6 Astra ad @OpenAI #ChatGPT_Partner #chatgpt Credits: Written and Directed by Joma Co-Writer - Henry Connor Coan IV @coan_henry Production by Dauntlus Studios @dauntlus.studios Producer/Co-Director - Marcus Liew @dauntlus Camera Operator - Amos Lee @amoselijahlee Art Director - Alethea Soo @aletheasoo Art Assistant - Alicia Lim @aliciatkl_ Assistant Editor - GPT-6 Astra Music: Tensions Run High by Soundridemusic youtube.com/watch?v=Ly9H63SL…
423
960
341
10,788
1,127,566
love how in 90s and 2000s romcoms, nyc was never just a backdrop but a character of its own
104
5,444
297
46,135
1,354,280
this was not in my bingo card for 2026
PSA: @mervenoyann is destined to teach @pewdiepie about local agents and post-training. upvote her plea and make a dream come true. github.com/odysseus-dev/odys…
715
Desh Raj retweeted
We’re looking for our next cohort of graduate fellows. Our 2027–2028 Graduate Fellowship Program is open to Ph.D. students worldwide with awards up to $60,000, plus mentorship and technical support. Apply by October 30: nvda.ws/4rGKDhv
🤖 Made with AI
10
108
8
595
116,747
Sigh. I wrote this post below, but then I didn't post it because I am tired of the consistent pushback I get from folks with the same safety worldview as Anthropic. I guess that means I should send it. Here goes! This post is pretty solipsistic and doesn't properly take recent events in cyber risks into it's discussion. It is really hard for me to watch how the US frontier labs like Anthropic don't have the ability to consider other approaches to safety and ways things could play out. It insinuates that Z ai (Chinese lab, builds GLM series) doesn't really care about safety and is reckless to take their business strategy. Saying things like "Given this evidence, we think it's likely both state and non-state actors will use models like GLM-5.3 to cause real-world harm." and "This is unlike any other similarly capable AI model, all of which were released with safeguards or through limited access programs." while closed models have been used on more of the documented cyber attacks is just bowing out of the interesting question. A plausible view is that open model weights and closed model apis (with some safe guards) are both far closer to being easy to mis-use, rather than API models being closer to safe. The trope "Open Dangerous, Closed Safe" may be closer to "Open Unsafe, Closed Unsafe" Closed models have stronger capabilities and stronger safeguards, but the stronger capabilities part could matter more in net harm if both the safeguards are porous. At the same time, as the authors do acknowledge (thanks - thats progress!), open models without extreme cyber guardrails are important to rapidly diffuse cyber readiness in the economy -- as programs like project glasswing are not perfect in getting all critical industry onboarded. There will be more issues like the time when Fable was released and Amazon found a workaround, which allowed them to access the full capabilities of the model served readily at an API. The blog overall is reasonable in it's narrow line, but it's a very effective tool in a complicated, rapidly evolving media ecosystem to reinforce a certain type of safety thinking.
52
86
16
696
77,854
Who wants more speech training data? YODAS v3 is now available @huggingface It’s 1.1M hours - the biggest audio dataset ever. And the first at this scale with stereo audio at 48kHz, with timestamped transcripts and translations. 100+ langs hf.co/blog/espnet/yodasv3 CC-BY-3.0
23
73
13
456
49,448
The phrase "stochastic parrot" is full of sound and fury, signifying nothing. Which, ironically, is exactly what the authors got wrong about language, and the core technical mistake of the paper. 1/
Everyone! This is a straw-person argument. The Stochastic Parrot paper was about LLMs of 2021, not the AI of today, which are not LLMs but complex software systems with vast post training and many external software components.
21
42
9
368
91,341
Sydney, Australia 🐨🇦🇺 #Interspeech2026 We’re presenting 7 papers this year! I’m also looking for motivated PhD students to join my lab at JHU CLSP in Fall 2027. If you’re interested, feel free to come find me at our paper presentations. (Check Interspeech app for details)
1
5
28
1,160
This is a great post on how to evaluate fundamentally unmeasurable capabilities like voice steering.
We trained our new TTS model to follow free-form voice instructions, then needed a way to measure how well it followed them. I decided to share some interesting details of that project in my personal blog: altsoph.substack.com/p/wtf-i…
2
14
1,869
I don't have any accepted papers at NeurIPS, and not attending InterSpeech in Sydney either. But I advanced to French level B1 yesterday, and I'm happy to show you my new, big-boy textbooks 🚀
8
53
1,500
With all the corp-speak around model releases, sometimes it's easy to forget that there are real people behind them, spending long hours improving the models. Early in my PhD at JHU, I helped build the community baseline for the CHiME-6 challenge (organized by @shinjiw_at_cmu and others). For those who don't know, this is an extremely challenging dinner-party conversation data (far-field, tons of overlapped speech, ...), and our baseline clustering-based diarization system could only get ~51% error rate. Imagine our surprise when, during the challenge, we saw the STC team from Russia top the leaderboard by cutting the DER to ~32% (every other team was still hovering above 40%). It turned out that Ivan Medennikov had created a new end-to-end method called TS-VAD, which became the new state-of-the-art. After the challenge, I helped Ivan integrate TS-VAD into Kaldi, so that it could become part of the next year's baseline. Having worked a bit on diarization during my PhD, I was very familiar with Taejin Park. He was close to graduating from USC at the time, and had written a very interesting paper on auto-tuning spectral clustering. I had some correspondence with him about this and ended up extending this method for overlapping speaker diarization. Taejin's most cited paper, of course, is "A review of speaker diarization", which very recently won CSL's best review paper award. Behind the chart-topping Nemotron Diarization model are folks who have been pushing speaker diarization for many years. And that is why, despite not being part of this effort, it makes me so happy to see this team's success. In a world where new and better models are shipped every week, it's good to take a moment to appreciate the real people behind the science.
Replying to @PiotrZelasko
Shout out to our incredible diarization team: Ivan Medennikov, Taejin Park, Tatiana Timofeeva, He Huang, Sofia Kostandian, Weiqing Wang, Dongji Gao
3
33
2,362