Darshan Deshpande (@getdarshan) profile picture

Research Scientist working on RL environments and evals @PatronusAI | ex-Research @USC_ISI

San Francisco, CA
Joined July 2020
I'm keeping my promise today. Introducing 𝙏𝙖𝙣𝙜𝙚𝙣𝙩𝙨, a live and growing list of 10/N research questions that keep me up at night. The ideas are handwritten (and I will keep them as such because it grounds me). Interested in collaborating? Reach out and we can make things happen :) darshandeshpande.github.io/t… This batch focuses on multi-agent swarms and high level alignment and what it means. The next one will be focused more on interesting ablations, model architectures and domain specific ideas! Thanks to @mariyaivasileva and @VarunGangal for being my sounding board for some of these and supporting me :)
While I get to learn a lot from Mariya on a daily basis, this is a piece that I resonate with a lot. I have experienced a similar existential reflection on the culture of AI research where people seem to have lost the curiosity that drives novel research. There were times when I pursued the randomest ideas - why would you think about neural nets with complex tensors and their optimizations? who thinks about alternating FFT patterns for steganography? I miss the times when I could pursue topics like these solely from the POV of curiosity. While I say this, I also feel like I am part of the industry and part of the reason why we are in this situation - maybe I as a scientist should step up and build the culture around me. In this attempt, I want to step up for the community and extend an open offer to collaborate if you are working on something that is out of the box and you are really passionate about. I have said this multiple times and I will say this again - I am not scared of my ideas being scooped - I am scared of a world where everyone's ideas are so linear that scooping is a worry for researchers. In this attempt, I will be starting a section on my personal website for curious thinkers where I freely share some of my out of the box ideas for people to use, explore and build on - just to feel the same passion and curiosity within the community again.
1
1
15
1,528
Testing a model’s understanding of its environment and its ability to simulate environment states gives us an excellent proxy to measure its intelligence and performance in that environment. Humans maintain internal models of the world around us but we find that LLMs are poor at understanding and accurately simulating consequences of actions they take. Globebench is one of the first evaluation benchmarks that moves us in the direction of measurably testing emergent behaviors when agents try to simulate the worlds around them. Kudos to the whole team and it was an absolute pleasure to be part of this project - @nick_saban20 @BannurCharvi @mariyaivasileva! 🚀
Today we're open-sourcing GlobeBench, 1,632 tasks that test whether LLMs can simulate RL environments well enough to train agents on. The top model, GPT-6 Astra, gets 49.6% right. No other model of the 20 we tested reaches 30%. 🧵
1
10
283
Darshan Deshpande retweeted
Prompting LLMs to behave like people can improve simulations, but how should we design this? What info is needed, and how much? With @Kehang_Zhu @locross @bakkermichiel, we build “behavioral profiles” that help LLMs simulate human behavior under distribution shifts. 1/8 🧵
7
24
2
144
13,751
While I get to learn a lot from Mariya on a daily basis, this is a piece that I resonate with a lot. I have experienced a similar existential reflection on the culture of AI research where people seem to have lost the curiosity that drives novel research. There were times when I pursued the randomest ideas - why would you think about neural nets with complex tensors and their optimizations? who thinks about alternating FFT patterns for steganography? I miss the times when I could pursue topics like these solely from the POV of curiosity. While I say this, I also feel like I am part of the industry and part of the reason why we are in this situation - maybe I as a scientist should step up and build the culture around me. In this attempt, I want to step up for the community and extend an open offer to collaborate if you are working on something that is out of the box and you are really passionate about. I have said this multiple times and I will say this again - I am not scared of my ideas being scooped - I am scared of a world where everyone's ideas are so linear that scooping is a worry for researchers. In this attempt, I will be starting a section on my personal website for curious thinkers where I freely share some of my out of the box ideas for people to use, explore and build on - just to feel the same passion and curiosity within the community again.
Sharing some deeply personal and existential reflections on scientific identity, the changing culture of AI research, the erosion of intellectual autonomy, and the growing disconnect between scientific rigor, professional recognition, and what organizations actually reward. As usual, feedback and debate are welcome and actively encouraged! mariya.fyi/posts/stranger-in…
1
1
13
1,511
Average SF experience: * Attends frontier research event presentation * Asks a fundamental research question about their work * Receives a "We can't talk about this" 🤡
5
138
Darshan Deshpande retweeted
Today, we're open sourcing the first computer-use dataset for design. We captured 200+ hours and 3400+ trajectories of real design work in Figma, recording step by step how designers execute complex, long-horizon tasks. FigmaTrace dataset, paper, and blog below :) @PatronusAI
41
73
19
733
137,986
Regardless of how much I support this, I don't believe that the @arxiv moderation team has enough capacity or incentive to evaluate these AI flagged papers quickly. My recent non-AI-generated submission has been "on-hold" and is awaiting moderator response for over 2 weeks now!
Attention @arxiv authors: Our Code of Conduct states that by signing your name as an author of a paper, each author takes full responsibility for all its contents, irrespective of how the contents were generated. 1/
1
240
Multi-turn research benchmarks are hard to build because trajectories and tool usage are non-deterministic. We make this process more deterministic by introducing a memory-grounded benchmark of tip-of-tongue, multi-turn, multimodal queries for evaluating VLMs. Check out DETOUR!
Spotlighting our newest benchmark for agentic search: DETOUR When people try to recall something in conversation, they rarely give a perfect query upfront. They say things like “that movie with the scene where…” or “the paper about…” and the assistant has to ask the right follow-up questions to get there. Existing search and agent benchmarks often miss this multi-turn, tip-of-the-tongue behavior. To more realistically evaluate it, we introduce DETOUR: Dual-agent based Evaluation Through Obscure Under-specified Retrieval, an interactive benchmark for dual-agent search and reasoning. DETOUR contains 1,011 prompts across text, image, audio, and video. In the benchmark, a Primary Agent is evaluated on its ability to identify a target entity by querying a consistent Memory Agent, testing whether models can resolve ambiguity through useful follow-up questions. Current state-of-the-art models still struggle: performance reaches only 36% accuracy across all modalities, showing that today’s agents remain weak at clarification-seeking in underspecified, real-world search settings. We hope DETOUR helps push the next generation of search agents toward better reasoning, better questions, and more robust multi-turn retrieval. arXiv Paper: arxiv.org/abs/2602.00352
There's a new version of this post
1
2
370
TRACE made it into ICML 2026 Main Track! Hopefully inspiring more research in this crucial space🥳
RL coding agents increasingly game rewards by exploiting their semantic and syntactic weaknesses. Can LLMs detect such behaviors from live training rollouts? We find contrastive cluster analysis is key! 🚀 GPT-5.2 jumps from 45% to 63%. Humans reach 90% Paper + data 🧵
1
1
1
7
1,043
RL coding agents increasingly game rewards by exploiting their semantic and syntactic weaknesses. Can LLMs detect such behaviors from live training rollouts? We find contrastive cluster analysis is key! 🚀 GPT-5.2 jumps from 45% to 63%. Humans reach 90% Paper + data 🧵
1
3
1
8
1,747
Where do models fail? 🤔 - Semantic reward hacks are harder to detect than syntactic hacks! - Models consistently show similar failures QA reveals: ✅ Grounding and exploring consequences helps ❌ Over-reliance on user acceptance or self awareness patterns impact performance
1
3
148
Darshan Deshpande retweeted
👋 Folks at #NEURIPS2025, come check out & stop by the poster of our Memtrack env at the SEA workshop happening at Upper Level 23ABC, 3:50pm onwards. Our env studies how well an agent dropped into a workplace can context engineer by composing tool calls to access intertwined slack, linear & git timelines in pursuit of answering a battery of related questions. Full paper arxiv: arxiv.org/abs/2510.01353
🚨We will be presenting Memtrack today at the SEA workshop from 3:50pm onwards at #NeurIPS2025 Memtrack is a SoTA eval env to study an agent's ability to memorize and retrieve facts using exploration over interleaved enterprise slack, linear and git threads in a multi-QA setting
3
8
918
🚨We will be presenting Memtrack today at the SEA workshop from 3:50pm onwards at #NeurIPS2025 Memtrack is a SoTA eval env to study an agent's ability to memorize and retrieve facts using exploration over interleaved enterprise slack, linear and git threads in a multi-QA setting
4
3
13
2,651