@DanRothNLP

Chief AI Scientist, Oracle, and the Eduardo D. Glandt Distinguished Professor, CIS, University of Pennsylvania. Former VP/Distinguished Scientist, AWS AI Labs.

 Philadelphia, PA
seas.upenn.edu/~danroth/
Joined May 2010
  • Tweets46
  • Following55
  • Followers1.9K
Social Impact Award: "AccessEval: Benchmarking Disability Bias in Large Language Models" by Srikant Panda, Amit Agarwal, and Hitesh Laxmichand Patel aclanthology.org/2025.emnlp-… 10/n

AccessEval: Benchmarking Disability Bias in Large Language Models

Srikant Panda, Amit Agarwal, Hitesh Laxmichand Patel. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025.

aclanthology.org
1
4
13
3,956
📷 New #EMNLP2025 Findings survey paper! “Conflicts in Texts: Data, Implications, and Challenges” Paper: aclanthology.org/2025.findin… Conflicts are everywhere in NLP — news articles reflecting different perspectives or opposing views, annotators who disagree, LLMs that hallucinate or contradict themselves, and personal/enterprise document collections that grow apart and are conflicting. Most research tackles these in isolation, and our survey provides the first unified view of conflicting information in NLP. We chart the path toward conflict-aware, reliable NLP systems. Builds on our earlier work on: - Multi-perspective dataset aclanthology.org/2021.naacl-… and search aclanthology.org/2022.findin… - Hallucination detection aclanthology.org/2025.findin… - Open-domain QA with conflicting contexts aclanthology.org/2025.findin…
4
12
915
✨Yesterday we released MoNaCo, an @allen_ai benchmark of 1,315 hard human-written questions that, on average, require 43.3 documents per question!✨ The three aforementioned questions were actually some of the easier ones in MoNaCo 😉 (8/) xitter.cf/allen_ai/status/195747…
Ai2
@allen_ai18 Aug 2025
LLMs power research, decision‑making, and exploration—but most benchmarks don’t test how well they stitch together evidence across dozens (or hundreds) of sources. Meet MoNaCo, our new eval for question-answering cross‑source reasoning. 👇
1
1
3
655
Dan Roth retweeted
Ai2
@allen_ai18 Aug 2025
MoNaCo evaluates complex question-answering with: 📚 1,315 multi‑step queries 🔎 Retrieval, filtering & aggregation across text and tables 🌟 Avg 43.3 distinct documents per query
1
1
15
1,294
Dan Roth retweeted
Ai2
@allen_ai18 Aug 2025
LLMs power research, decision‑making, and exploration—but most benchmarks don’t test how well they stitch together evidence across dozens (or hundreds) of sources. Meet MoNaCo, our new eval for question-answering cross‑source reasoning. 👇
10
36
4
223
21,977
Augmenting GPT-4o with Visual Sketchpad ✏️ We introduce Sketchpad agent, a framework that equips multimodal LLMs with a visual canvas and drawing tools 🎨 . Improving GPT-4o's performance in vision and math tasks 📈 🔗: visualsketchpad.github.io
This quoted post is unavailable.
Try viewing anyway · View on X
10
52
4
281
50,254
😺 This work is done with my amazing collaborators: @yujielu_10, muyu he, @WilliamWangNLP @DanRothNLP YOU ARE THE BEST!!! 😎🔥 (n/n)
3
1
7
1,268
🔥Error Examples from DALL-E 3 👀More Visualizations: zeyofu.github.io/Commonsense… (3/n)
1
1
11
1,439
🔥Highlights of the Commonsense-T2I benchmark: 📚Pairwise text prompts with minimum token change ⚙️Rigorous automatic evaluation with descriptions for expected outputs ❗️Even DALL-E 3 only achieves below 50% accuracy (2/n)
1
2
10
1,633
Can Text-to-Image models understand common sense? 🤔 Can they generate images that fit everyday common sense? 🤔 tldr; NO, they are far less intelligent than us 💁🏻‍♀️ Introducing Commonsense-T2I 💡 zeyofu.github.io/Commonsense…, a novel evaluation and benchmark designed to measure commonsense reasoning in T2I models 🔥🔥 Paper: arxiv.org/abs/2406.07546 (1/n)
7
39
7
129
48,913
Best-fit Packing completely eliminates unnecessary truncations while retaining the same training efficiency as concatenation with <0.01% overhead tested on popular pre-training datasets like @TIIuae's RefinedWeb and @BigCodeProject's Stack.🧵5/n
1
1
3
638
The common practice in LLM pre-training is to concat all docs then split into equal-length chunks. This is efficient but hurts data integrity: doc fragmentation leads to loss of info, and causes next-token prediction to be ungrounded, making model prone to hallucination.🧵2/n
1
2
4
1,162
🚀Introducing "Fewer Truncations Improve Language Modeling" at #ICML2024 We tackle a fundamental issue in LLM pre-training: docs are often broken into pieces. Such truncation hinders model from learning to compose logically coherent and factually grounded content. 👇🧵1/n
4
10
3
45
7,316
Can GPT-4V and Gemini-Pro perceive the world the way humans do? 🤔 Can they solve the vision tasks that humans can in the blink of an eye? 😉 tldr; NO, they are far worse than us 💁🏻‍♀️ Introducing BLINK👁 zeyofu.github.io/blink/, a novel benchmark that studies visual perception abilities NOT yet “emerged” in Multimodal LLMs 🔥🔥 Paper: arxiv.org/abs/2404.12390 (1/n)
AK
@_akhaliq19 Apr 2024
BLINK Multimodal Large Language Models Can See but Not Perceive We introduce Blink, a new benchmark for multimodal language models (LLMs) that focuses on core visual perception abilities not found in other evaluations. Most of the Blink tasks can be solved by humans
9
124
5
406
105,908
Super excited to announce that our "3rd Workshop on NLP for Medical Conversations" will be co-located with IJCNLP-AACL 2023!! Website and CFP: nlpmc-2023.github.io @aaclmeeting #AACL2023 #NLProc #NLP #AI #DigitalHealth #HealthTech #Healthcare
1
5
1
10
2,834
We are thrilled to announce our second workshop on natural language interfaces, held in conjunction with the prestigious IJCNL-AACL conference! In collaboration with researchers from AWS AI Labs, Google Research, Meta AI Research, and Microsoft Research, this workshop aims to
1
2
6
1,568
I’ve been working with @awscloud’s #Bedrock service for a couple of months now at @caylentinc, and I’d like to share some of what I’ve learned. 🧵
8
90
13
311
89,954
Just out from AWS AI: aws.amazon.com/blogs/machine…

Announcing New Tools for Building with Generative AI on AWS | Amazon Web Services

The seeds of a machine learning (ML) paradigm shift have existed for decades, but with the ready availability of scalable compute capacity, a massive proliferation of data, and the rapid advancement...

aws.amazon.com
5
1
23
3,258
aws.amazon.com/codewhisperer… is really neat. Helps you code faster, checks for security vulns, discloses licenses of code it drew from, and works great for AWS APIs. Boom! @awscloud putting ML to work for developers

Amazon Q Developer

Accelerate software development with Amazon Q Developer — an AI coding assistant that writes, debugs, and refactors code natively in your IDE.

aws.amazon.com
1
3
5
Excited to announce a new product from AWS AI: Amazon CodeWhisperer aws.amazon.com/blogs/aws/now…

Now in Preview – Amazon CodeWhisperer- ML-Powered Coding Companion | Amazon Web Services

As I was getting ready to write this post I spent some time thinking about some of the coding tools that I have used over the course of my career. This includes the line-oriented editor that was an...

aws.amazon.com
1
10
Load more