@DrorRotem

Asst. Prof. of Natural Language Processing @UofHaifa

rtmdrr.github.io
Joined November 2018
  • Tweets206
  • Following249
  • Followers333
3Media
Rotem Dror retweeted
eaclmeeting@eaclmeetingMay 4
Attention #NLProc researchers, the EACL 2027 website is officially LIVE: 2027.eacl.org/! ๐ŸŽ‰ ๐Ÿ‡ฌ๐Ÿ‡ท Join us in Athens, Greece (Mar 9-13, 2027) at #EACL2027 ๐Ÿ“… ARR submission deadline: Aug 6, 2026. Open to all areas of CL/NLP + related fields. Stay tuned for the detailed CfP!
The Acropolis in Athens, Greece

ALT The Acropolis in Athens, Greece

3
29
2
142
17,764
Rotem Dror retweeted
Nitay Calderon
@NitCalFeb 24
[1/7] Why do frontier LLMs make factual errors? Is it because they never learned the factโ€ฆ or because they canโ€™t access knowledge they already encoded? In our new paper, we show: The bottleneck is not encoding; it is recall. ๐Ÿงต๐Ÿ‘‡ Paper: arxiv.org/abs/2602.14080 Many thanks to @_galyo @bd_eyal @zorikgekhman @eran_ofek59358 @GoogleResearch
4
32
3
125
13,374
Rotem Dror retweeted
Yanai Elazar@yanaiela4 Aug 2025
A strange trend I've noticed at #ACL2025 is that people are hesitant to reach out to papers/"academic products" authors. This is unfortunate for both parties! A simple email can save a lot of time to the sender, but is also one of my favorite kind of email as the receiver!
1
33
1,547
The Alternative Annotator Test (alt-test) is a new statistical procedure proposed in our ACL 2025 paper! ๐Ÿ‡ฆ๐Ÿ‡น๐Ÿ‡ฆ๐Ÿ‡น @DrorRotem @roireichart The goal? To help justify using LLMs over humans. If the LLM passes the test, its annotations can be trusted ๐Ÿ˜Ž arxiv.org/abs/2501.10970

The Alternative Annotator Test for LLM-as-a-Judge: How to...

The "LLM-as-an-annotator" and "LLM-as-a-judge" paradigms employ Large Language Models (LLMs) as annotators, judges, and evaluators in tasks traditionally performed by humans. LLM annotations are...

arxiv.org
1
1
9
455
Everyone uses LLMs to annotate data or evaluate models in their research. But how can we convince others (readers, collaborators, reviewers!!!) that LLMs are reliable? ๐Ÿค– Hereโ€™s a simple (and low-effort) solution: show the LLM is a *comparable alternative annotator* โœ…
3
20
1
68
6,020
๐Ÿ”ฅื”ืกืงื™ืจื•ืช ืžืžืฉื™ื›ื•ืช ืœื–ืจื•ื ืœ-X๐Ÿ”ฅ ๐Ÿงต ื”ืžืืžืจ ื”ื™ื•ืžื™ ืฉืœ ืžื™ื™ืง: 25.06.25 The Alternative Annotator Test for LLM-as-a-Judge: How to Statistically Justify Replacing Human Annotators with LLMs ืžืืžืจ ๐Ÿ‡ฎ๐Ÿ‡ฑ ืชืคื ื™ืช ืžืขื ื™ื™ื ืช ืžืชืจื—ืฉืช ื‘ืชืงื•ืคื” ื”ืื—ืจื•ื ื” ื‘ืขื•ืœื ืฉืœ ื”ืขืจื›ืช ื‘ื™ืฆื•ืขื™ ืžื•ื“ืœื™ื. ืื ื—ื ื• ื›ื‘ืจ ืœื ืฉื•ืืœื™ื ืจืง ืขื“ ื›ืžื” ื”ืžื•ื“ืœ ืžืฆืœื™ื— ื‘ืžื‘ื—ืŸ ื›ืœืฉื”ื•, ืืœื ืฉืืœื” ืžื”ื•ืชื™ืช ื™ื•ืชืจ: ื”ืื ื ื™ืชืŸ ืœืกืžื•ืš ืขืœ ืžื•ื“ืœ ืฉืคื” ืฉื™ื—ืœื™ืฃ ืžืชื™ื™ื’ ืื ื•ืฉื™? ื–ื• ืœื ืฉืืœื” ืฉืžื“ื“ื™ื ืžืกื•ืจืชื™ื™ื ื›ืžื• ื“ื™ื•ืง, F1 ืื• ื”ืกื›ืžื” ื‘ื™ืŸ ืžืชื™ื™ื’ื™ื ื™ื›ื•ืœื™ื ืœืขื ื•ืช ืขืœื™ื” ื›ืจืื•ื™. ืชื—ืช ื–ืืช, ื”ืžืืžืจ ืฉื ืกืงื•ืจ ื”ื™ื•ื ืžืฆื™ื’ ืฉื™ื˜ื” ืžื‘ื•ืกืกืช ืกื˜ื˜ื™ืกื˜ื™ืงื” ืœืคืชืจื•ืŸ ื‘ื™ืขื” ื–ื•. ื‘ืœื‘ ื”ืžืืžืจ ืขื•ืžื“ืช ืงืจื™ืื” ืœื”ืชืจื—ืง ืžืžื“ื“ื™ ื”ืชืืžื” ืฉื˜ื—ื™ื™ื, ื•ืœืขื‘ื•ืจ ืœื ื™ืžื•ืงื™ื ืžื‘ื•ืกืกื™ ื”ืฉืขืจื•ืช ืกื˜ื˜ื™ืกื˜ื™ื•ืช ื•ื ื™ืชื•ื— ืขืœื•ืช-ืชื•ืขืœืช.
3
7
2
23
4,072
Rotem Dror retweeted
Nitay Calderon
@NitCal4 Jun 2025
Preferences drive modern LLM research and development: from model alignment to evaluation. But how well do we understand them? Excited to share our new preprint: Multi-domain Explainability of Preferences arxiv.org/abs/2505.20088 @roireichart @LiatEinDor ๐Ÿงต๐Ÿ‘‡ 1/11
2
18
1
35
2,237
In our new preprint, @roireichart ,@DrorRotem and I propose a new statistical procedure: The Alternative Annotator Test (alt-test) The goal? To help researchers justify using LLMs over humansโ€”if the LLM passes, its annotations can be confidently trusted๐Ÿ˜Ž arxiv.org/abs/2501.10970

The Alternative Annotator Test for LLM-as-a-Judge: How to...

The "LLM-as-an-annotator" and "LLM-as-a-judge" paradigms employ Large Language Models (LLMs) as annotators, judges, and evaluators in tasks traditionally performed by humans. LLM annotations are...

arxiv.org
1
3
1
13
1,624
Check out our new pre-print on statistically sound methodology to verify the quality of LLM-as-a-judge annotation. @NitCal @DrorRotem
Nitay Calderon
@NitCal25 Jan 2025
Replying to @NitCal
In our new preprint, @roireichart ,@DrorRotem and I propose a new statistical procedure: The Alternative Annotator Test (alt-test) The goal? To help researchers justify using LLMs over humansโ€”if the LLM passes, its annotations can be confidently trusted๐Ÿ˜Ž arxiv.org/abs/2501.10970

The Alternative Annotator Test for LLM-as-a-Judge: How to...

The "LLM-as-an-annotator" and "LLM-as-a-judge" paradigms employ Large Language Models (LLMs) as annotators, judges, and evaluators in tasks traditionally performed by humans. LLM annotations are...

arxiv.org
2
8
555
Do you think LLMs could win a Nobel Prize one day? ๐Ÿค” Can NLP predict heroin addiction outcomes, uncover suicide risks, or simulate brain activity? ๐Ÿง  @lotemi_peled, @roireichart, and I wrote a blog post about NLP for Human-Centric Sciences ๐Ÿค–๐Ÿ‘ฉโ€๐Ÿ”ฌ ๐Ÿ‘‡๐Ÿ‘‡๐Ÿ‘‡ nitaytech.github.io/blog/202โ€ฆ
8
16
596
I am looking for NLP/CV datasets that: 1. include annotator IDs 2. have multiple annotators per item 3. feature interesting tasks (including generation) Any suggestions?
5
3
2
11
2,322
Excited to have presented our TACL paper about the sensitivity of LLMs to prompt paraphrases at #ACL2024! ๐Ÿ“Šโœจ @guymkaplan, @malk_dan, @DrorRotem, @HyadataLab, @GabiStanovsky direct.mit.edu/tacl/article/โ€ฆ
6
5
1
62
3,886
Submission deadline extended- August 23rd! Looking forward to reading your works.
Lotem Peled@lotemi_peled5 Aug 2024
Reminder - *Call for Papers* for The NLP4Science workshop, to be held in EMNLP 2023 in Miami, Florida ๐Ÿ’ช Looking forward to your submissions utilizing NLP for exploration of the human mind (in fields like psychology, psychiatry, neuroscience, and economic behavior) as well as
2
1
255
i wrote a few remarks on @emilymbender 's presidential address (her slides are linked from the text, for those who weren't there). let's discuss. gist.github.com/yoavg/f952b7โ€ฆ

acl-presedential-response.md

GitHub Gist: instantly share code, notes, and snippets.

gist.github.com
18
37
13
277
112,068
Attending #ACL2024? ๐ŸŽคDon't miss my presentation on our TACL paper about the sensitivity of LLMs to prompt paraphrases! ๐Ÿ—ฃ๏ธ Oral: Mon, 12/8, 12:15 PM (World A) ๐Ÿ“Š Poster: Tue, 13/8, 16:00 PM Details below โฌ‡๏ธ w/ @guymkaplan, @malk_dan, @DrorRotem, @HyadataLab, @GabiStanovsky
Moran Mizrahi@moranmiz1 Feb 2024
๐Ÿš€ Excited to share our latest paper about the sensitivity of LLMs to prompts! arxiv.org/abs/2401.00595 Our work may partly explain why some models seem less accurate than their formal evaluation may suggest. ๐Ÿง @guymkaplan, @malk_dan, @DrorRotem, @HyadataLab, @GabiStanovsky
5
20
933
Reminder - *Call for Papers* for The NLP4Science workshop, to be held in EMNLP 2023 in Miami, Florida ๐Ÿ’ช Looking forward to your submissions utilizing NLP for exploration of the human mind (in fields like psychology, psychiatry, neuroscience, and economic behavior) as well as
1
7
2
7
1,541
๐ŸŒŸ The NLP4Science workshop, to be held in EMNLP 2023 in Miami, Florida, is now accepting papers! If your work leverages NLP tools to explore the human mind in fields like psychology, psychiatry, neuroscience, and economic behavior, we can't wait to read it.
1
7
1
9
797
๐ŸŒŸ Exciting News! Announcing the first-ever workshop on Natural Language Processing for Science at EMNLP 2024 in Miami, Florida! ๐ŸŒŸ
2
7
17
1,073
I was deeply offended by a slide in a recent talk at #CVPR2024 that falsely accused my country of genocide. Such baseless political statements have no place in our scientific community. Let's keep our focus on advancing science and leave politics at the door. @CVPR
300
97
159
1,457
562,089
Replying to @moranmiz @guymkaplan @malk_dan @DrorRotem @HyadataLab @GabiStanovsky
Thanks for making your data available, Moran and team! We have built from your work :) Now, we can compute performance summary statistics across prompts efficiently xitter.cf/felipemaiapolo/status/โ€ฆ
Felipe Maia Polo@felipemaiapolo13 Jun 2024
PromptEval: multi-prompt evaluation made practical! PromptEval efficiently estimates the performance distribution of LLMs over a big set of prompts (>100) using less than 5% of the compute๐Ÿฆพ Paper: arxiv.org/abs/2405.17202 GitHub: github.com/felipemaiapolo/prโ€ฆ Thread below๐Ÿงต1/8
5
9
302
Load more