So, what can help recall? Thinking helps LLMs access the knowledge they already store. It recovers 40–65% of encoded-but-not-directly-known facts, with the largest gains for rare facts and reverse questions. This resembles the human “tip-of-the-tongue” effect. Bottom line: Given that encoding in frontier LLMs is nearing saturation while substantial headroom remains for recall, future improvements are likely to come not from scaling but from better utilization of existing knowledge. Paper: arxiv.org/abs/2602.14080

[6/7] The reversal curse is when LLMs know "A is B" but can't answer "What is B?" If bidirectional knowledge were missing, reverse questions would be hard in any format. But in multiple-choice, reverse questions are as easy as direct ones. The problem, again, is recall.

[5/7] Prior work established LLMs struggle with long-tail facts due to limited capacity. We suggest a complementary picture: Rare facts are encoded at high rates; however, recalling them is the challenge. Encoding Δ≈7 Recall Δ≈20 The long-tail problem is largely a recall problem.

[4/7] RESULTS 📊 For frontier LLMs (Gemini-3-Pro, GPT-5): ✅ 95–98% of facts are encoded ❌ 25–33% cannot be directly recalled Even with thinking, 11–12% remain inaccessible. Currently, the bottleneck has shifted from knowledge acquisition to utilization. So, why does recall fail? Recall is tied to how facts were acquired. During pre-training, facts are encountered in specific contexts, phrasings, and orderings. When queries diverge from these patterns, recall becomes harder. We focus on two classic "knowledge gaps": * Long-tail facts * Reversal curse

[3/7] To support knowledge profiling, we introduce WikiProfile, built via a fully automated pipeline that uses prompted LLMs grounded in web search. * 2,150 facts extracted from Wikipedia documents * 10 tasks per fact, measuring encoding and recall We evaluate 13 LLMs (4.5M responses)

[2/7] We propose Knowledge Profiling, where each fact is categorized as: * Not encoded * Encoded but not recalled * Directly recalled * Recalled only with "thinking" (CoT/reasoning) * Inferred without encoding Why does this matter? Encoding failures can be mitigated by scaling pre-training: increasing model size/capacity or expanding data coverage. Recall failures, OTOH, can also be mitigated by inference-time and post-training methods that help LLMs utilize what they already store.

[1/7] Why do frontier LLMs make factual errors? Is it because they never learned the fact… or because they can’t access knowledge they already encoded? In our new paper, we show: The bottleneck is not encoding; it is recall. 🧵👇 Paper: arxiv.org/abs/2602.14080 Many thanks to @_galyo @bd_eyal @zorikgekhman @eran_ofek59358 @GoogleResearch

Excited to be at #EMNLP2025 in Suzhou 🇨🇳! I’ll present three papers, and I'm happy to chat about any of these works! 🏅 "Multi-Domain Explainability of Preferences" - Oral, Interpretability 2, Nov 5, 17:30 (A104-105) w/ @roireichart Liat Ein-Dor 🧠 "Dementia Through Different Eyes: Explainable Modeling of Human and LLM Perceptions for Early Awareness" 14:00 (Hall C3) w/ @lotemi_peled @MayaZadok @hila_gonen ✅"Are LLMs Better than Reported? Detecting Label Errors and Mitigating Their Effect on Model Performance" - Poster, Session 7, Nov 7, 14:00 (Hall C3) w/ @omer6nahum

🥳🥳 Happy to share that we have three papers accepted to EMNLP 2025 🇨🇳 (2 main, 1 findings)! What makes this special is that all three belong to a new research line I began last year: LLM-as-a-judge/LLM-as-an-annotator 🤖🧑‍⚖️

🚨 TOMORROW 🚨 I’ll present my #ACL2025 paper: “The Alternative Annotator Test for LLM-as-a-Judge: How to Statistically Justify Replacing Human Annotators with LLMs” Resources & Evaluation 2 oral session 🕑 14:00 (2nd speaker), 29.7, Hall B Come join!🤩