Lizhou “Leo” Fan (@LeegeoF) header
Lizhou “Leo” Fan (@LeegeoF) profile picture

Vice-Chancellor Assistant Professor @CUHKofficial | Previouly Postdoc @harvardmed, PhD @umsi, BS @uclastat | Medical AI, AI Agents, Trustworthy AI, Psychiatry

Joined April 2019
What’s is the agent? What is the optimal behavior to achieve the predefined goal? And how to learn that behavior policy? We formally introduce a systematic Theory of Agent (ToA), analogous to the cognitive framework of Theory of Mind (ToM). Where ToM refers to the ability to attribute mental states (e.g., beliefs, intentions, knowledge) to oneself and others, enabling the prediction and interpretation of behavior, ToA characterizes an agent’s capacity to model not only external environments but also its own internal knowledge state to make decisions and complete the goal. We provide new definition, three specific principles to guide the optimal behavior of agent where the keys lies in the alignment between knowledge boundary with the decision boundary. More importantly, we provide an actionable roadmap to agent foundation model. Almost everything you need for the agent can be found in the paper: arxiv.org/pdf/2506.00886
1
32
3
128
18,976
Lizhou “Leo” Fan retweeted
The GAIA game is over, and Alita is the final answer. Alita takes the top spot in GAIA, outperforming OpenAI Deep Research and Manus. Many general-purpose agents rely heavily on large-scale, manually predefined tools and workflows. However, we believe that for general AI assistants: "Simplicity is the ultimate sophistication." 🔗Full paper: arxiv.org/abs/2505.20286 🔗More Details will be updated here: github.com/CharlesQ9/Alita #AI #Agent #LLM
17
31
11
95
26,536
Lizhou “Leo” Fan retweeted
LLM Engineer Toolkit: A curated list of 120+ LLM libraries for training, fine-tuning, building, evaluating, deploying, RAG, and AI agents! 100% Open Source
42
477
13
2,510
229,669
Lizhou “Leo” Fan retweeted
🌟🎲🎲How to create a rational LLM-based agent? using game-theoretic workflow! Game-theoretic LLM: Agent Workflow for Negotiation Games 😊 paper link: arxiv.org/abs/2411.05990 github link: github.com/Wenyueh/game_theo… 😼 This paper aims at observing and enhancing the performance of agents in interactions guided by self-interest maximization 😼 😼 We chose game theory as the foundation, with rationality and Pareto optimality as the two basic evaluation metrics: whether an individual is rational and whether a globally optimal solution is developed based on individual rationality. ❣️ Complete information games They are classic games such as Prisoner's Dilemma. We selected 5 simultaneous games and 5 sequential games. We found that, except for o1, other LLM generally lack a robust ability to compute Nash equilibria, meaning they are not very rational. They are not robust to noise, perturbations, or random talks among them. Therefore, based on classical game theory methods (Iterative Elimination of Dominated Strategy & Backward Induction), we designed two workflows to guide large models step-by-step in computing Nash equilibria during inference time. ❣️ Incomplete information games We used the classic "Deal or No Deal" resource allocation game with private valuation, where agents do not know the opponent's valuation of resources. Game theory does not provide a solution for this, and previous work has been based on reinforcement learning. 👉 Sonnet and o1 perform better than humans in terms of negotiation success rate and results 👉 Opus and 4o are far behind. 👉 We designed an algorithmic workflow based on the rational actor assumption, allowing agents to infer the opponent's valuation based on their reactions to various resource allocation schemes. The workflow is very effective, reducing the possible estimated valuations from an initial 1000 possibilities to 2-3 within 5 rounds of dialogue, and always including the opponent's true valuation. 🌟🌟Based on the estimated valuation of opponent's resource, we guide the agents in each step to calculate and propose an allocation proposal that maximizes their own interests while having a non-zero probability of being envy-free, ensuring that both parties are relatively satisfied and the negotiation can proceed. 🌟🌟 But very interestingly, we found that if only one agent uses this workflow during negotiation, it will be exploited. Although the workflow improves the overall negotiation outcome and brings more benefits to the individual agent, the benefits will always be less than the opponent's. 🔥In the future, we will need a meta-strategy to choose which workflows to use!
4
49
7
199
26,738
Lizhou “Leo” Fan retweeted
🚀 Exciting News for AI4Health! 🌐 We’re thrilled to release WorldMedQA-V, a multilingual, multimodal medical examination dataset designed to benchmark vision-language models in healthcare! 🩺💻 👉 Check it out: huggingface.co/datasets/Worl… 🧵👇 #AI #HealthcareAI
1
7
3
20
4,812
Lizhou “Leo” Fan retweeted
Really nice reference for work in the LLM/AI and health space by @YuHuizi and @LeegeoF et al. Relevant to what we've been planning @DrJoDaniels link.springer.com/article/10…
2
1
275
Lizhou “Leo” Fan retweeted
Google presents On scalable oversight with weak LLMs judging strong LLMs arxiv.org/abs/2407.04622
6
63
3
398
34,035
Lizhou “Leo” Fan retweeted
Why did language evolve among humans? To facilitate thinking? Or to facilitate communication? This new perspective article just published in @Nature suggests that language evolved as a tool for communication. The main evidence against the language-for-thought hypothesis comes from patients with brain damage and disorders. There are people with major linguistic impairments but virtually intact cognitive abilities. Additional evidence comes from fMRI studies suggesting that the language network is virtually disjoint from networks involved in thinking. The main evidence supporting the language-for-communication hypothesis comes from research highlighting that the structure of language seems to be optimized for ease of production, learning, and robustness to noise. For example, the tendency to minimise dependency length, the preference for certain word orders (e.g., subject-object-verb and subject-verb-object), and the presence of ambiguity, seem to be useful for decreasing the cognitive cost of communicating. Full paper: nature.com/articles/s41586-0…
54
188
39
908
160,606
Lizhou “Leo” Fan retweeted
Is that what we call Bingo? 🎯 "Samba = Mamba + MLP + Sliding Window Attention + MLP stacking at the layer level." => infinite context length with linear complexity Samba-3.8B-instruct outperforms Phi-3-mini across all benchmarks using the same dataset (trained on 3.2 trillion tokens from the Phi3 dataset)
1
21
119
10,026
Lizhou “Leo” Fan retweeted
Congratulations, Dr. Fan! 🎉 UMSI’s Lizhou “Leo” Fan successfully defended his dissertation, “Generative AI-augmented and User-centric Research Data Discovery and Reuse.” He’ll soon join @BrighamWomens as a postdoctoral research fellow 👏 @LeegeoF #PhDone
1
1
1
2
1,437
Lizhou “Leo” Fan retweeted
New research from UMSI: A dataset for measuring the impact of research data and their curation Libby Hemphill, Andrea Thomer, Sara Lafia, Lizhou Fan, David Bleckley, Elizabeth Moss @libbyh @lafia_s @LeegeoF @ScientificData nature.com/articles/s41597-0…
1
1
179
Lizhou “Leo” Fan retweeted
😳 How much is VLM's reasoning ability lagging behind LLM's reasoning ability? 🔖 We construct NPHardEval4V 🔥, the visual counterpart of NPHardEval ▶️▶️▶️ After removing the effect of recognition failure and instruction-following failure, VLMs are much worse on reasoning
1
3
4
1,150
Thank you @huggingface and @clefourrier for featuring our NPHardEval benchmark and leaderboard. Started later Jan, we update the test data monthly and will release updated evaluation (possibly with more models) the next Month. Have your favorite LLM in mind? Let us know!
1
3
1,021
Lizhou “Leo” Fan retweeted
NPHardEval benchmark updated version! 1. Phi-2 has a good performance (Huggingface leaderboard here: huggingface.co/spaces/hyfran…) 2. Robustness experiment: Our benchmark is actually robust under finetuning hack! See details in the paper arxiv.org/abs/2312.14890
4
2
7
6,282
Lizhou “Leo” Fan retweeted
Under NPHardEval benchmark, GPT-4 is still far beyong all other models. Open source models are going close after GPT-3.5 and Claude: Qwen, Yi, Mistral and Phi-2 (even it is only 2.7b! Textbook is indeed all we need haha) show good performance.
3
3
11
2,606
Lizhou “Leo” Fan retweeted
What is LLM's ability in solving P, NP, NPC, and NP-Hard problems? Check out @LeegeoF @HuaWenyue31539 etc.'s paper on evaluating LLM's reasoning ability via complexity classes.
Can LLM solve NP-hard problem? We proposed a new benchmark to rigorously evaluate LLMs' reasoning ability: NPHardEval: 1. Built based computational complexity hierarchy 2. Automatic datapoint generation 3. Automatic result checking 4. Monthly refresh -- no overfitting!
1
7
1,466
Lizhou “Leo” Fan retweeted
Can LLM solve NP-hard problem? We proposed a new benchmark to rigorously evaluate LLMs' reasoning ability: NPHardEval: 1. Built based computational complexity hierarchy 2. Automatic datapoint generation 3. Automatic result checking 4. Monthly refresh -- no overfitting!
3
5
2
5
2,189
Preview (2/2). I was honored to work with @ToddPresner at @UCLA_DH where I grew from a UG student to a data scientist. We used semantic triplets (RDF triple) for "Algorithmic Close Reading". digitalhumanities.org/dhq/vo… I'll carry experiences from both projects and explore more!
1
And please feel free to give us any suggestions, as both articles are previews for now. I these both works can converge if we think from the perspective of the human knowledge network. Maybe that's gonna be one next research topic for me.
I was lucky to collaborate with Prof. Anne Gilliland, especially during the pandemic. Her advice helped me through many difficulties and her suggestions are always helpful for my archival science research. Although she's not on Twitter much, I want to shout out to her!