
Vice-Chancellor Assistant Professor @CUHKofficial | Previouly Postdoc @harvardmed, PhD @umsi, BS @uclastat | Medical AI, AI Agents, Trustworthy AI, Psychiatry
Joined April 2019
- Tweets31
- Following583
- Followers121
- Likes131
Lizhou “Leo” Fan retweeted
What’s is the agent? What is the optimal behavior to achieve the predefined goal? And how to learn that behavior policy?
We formally introduce a systematic Theory of Agent (ToA), analogous to the cognitive framework of Theory of Mind (ToM). Where ToM refers to the ability to attribute mental states (e.g., beliefs, intentions, knowledge) to oneself and others, enabling the prediction and interpretation of behavior, ToA characterizes an agent’s capacity to model not only external environments but also its own internal knowledge state to make decisions and complete the goal.
We provide new definition, three specific principles to guide the optimal behavior of agent where the keys lies in the alignment between knowledge boundary with the decision boundary. More importantly, we provide an actionable roadmap to agent foundation model.
Almost everything you need for the agent can be found in the paper: arxiv.org/pdf/2506.00886
Lizhou “Leo” Fan retweeted
The GAIA game is over, and Alita is the final answer.
Alita takes the top spot in GAIA, outperforming OpenAI Deep Research and Manus.
Many general-purpose agents rely heavily on large-scale, manually predefined tools and workflows. However, we believe that for general AI assistants:
"Simplicity is the ultimate sophistication."
🔗Full paper: arxiv.org/abs/2505.20286
🔗More Details will be updated here: github.com/CharlesQ9/Alita
#AI #Agent #LLM
Lizhou “Leo” Fan retweeted
LLM Engineer Toolkit: A curated list of 120+ LLM libraries for training, fine-tuning, building, evaluating, deploying, RAG, and AI agents!
100% Open Source
Lizhou “Leo” Fan retweeted
🌟🎲🎲How to create a rational LLM-based agent? using game-theoretic workflow!
Game-theoretic LLM: Agent Workflow for Negotiation Games 😊
paper link: arxiv.org/abs/2411.05990
github link: github.com/Wenyueh/game_theo…
😼 This paper aims at observing and enhancing the performance of agents in interactions guided by self-interest maximization
😼 😼 We chose game theory as the foundation, with rationality and Pareto optimality as the two basic evaluation metrics: whether an individual is rational and whether a globally optimal solution is developed based on individual rationality.
❣️ Complete information games
They are classic games such as Prisoner's Dilemma. We selected 5 simultaneous games and 5 sequential games. We found that, except for o1, other LLM generally lack a robust ability to compute Nash equilibria, meaning they are not very rational. They are not robust to noise, perturbations, or random talks among them.
Therefore, based on classical game theory methods (Iterative Elimination of Dominated Strategy & Backward Induction), we designed two workflows to guide large models step-by-step in computing Nash equilibria during inference time.
❣️ Incomplete information games
We used the classic "Deal or No Deal" resource allocation game with private valuation, where agents do not know the opponent's valuation of resources. Game theory does not provide a solution for this, and previous work has been based on reinforcement learning.
👉 Sonnet and o1 perform better than humans in terms of negotiation success rate and results
👉 Opus and 4o are far behind.
👉 We designed an algorithmic workflow based on the rational actor assumption, allowing agents to infer the opponent's valuation based on their reactions to various resource allocation schemes.
The workflow is very effective, reducing the possible estimated valuations from an initial 1000 possibilities to 2-3 within 5 rounds of dialogue, and always including the opponent's true valuation.
🌟🌟Based on the estimated valuation of opponent's resource, we guide the agents in each step to calculate and propose an allocation proposal that maximizes their own interests while having a non-zero probability of being envy-free, ensuring that both parties are relatively satisfied and the negotiation can proceed.
🌟🌟 But very interestingly, we found that if only one agent uses this workflow during negotiation, it will be exploited. Although the workflow improves the overall negotiation outcome and brings more benefits to the individual agent, the benefits will always be less than the opponent's.
🔥In the future, we will need a meta-strategy to choose which workflows to use!
Lizhou “Leo” Fan retweeted
🚀 Exciting News for AI4Health! 🌐
We’re thrilled to release WorldMedQA-V, a multilingual, multimodal medical examination dataset designed to benchmark vision-language models in healthcare! 🩺💻
👉 Check it out: huggingface.co/datasets/Worl…
🧵👇 #AI #HealthcareAI
Lizhou “Leo” Fan retweeted
Really nice reference for work in the LLM/AI and health space by @YuHuizi and @LeegeoF et al. Relevant to what we've been planning @DrJoDaniels
link.springer.com/article/10…
Lizhou “Leo” Fan retweeted
Google presents On scalable oversight with weak LLMs judging strong LLMs
arxiv.org/abs/2407.04622
Lizhou “Leo” Fan retweeted
Why did language evolve among humans?
To facilitate thinking? Or to facilitate communication?
This new perspective article just published in @Nature suggests that language evolved as a tool for communication.
The main evidence against the language-for-thought hypothesis comes from patients with brain damage and disorders. There are people with major linguistic impairments but virtually intact cognitive abilities.
Additional evidence comes from fMRI studies suggesting that the language network is virtually disjoint from networks involved in thinking.
The main evidence supporting the language-for-communication hypothesis comes from research highlighting that the structure of language seems to be optimized for ease of production, learning, and robustness to noise.
For example, the tendency to minimise dependency length, the preference for certain word orders (e.g., subject-object-verb and subject-verb-object), and the presence of ambiguity, seem to be useful for decreasing the cognitive cost of communicating.
Full paper: nature.com/articles/s41586-0…
Lizhou “Leo” Fan retweeted
Is that what we call Bingo? 🎯
"Samba = Mamba + MLP + Sliding Window Attention + MLP stacking at the layer level." => infinite context length with linear complexity
Samba-3.8B-instruct outperforms Phi-3-mini across all benchmarks using the same dataset (trained on 3.2 trillion tokens from the Phi3 dataset)
Lizhou “Leo” Fan retweeted
Congratulations, Dr. Fan! 🎉
UMSI’s Lizhou “Leo” Fan successfully defended his dissertation, “Generative AI-augmented and User-centric Research Data Discovery and Reuse.” He’ll soon join @BrighamWomens as a postdoctoral research fellow 👏
@LeegeoF #PhDone
Lizhou “Leo” Fan retweeted
New research from UMSI:
A dataset for measuring the impact of research data and their curation
Libby Hemphill, Andrea Thomer, Sara Lafia, Lizhou Fan, David Bleckley, Elizabeth Moss
@libbyh @lafia_s @LeegeoF @ScientificData
nature.com/articles/s41597-0…
Lizhou “Leo” Fan retweeted
😳 How much is VLM's reasoning ability lagging behind LLM's reasoning ability?
🔖 We construct NPHardEval4V 🔥, the visual counterpart of NPHardEval
▶️▶️▶️ After removing the effect of recognition failure and instruction-following failure, VLMs are much worse on reasoning
Thank you @huggingface and @clefourrier for featuring our NPHardEval benchmark and leaderboard. Started later Jan, we update the test data monthly and will release updated evaluation (possibly with more models) the next Month. Have your favorite LLM in mind? Let us know!
New leaderboard: NPHardEval!
It uses logical questions of diff. complexities as a proxy for reasoning abilities 💪
Since the questions can be generated automatically, it's going to be dynamic, updated monthly! 🚀
Congrats to @HuaWenyue31539 @LeegeoF !
huggingface.co/blog/leaderbo…
Lizhou “Leo” Fan retweeted
NPHardEval benchmark updated version!
1. Phi-2 has a good performance (Huggingface leaderboard here: huggingface.co/spaces/hyfran…)
2. Robustness experiment: Our benchmark is actually robust under finetuning hack! See details in the paper
arxiv.org/abs/2312.14890
Lizhou “Leo” Fan retweeted
Under NPHardEval benchmark, GPT-4 is still far beyong all other models.
Open source models are going close after GPT-3.5 and Claude: Qwen, Yi, Mistral and Phi-2 (even it is only 2.7b! Textbook is indeed all we need haha) show good performance.
Lizhou “Leo” Fan retweeted
What is LLM's ability in solving P, NP, NPC, and NP-Hard problems? Check out @LeegeoF @HuaWenyue31539 etc.'s paper on evaluating LLM's reasoning ability via complexity classes.
Lizhou “Leo” Fan retweeted
Can LLM solve NP-hard problem?
We proposed a new benchmark to rigorously evaluate LLMs' reasoning ability: NPHardEval:
1. Built based computational complexity hierarchy
2. Automatic datapoint generation
3. Automatic result checking
4. Monthly refresh -- no overfitting!
Lizhou “Leo” Fan retweeted
My team (@libbyh @an_dre_a_ @LeegeoF) published our analysis of data reuse communities in the @ICPSR Bibliography: "Subdivisions and Crossroads: Identifying Hidden Community Structures in a Data Archive’s Citation Network" (doi.org/10.1162/qss_a_00209) @QSS_ISSI #datacitation
Preview (2/2). I was honored to work with
@ToddPresner at @UCLA_DH where I grew from a UG student to a data scientist. We used semantic triplets (RDF triple) for "Algorithmic Close Reading". digitalhumanities.org/dhq/vo… I'll carry experiences from both projects and explore more!
1
Two article previews to share (1/2). They are respectively the remarks of my previous work at UCLA. One is a (computational) archival science work with Prof. Anne Gilliland @uclaseis. We used ML to enhance the processing of social media archives dl.acm.org/doi/abs/10.1145/3…
1
1




























