Evaluating LLMs on the Chatbot Arena leaderboard presents substantial challenges. Statistical analysis indicates that while the difference between GPT-4 and Claude-v1 is statistically significant, the sample size is insufficient.
Full post: mo-arvan.github.io/blog/2024…
In my years of data collection, I've noticed even expert annotators make mistakes, yet annotation validity is rarely discussed. Real-world data is messy and needs cleaning. Highlighting these issues in papers is crucial, as quality data is essential for a successful ML pipeline.
I'm looking into released models that can be used for training other models; while Llama 3 is by far the most capable, its license is to restrictive, will be using Phi and Mistral models.
Mo Arvan retweeted
Now we have our second ReproNLP paired presentation, @moarvan and @LewisNWatson presenting their reproductions of aclanthology.org/2022.acl-lo… #HumEval2024
ALT Lewis presenting (remotely)
Mo Arvan retweeted
Pls help us complete the 2024 edition of the #ReproHum survey of #NLProc and ML Researchers' Experience and Views of Reproducibility forms.gle/Jqh7yWnDfeSWm3MfA
Mo Arvan retweeted
The number of experimental flaws we found in NLP papers was surprising and depressing. I hope paper will encourage people to be more careful in their own experiments, and perhaps more skeptical when reading NLP papers.
Preprint of our CL paper is out reporting #ReproHum project results on Common Flaws in Running Human Evaluation Experiments in NLP. Including the 12 fun flaws in 6 non-cherrypicked experiments below. With @ThomsonSoftware and @EhudReiter. direct.mit.edu/coli/article-…
There was a question in the latest Fei-Fei Li interview with the Guardian that caught my eye; so I decided to use ChatGPT to analyze it.
Mo Arvan retweeted
#HumEval2023 Workshop on Human Evaluation of NLP Systems is happening this Thu 7 Sep, also presenting results from #ReproNLP Shared Task on Reproducibility of Evaluations in NLP - schedule here: humeval.github.io/
Mo Arvan retweeted
Attending my first #CogSci2023 conference this week and thoroughly enjoying it! If you're in Sydney, make sure to check out @moarvan's presentation (joint work with Mina Valizadeh, @parianhaghighat, Toan Nguyen, and @_HeejinJeong) in the 10:30 session: escholarship.org/uc/item/84z…
ALT Screenshot of the first page of "Linguistic Cognitive Load Analysis on Dialogues with an Intelligent Virtual Assistant" by Mohammad Arvan, Mina Valizadeh, Parian Haghighat, Toan Nguyen, Heejin Jeong, and Natalie Parde.
Mo Arvan retweeted
The US is notorious for it's mass shootings, but its immigration policies (or lack thereof) set a new standard for the art of shooting yourself in the foot with a bazooka.
Advice to graduate students from countries the US doesn't like: just go to Europe.
Run Llama-2 locally without the hassle!
Set up with ease using huggingface text-generation-inference and huggingface chat UI - just one line! By using a single Docker Compose config file.
Check it out 👉 github.com/mo-arvan/local-ll… #Llama2 #AI #HuggingFace #TextGeneration
Mo Arvan retweeted
Final call for papers for the workshop on Human Evaluation (HumEval). Submission deadline is 20 July
humeval.github.io/
I hate jupyter notebook.
I suppose it's fine for a short demo, short meaning less than 50 lines.
Otherwise, it's a pain to read, impossible to debug, and awful to reproduce. Where did we go wrong? What was wrong with Python? How did we get to this mess?
Mo Arvan retweeted
Our other paper in Findings of ACL, with @amelija16mp, @moarvan, and @anyabelz, looks at how empirical results in NLP research can vary based on normal variations in experimental conditions. 👩💻
Link: aclanthology.org/2023.findin…
Excited to chat more with anyone interested! (3/3)
Mo Arvan retweeted
The great majority of human evaluations in NLP is not repeatable and/or not reproducible and/or too flawed to justify reproduction.
A dire picture but also an opportunity for a rethink about how to design and report human evaluations in #NLP.
arxiv.org/abs/2305.01633
Special thanks for my collaborators Mina Valizadeh, @parianhaghighat, Toan Nguyen, @_HeejinJeong, and my advisor @NatalieParde
I came across a new work that outperformed Microsoft deepspeed a few days ago but I cannot find it now.
I've looked for hours on Twitter and LinkedIn, still nothing. It's driving me insane, any one knows what I'm talking about?!