New preprint! 📜
We apply classic visual search paradigms from cognitive psychology to multimodal LLMs. Instead of just benchmarking outputs, we probe the human-likeness of their responses. How do models like GPT‑4o, Claude and Llama respond to simple vs. compound visual features? A thread 🧵
We think this type of work is valuable --- rather than identifying whether a model *can* perform some task, we want to get a better picture of the computational/cognitive processes driving performance. This sits somewhere between benchmarking and Mechanistic Interpretability---we try to see what the model is doing at a higher level of abstraction.
John Burden retweeted
#IJCAI2025 John Burden, University of Cambridge, delivering their talk #8954 on Paradigms of AI Evaluation: Mapping Goals, Methodologies and Culture at the Machine Learning (1/4).
"Just-In-Time" AI Evaluation: use the model for what you want, when you need it. If it works, use it. If it does t work, don't use it.
Stumbled upon some writing on mine from 2017. Em-dashes everywhere. Can't believe my natural writing style is now considered evidence of AI usage.
John Burden retweeted
it's not enough to have good ideas. they must be transcribed in the correct sacred format (illuminated manuscript), pay tribute to established scholars (tithe to the clergy), be judged by the ruling guild (peer review by the elders), and presented at costly gatherings (royal court feasts). all this, before they are locked behind castle walls (paywalled journals), where only the nobility (elite institutions) may access them
Are humans just "stochastic parrots"? Rather than underestimating AI have we been overestimating ourselves?
Vibecoders, is there a good way to use cursor/windsurf/whatever with my chatgpt o3? I.e custom instructions and visibility of other conversation threads?
John Burden retweeted
🚨The UK AISI identified four methodological flaws in AI "scheming" studies (deceptive alignment) conducted by Anthropic, MTER, Apollo Research, and others:
"We call researchers studying AI 'scheming' to minimise their reliance on anecdotes, design research with appropriate control conditions, articulate theories more clearly, and avoid unwarranted mentalistic language."
1/4
Someone on here recommended Permutation City a week or so ago. It was super interesting. Part 1 was fascinating and asked some deep questions (even if the resolution in the novel was non-computable!). Part 2 was quite... Different....
John Burden retweeted
The June edition of the AI evaluation digest. If you want to be up to speed with the scientific literature on AI evaluation, this is a good place to start.
open.substack.com/pub/aieval…
John Burden retweeted
LLMs • agentic AI • #DataScience 🧵
1/ 🚨 New paper: “Measuring Data-Science Automation: A Survey of Evaluation Tools for AI Assistants & Agents.” If you care about the impact of LLMs and LLM agents on Data Science and how to measure it, this is for you!
ALT Screenshot of paper header: data science automation with AI assistants and AI agents based on LLMs