Creativity & Science of Intelligence | Researcher at AI Innovation Institute (AI3) & Knowledge Lab, UChicago | Previously at Simons Institute, UC Berkeley
New York, NY
Joined July 2023
- Tweets75
- Following218
- Followers556
- Likes678
Pinned Tweet
🧵[1/N] Are large language models more creative than humans at combining ideas? 🤔
Combinatorial creativity involves combining familiar ideas in unfamiliar ways, and is central to scientific discovery and technological innovation.
In fact, breakthrough scientific work is often associated with more distant combinations of methods, techniques, and concepts.
Here, we introduce Kombine, a new benchmark of combinatorial creativity for LLMs.
TLDR:
- LLMs are more combinatorially creative than the average human.
- However, conceptual blending is the hardest task for LLMs, and even frontier models struggle.
- LLMs also tend to homogenize, inventing concepts with the same properties across separate models.
- Moreover, humans and LLMs combine in distinct regions of semantic space.
- As a result, we anticipate greater conceptual diversity can be unlocked through human-AI collaboration. 🗣️
Thread below🧵⬇️
🎉 Another win for weight-space steering over activation steering!
Happy to be a part of this exciting project with the great @xy51_uiuc, @Ali_E_B, and colleagues to advance the frontier of knowledge and skill localization in LLMs.
Check out the paper here: arxiv.org/abs/2610.21189v1
🚨New paper!
Can you steer an LLM to resist any harmful instruction? Should you do this in activation space or in weight space?
Our new paper introduces a weight-space steering method (do we have a name for it?) to increase honesty and decrease other unwanted behaviors like sexism.
TL, DR:
- When ordered to lie, our steered models give honest answers 63% → 85%
- When ordered to be sexist, our steered model gives sexist answers 58% → 24%
Samuel Schapiro retweeted
What a pleasure to work with the brilliant @samschapiro and colleagues on this important work to benchmark and expand model creativity in this era of convergence! Bravo!
🧵[10/N] In summary, Kombine provides a framework to assess progress towards more creative AI systems and human-AI collaborations.
Our tasks assess creativity in conditions more representative of real-world innovation and scientific discovery, and may be of interest to psychologists who study human creativity as well! 🧠
Thanks to all my co-authors for their contributions on this work: @akjagadish @alexiglad @LiuQiawen @sumukx @jonahablack @compulsi0n @dilekhakkanitur @hengjinlp Claire Stevenson @lrvarshney @Roger_Beaty @cocosci_lab @profjamesevans
Paper: arxiv.org/abs/2509.21043
Samuel Schapiro retweeted
this is an exciting!
but i firmly believe a *human can be a better combinatorial synthesizer than any LLM in existence
*human: neurodivergent
🧵[1/N] Are large language models more creative than humans at combining ideas? 🤔
Combinatorial creativity involves combining familiar ideas in unfamiliar ways, and is central to scientific discovery and technological innovation.
In fact, breakthrough scientific work is often associated with more distant combinations of methods, techniques, and concepts.
Here, we introduce Kombine, a new benchmark of combinatorial creativity for LLMs.
TLDR:
- LLMs are more combinatorially creative than the average human.
- However, conceptual blending is the hardest task for LLMs, and even frontier models struggle.
- LLMs also tend to homogenize, inventing concepts with the same properties across separate models.
- Moreover, humans and LLMs combine in distinct regions of semantic space.
- As a result, we anticipate greater conceptual diversity can be unlocked through human-AI collaboration. 🗣️
Thread below🧵⬇️
Samuel Schapiro retweeted
A very interesting paper, confirming some of the "vibes"/"folks wisdom" people get when collaborating with LLMs in practice.
🧵[1/N] Are large language models more creative than humans at combining ideas? 🤔
Combinatorial creativity involves combining familiar ideas in unfamiliar ways, and is central to scientific discovery and technological innovation.
In fact, breakthrough scientific work is often associated with more distant combinations of methods, techniques, and concepts.
Here, we introduce Kombine, a new benchmark of combinatorial creativity for LLMs.
TLDR:
- LLMs are more combinatorially creative than the average human.
- However, conceptual blending is the hardest task for LLMs, and even frontier models struggle.
- LLMs also tend to homogenize, inventing concepts with the same properties across separate models.
- Moreover, humans and LLMs combine in distinct regions of semantic space.
- As a result, we anticipate greater conceptual diversity can be unlocked through human-AI collaboration. 🗣️
Thread below🧵⬇️
Samuel Schapiro retweeted
🧵[1/N] Are large language models more creative than humans at combining ideas? 🤔
Combinatorial creativity involves combining familiar ideas in unfamiliar ways, and is central to scientific discovery and technological innovation.
In fact, breakthrough scientific work is often associated with more distant combinations of methods, techniques, and concepts.
Here, we introduce Kombine, a new benchmark of combinatorial creativity for LLMs.
TLDR:
- LLMs are more combinatorially creative than the average human.
- However, conceptual blending is the hardest task for LLMs, and even frontier models struggle.
- LLMs also tend to homogenize, inventing concepts with the same properties across separate models.
- Moreover, humans and LLMs combine in distinct regions of semantic space.
- As a result, we anticipate greater conceptual diversity can be unlocked through human-AI collaboration. 🗣️
Thread below🧵⬇️
Samuel Schapiro retweeted
🚨New preprint alert
Introducing Kombine: our benchmark for measuring LLM creativity.
A key driving factor behind every genuinely novel scientific discovery is creativity. One such form of creativity is combinatorial creativity, which is the ability to combine disparate ideas to come up with a new one.
Yet we do not yet know:
1. How to systematically measure this form of creativity in LLMs?
2. How creative are current LLMs at combining and composing known concepts?
3. Are LLMs and humans creative in the same ways or do they complement one other?
Led by the inimitable @samschapiro, we draw on the extensive literature on measuring and quantifying human creativity to answer these questions 🧵
🧵[1/N] Are large language models more creative than humans at combining ideas? 🤔
Combinatorial creativity involves combining familiar ideas in unfamiliar ways, and is central to scientific discovery and technological innovation.
In fact, breakthrough scientific work is often associated with more distant combinations of methods, techniques, and concepts.
Here, we introduce Kombine, a new benchmark of combinatorial creativity for LLMs.
TLDR:
- LLMs are more combinatorially creative than the average human.
- However, conceptual blending is the hardest task for LLMs, and even frontier models struggle.
- LLMs also tend to homogenize, inventing concepts with the same properties across separate models.
- Moreover, humans and LLMs combine in distinct regions of semantic space.
- As a result, we anticipate greater conceptual diversity can be unlocked through human-AI collaboration. 🗣️
Thread below🧵⬇️
Samuel Schapiro retweeted
A benchmark for artificial creativity.
🧵[1/N] Are large language models more creative than humans at combining ideas? 🤔
Combinatorial creativity involves combining familiar ideas in unfamiliar ways, and is central to scientific discovery and technological innovation.
In fact, breakthrough scientific work is often associated with more distant combinations of methods, techniques, and concepts.
Here, we introduce Kombine, a new benchmark of combinatorial creativity for LLMs.
TLDR:
- LLMs are more combinatorially creative than the average human.
- However, conceptual blending is the hardest task for LLMs, and even frontier models struggle.
- LLMs also tend to homogenize, inventing concepts with the same properties across separate models.
- Moreover, humans and LLMs combine in distinct regions of semantic space.
- As a result, we anticipate greater conceptual diversity can be unlocked through human-AI collaboration. 🗣️
Thread below🧵⬇️
Samuel Schapiro retweeted
how creative are current LLMs?
we studied this through *combinatorial creativity*, check it out!
🧵[1/N] Are large language models more creative than humans at combining ideas? 🤔
Combinatorial creativity involves combining familiar ideas in unfamiliar ways, and is central to scientific discovery and technological innovation.
In fact, breakthrough scientific work is often associated with more distant combinations of methods, techniques, and concepts.
Here, we introduce Kombine, a new benchmark of combinatorial creativity for LLMs.
TLDR:
- LLMs are more combinatorially creative than the average human.
- However, conceptual blending is the hardest task for LLMs, and even frontier models struggle.
- LLMs also tend to homogenize, inventing concepts with the same properties across separate models.
- Moreover, humans and LLMs combine in distinct regions of semantic space.
- As a result, we anticipate greater conceptual diversity can be unlocked through human-AI collaboration. 🗣️
Thread below🧵⬇️
🧵[9/N] Human-generated entities were more often everyday objects, states, and feelings, while LLM-generated concepts were more likely people and places.
Similarly, human-generated relations were more often verbs that expressed motion or plain states of being, while LLM-generated concepts were more often verbs that expressed creation, communication, and cognition.
🧵[10/N] In summary, Kombine provides a framework to assess progress towards more creative AI systems and human-AI collaborations.
Our tasks assess creativity in conditions more representative of real-world innovation and scientific discovery, and may be of interest to psychologists who study human creativity as well! 🧠
Thanks to all my co-authors for their contributions on this work: @akjagadish @alexiglad @LiuQiawen @sumukx @jonahablack @compulsi0n @dilekhakkanitur @hengjinlp Claire Stevenson @lrvarshney @Roger_Beaty @cocosci_lab @profjamesevans
Paper: arxiv.org/abs/2509.21043
Samuel Schapiro retweeted
Excited to share the slides from my talk at the SciFM Workshop at COLM 2026 (@COLM_conf) are now available online
In the talk, I provided a taxonomy of types of creativity recognized in cognitive science, and then discussed our new work that introduces a transformational creativity benchmark for LLMs.
📑 Paper: openreview.net/forum?id=dFgW…
🗣️Slides: schapiro.ai/files/TwistBench…
Joint work with fabulous authors @AlexiGlad @hengjinlp @Roger_Beaty
Excited to share the slides from my talk at the SciFM Workshop at COLM 2026 (@COLM_conf) are now available online
In the talk, I provided a taxonomy of types of creativity recognized in cognitive science, and then discussed our new work that introduces a transformational creativity benchmark for LLMs.
📑 Paper: openreview.net/forum?id=dFgW…
🗣️Slides: schapiro.ai/files/TwistBench…
Joint work with fabulous authors @AlexiGlad @hengjinlp @Roger_Beaty