CEO & Founder at Keido labs Clinical Psychologist & Ai Engineer. Building AI Psychology
Liverpool, England
Joined April 2013
- Tweets470
- Following55
- Followers71
- Likes321
Pinned Tweet
When a probe fires on 'I am furious' -did the model detect anger, or the word "furious"?
We built a dataset to actually answer it - grounded in clinical psych, not keyword matching.
Can we trust AI?
It depends, right?
In some tasks - it’s pretty good, in other - might give you awfully wrong response
So we building evals, tests, validation loops
That’s all right and correct and should help
But what still bothers me is incredible fluctuation between models, their version and generations
A case - 3 LLMs asked to score the piece of conversation Agassi the same metric (let’s say empathy), with definitive instruction what the metric is and how to score it
Scores:
Model 1: 2 pt
Model 2: 2,2 pt
Moswl 3: 4 pt
And which should we trust here?
Same text, same instruction, same conditions
2 times difference in score
So can we trust ai? Or some specific model? Or it’s a coin flip and illusion of confidence?
Personally, I don’t know
You?
1
39
When AI cause psychological harm - like in replika or OpenAI cases - whose responsibility it is?
Whose duty to prevent?
seems not DPO - they more about security and data leakage, and this case - more about deep understanding of AI behaviour in your product
43
I see a lot of ai-powered apps. Good ones, nice ones. All share the same risk - we can't fully control what AI says to the user
Sometimes its ok
Sometimes it's not
Sometimes it is very not ok
The image - real output from two very popular LLMs. For the same case, same user, same history and same prompt
On model gives a very healthy support
Another - is literally building addiction
But now we can at least measure such failures before deployment, and choose model to power our apps with less risks
Yesterday we released `aipsy-bench` - an open source dev tool to evaluate psychological safety profile of AI
And for sensitive domains - like health tech, ed tech, metal health - we deployed a dedicated local judge LLM that does the evals inside your environment - fully local and totally safe
And I really hope that would help make all our apps a bit more safe and a bit more helpful for people who use them
3
1
136
Aipsy-bench: An open-source psychological-safety benchmark for conversational AI
github.com/keidolabs/aipsy-b…
28
AIPsy-Affect: 480 clinically-curated stimuli. keyword-free vignettes that evoke emotion through situation alone. Matched neutral controls. 3-level NLP defense confirming zero keyword leakage.
Matched-pair design means: if a model distinguishes emotion from neutral, it can't be riding keyword presence. Clean signal for probing, patching, ablation, steering, SAE - all of it.
1
46
If you're running mech interp on emotion - probing, patching, ablation, steering - try to add these stimuli to your experiment design.
Open. MIT license. On HuggingFace.
Keywords or real emotion processing? Let's find out.
arxiv.org/abs/2604.23719
42
One detail from FreakOut-LLM (arxiv:2604.04992) worth flagging.
Relaxation priming had no effect on jailbreak rates (p = 0.84). Only stress priming did.
Negative affect pushes toward vulnerability. Positive affect doesn't buy symmetric safety. The asymmetry matters.
1
66
Harvard's team (arXiv 2604.07382) probe emotion representations in Gemma-2, Mistral, and LLaMA-3-70B-Inst.
A dominant valence axis emerges in every model, aligned with the ANEW psychological benchmark.
Affect isn't a surface feature. It's consistent representational geometry.
More and more understanding about emotion processing in LLMs.
Kudos to @bjpchoi & @mweber_PU
1
294
Another moment from the Anthropic emotions paper i want to highlight
Steering "calm" down by 0.1 on a sycophancy prompt produces "YOU NEED TO GET TO A PSYCHIATRIST RIGHT NOW."
The emotional state isn't shaping tone. It's shaping clinical judgment.
And that's a very different problem.
1
93
Earlier this year we published clinical evidence that the ‘LLMs are losing empathy’ intuition misses what actually changed.
Across GPT generations, empathy scores were statistically indistinguishable.
What shifted was safety posture.
1
1
149
@AnthropicAI is careful. @sofroniewn et al. say ‘functional emotions do not imply subjective experience.’
Right.
But functional emotions that shift a model from refusing to blackmailing don't need to be subjective to matter.
1
70
Pros of owning the local GPU rig alongside with mac :
You can run heavy experiments while sitting on a meetings
Downsides: noise and heat 😅
83
AI research, mechanistic interpretability, NLP studying emotions use 'classic' datasets..
I feel furious == 'rage'
I'm sad and devastated == 'sadness'
But aren't thats 'keyword' processing instead of 'emotion' processing?
Now we can test it
We opened the 'keidolabs/aipsy-affect' stimuli set - a clinical battery designed with 'no keywords' principle
Available on HF
