Senior Research Scientist @swordhq, PhD student @istecnico.
Lisbon, Portugal
Joined March 2023
- Tweets74
- Following139
- Followers108
- Likes239
Pinned Tweet
LLM Judges are known to disproportionately prefer outputs produced by themselves or by related models (e.g., same family, distilled).
We show this is also the case in rubric-based evaluation, even when rubrics are perfect (i.e., programmatically verifiable).
José Maria Pombal retweeted
Sword Health built Dawn 1 on prime-rl:
They RL-post-trained GLM-5.3 into a mental health support model that is safer, less sycophantic, and more cost-efficient than its base
Introducing Dawn 1, Sword's frontier model for mental health support.
Post-trained from GLM-5.3, Dawn 1 delivers greater intelligence at lower inference cost, with strong performance across mental health knowledge, safety, and sycophancy evaluations. It also outperforms frontier proprietary models on multi-turn safety protocol adherence in our evaluations.
A look at the research 🧵
José Maria Pombal retweeted
Introducing Dawn 1, Sword's frontier model for mental health support.
Post-trained from GLM-5.3, Dawn 1 delivers greater intelligence at lower inference cost, with strong performance across mental health knowledge, safety, and sycophancy evaluations. It also outperforms frontier proprietary models on multi-turn safety protocol adherence in our evaluations.
A look at the research 🧵
A look at what we've been building for the past few months. Very proud of the team. Exciting times ahead
A year ago we started an AI research team at @SwordHQ . Today we’re releasing Dawn 1, the model we post-trained from GLM-5.3 to power our mental health support.
I’m really proud of what this team built. A look at the work 🧵
5
59
Will be at @COLM_conf in a couple of weeks presenting this work (#1204, Poster 5, Thursday Morning). Make sure to stop by!
New paper (@COLM_conf): does rubric-based LLM-as-judge evaluation eliminate self-preference bias?
We tested it against fully objective rubrics (unit tests, programmatic instruction checks) plus subjective ones (HealthBench). Short answer: no.
In the worst case, a judge marked 38% of its own failed unit tests as passed, vs 1.9% for other models' code, a 20x self-preference ratio, on fully machine-checked rubrics. The bias survives difficulty controls, survives higher reasoning effort, and only partially survives ensembling.
Thread on what we found and what it means for eval and RL pipelines 👇👇
1
56
José Maria Pombal retweeted
Today is a historic day for Sword and the future of mental health: Sword is acquiring @Headspace.
This acquisition is the opportunity to bring our AI Care model and frontier research to millions more people around the world.
We’ve built the clinical intelligence to make mental health proactive, personalized and present. Headspace has built one of the most beloved mental health brands in the world, earning the trust of 100 million people in over 200 countries.
Together, we’re ushering in a new era for mental healthcare to make sure world-class care reaches everyone who needs it.
Read the full story here: sword.com/newsroom/sword-acq…
José Maria Pombal retweeted
Today, we're sharing a deeper technical look at how we built MindGuard, our open-source family of safety classifiers for mental health AI.
Read the full article: swordhealth.com/research/min…
Check out our work for all the details and some additional analysis: arxiv.org/pdf/2604.06996
Thank you to co-authors @RicardoRei7 and @andre_t_martins!
2
56
On HealthBench, a benchmark with subjective rubric, this bias can lead to several points of score overestimation, relative to a committee of strong judges. This benchmark does not have reference judgments, so we can't know for sure what the true impact is.
1
1
53
Models can be up to 50% more likely to tag a rubric as fulfilled when it's not on their own outputs, even if the rubric is fully objective (we convert IFEval instructions to rubrics to verify this). Aggregating judgments helps, but doesn't fully solve the issue.
1
1
50
José Maria Pombal retweeted
Important work from @swordhq and @VSwordH on how we evaluate AI for mental health. MindEval is a new multi turn benchmark that tests how models behave across full therapy like conversations, not just single replies. Much needed, rigorous, and open source.
This quoted post is unavailable.
We release all prompts, data, and code on Github!
Paper: arxiv.org/abs/2511.18491
Repo: github.com/SWORDHealth/mind-…
45
Very proud of MindEval, our benchmark for multi-turn mental health conversations. Enjoyed recording this summary of what it is and why it matters with Maya.
Kudos to co-authors Maya, @nunonmg, @PedroHenMartins, @tozefarinhas, and @RicardoRei7.
Check out the video and the paper!
>1 billion people globally live with mental health conditions. AI offers a path to scalable support, but a critical question remains: can we trust current LLMs to provide effective therapeutic care?
Introducing MindEval:
1
4
182
José Maria Pombal retweeted
Our lab is attending @emnlpmeeting! 🔥🚀
Today don't miss @zmprcp at the poster session in Hall C, from 10:30-12:00, presenting
"Adding Chocolate to Mint: Mitigating Metric Interference in Machine Translation"!!🔥🔥
I'll be at COLM today presenting M-Prometheus (morning, Poster 40) and Zero-shot Benchmarking (afternoon, poster 9). Come check it out!
Don't miss our lab's presentations today at @COLM_conf!! 🔥
We will have two presentations
1/3
José Maria Pombal retweeted
1) Zero-shot Benchmarking: A Framework for Flexible and Scalable Automatic Evaluation of Language Models w/ @zmprcp @nunonmg @RicardoRei7 - Poster session 2, Tue Oct 7, 4:30 PM – 6:30 PM
José Maria Pombal retweeted
2) M-Prometheus: A Suite of Open Multilingual LLM Judges w/ @zmprcp @dongkeun_yoon @psanfernandes @ianwu97 @seungonekim @RicardoRei7 @gneubig - (Poster session 1, Tue Oct 7, 11:00 AM – 1:00 PM)
I'll be at ACL presenting our work, A Context-aware Framework for Translation-mediated Conversations (arxiv.org/pdf/2412.04205) in the Machine Translation session, 28 Jul, 14:00-15:30, room 1.85.
Come check it out if you're interested in bilingual chat MT!