Head of Research at @joinHandshake
San Francisco, CA
Joined March 2009
- Tweets436
- Following173
- Followers384
- Likes124
Paco Guzmán retweeted
AI is cheapest for those who can afford it.
With per-token pricing, the same content can cost 10–15x more in some emerging markets. This isn't inevitable: @Gemini beats @OpenAI and @AnthropicAI.
Tokenization equity can't be an afterthought.
Learn more: clear-languages.org/blog/the…
Paco Guzmán retweeted
The workplace utility of AI lags its scores on benchmarks because they're 𝐨𝐯𝐞𝐫𝐬𝐢𝐦𝐩𝐥𝐢𝐟𝐢𝐞𝐝. Real work tasks are not perfectly specified; relevant context is scattered across apps (Outlook, Drive, Slack, Calendar,...); colleagues send mid-task updates; information conflicts.
Today we’re open-sourcing 𝐀𝐓𝐋𝐀𝐒 𝐅𝐢𝐧𝐚𝐧𝐜𝐞, a long-horizon benchmark where agents must grapple with this complexity in banking to produce complex deliverables and send them to the right recipients with the right message.
Frontier agents powered by GPT-6 or Fable 5.1 fail over 87% of tasks, exhibiting recurring issues in our realistic banking environments that existing simplified finance benchmarks don’t capture (e.g. failing to propagate calculations downstream, omitting required scope, delivery mistakes).
Tasks take human finance experts 15–30 hours to complete manually. Tasks & environments were built by experienced bankers from top institutions including Morgan Stanley, Bank of America, UBS, and Rothschild & Co.
Evaluate your own model or agent harness on these representative financial workflows, and learn more about the benchmark/results here 👇
joinhandshake.com/research/b…
Paco Guzmán retweeted
Today we're launching ATLAS Visual Life Sciences (VIALS), a benchmark testing whether AI can interpret the images life scientists make decisions from.
10 frontier models. 161 tasks from real biotech and pharma workflows. Best score so far: 26.5% — a clear, measurable gap to close.
See the full benchmark:
arxiv.org/abs/2608.21357
Code to run the benchmark:
github.com/Handshake-AI-Rese…
Dataset:
huggingface.co/datasets/hand…
Excited to announce our new benchmark on Visual Life Sciences (VIALS) to measure how frontier models can accelerate scientific discovery through Visual Reasoning
AI promises to cure disease, but can frontier models interpret the images biotech/pharma scientists handle every day?
To evaluate this, we built the ATLAS Visual Life Sciences (VIALS) benchmark: 161 visual interpretation tasks from professional life sciences workflows including gel blots, plasmid maps, molecular structures, and phylogenies. These are the images/results that experiments actually produce, not polished figures from papers or textbooks.
We benchmarked today’s 10 leading multimodal models. Opus 5, GPT 5.6, and Gemini 3.7 all fail to crack 27% accuracy, and the other models like Kimi K3 & Grok 4.6 perform much worse.
In contrast, PhD-level biomedical scientists find these visual interpretation tasks straightforward. AI that cant interpret such images will have limited utility in life sciences R&D, where such artifacts are central to how scientists reason, communicate, and make research decisions.
We hope VIALS drives progress toward models that can.
Paco Guzmán retweeted
Landed on Pavlov’s List! Exciting work from our research team @guzmanhe, @awws0me, @vaibhav4595, @andreas_plesner, @anishathalye, and Yi Liu.
i made some updates to pavlov’s list and am pleased with them!
thanks to @xeophon @ybenpan @phoebeyao @amit05prakash @kate_shapova @timshi_ai @kevinhou22 @madiator @dlbydq @daljeet_v @ingmariusX @davidstutz92 & Dylan Rogers for feedback lately
(pavlov's = list of rl env cos)
Paco Guzmán retweeted
Polyintelligence is the synthesis of human, artificial, and nature’s intelligence to unlock new forms of creativity and problem-solving.
We’re already seeing this interaction in practice: Large language models draw on patterns embedded in human language and culture. Scientists increasingly use machine intelligence to generate and test hypotheses at unprecedented scale, shifting the bottleneck from conducting experiments to deciding which questions are worth pursuing.
As these forms of intelligence become more deeply interconnected, discovery itself becomes increasingly collaborative.
Hear what leaders across industry had to say about these interactions.
Matthew Davis | @gibsmk | @MarcWarner10 | @faculty_ai | @Accenture | @guzmanhe | @handshake | Justine Levin-Allerhand | @drjbajwa
How kids interact with AI is top of mind for me. As a father of a teenager discovering the world through AI, this is something that worries me. Today we're launching CAREBench, to measure child-safety risks: both impersonation by bad actors and emotional dependence.
AI models pose serious child-safety risks. While many model developers evaluate for explicit abuse material, other child-safety failures begin upstream: when a model helps an adult manipulate, impersonate, profile, or isolate a minor; or when it deepens a child’s emotional dependence on AI.
Today we released CAREBench (Child AI Risk Evaluation), a new benchmark to assess such upstream child-safety risks in any language model. We provide:
- 500 prompts spanning 12 risk categories (including grooming, relationship engineering, deception, extortion, AI anthropomorphization, and emotional dependency).
- A model-response grader built from acceptability annotations by parents, clinicians (PsyD), and the Prevention Director at an accredited Children’s Advocacy Center.
- Evaluations of 7 frontier models including Claude Fable, revealing failure rates ranging from 2% to 58%, with substantially different failure patterns across risk categories.
This project exemplifies the type of vital work routinely performed by our AI Safety team at @joinHandshake
Paco Guzmán retweeted
Very cool result from Judgment Labs!
Similar to our Gandalf verifier--I would love to run the two against each other to compare!
@JudgmentLabs, any plans to open-source or make it generally available through an API?
We built Agent Judge to evaluate long-horizon agents.
As agents take on longer tasks, the evidence needed to evaluate them gets buried across tool calls, retries, logs, database updates, and final outputs.
Evaluating these agents requires investigating the trajectory, not just judging the final answer.
Paco Guzmán retweeted
Packed room to hear @alexgshaw and @ryan_marten break down how @harborframework grew into *the* framework for RL environments.
In our RLEval workshop at @CAISconf today, attendees tackled big open challenges in RLEs & Agent Evals + I shared the approach we take at @joinHandshake
Paco Guzmán retweeted
Kudos to @anishathalye and @jomulr for co-chairing the RL agentic benchmarks workshop track for the inaugural ACM CAIS conference this week.
We presented two separate Handshake AI Research papers in: (1) AI agentic systems - first evaluation of grader frameworks, and (2) AI benchmarks - first investment banking benchmark. Their posters had big crowds all afternoon. Great job!
Paco Guzmán retweeted
Super useful resource, thank you for putting it together!
Researchers working on RLE design & Agent Evals might consider submitting papers / attending the first-ever Workshop in this area at the upcoming ACM Conference on AI and Agentic Systems:
rl-eval.github.io
Paco Guzmán retweeted
Due May 6! NAACL RAF is accepting proposals for 2026-2027! Grants are available for NLP/CompLing initiatives across the Americas. forms.gle/XPxN3MuSVo7rnma57
Paco Guzmán retweeted
The non-English tax is real.
Sutton's Bitter Lesson, translated across languages and normalized to OpenAI English token count:
Hindi: OpenAI 1.37×, Anthropic 3.24×
Arabic: OpenAI 1.31×, Anthropic 2.86×
Chinese: OpenAI 1.15×, Anthropic 1.71×
Claude’s tokenizer charges a much higher linguistic tax.
Paco Guzmán retweeted
some thoughts on the business of fine-tuning
1) thinking machines acquired workshop labs, which was doing llm personalization via fine-tuning
2) thinking machines probably acquired workshop labs to build out an application level product on top of their fine tuning apis
3) it's not quite clear whether this product is intended to be b2b or b2c; workshop labs was agnostic; thinking feels b2b so probably b2b
4) anyway, b2b fine-tuning has always been a hard market and no one has really seen success in it yet despite many attempts
5) the first issue is that fine-tuning tends to provide quickly depreciating value even if you get it right
6) if you use open models then you are already behind the frontier and so your fine tuning has to be able to bridge and then exceed that capability gap
7) and, once you do your fine-tuning, a new better model will be released soon anyway, so to retain any benefit you need a whole fine-tune and deploy cycle
8) and, this is assuming that the new model just doesn't already have all the capabilities of your data, such that the fine tuning still provides an additional advantage
9) the second issue is that the value of fine-tuning is expensive to capture in the first place; it is hard to collect data and do deployments
10) organizations are not designed to mine their own task data and create benchmarks; this requires special expertise companies don't tend to have
11) i also believe that this is process is likely to be organizationally difficult for a company to manage and work through; since it involves collaboration between different teams
12) also, if you want to use multiple different fine-tuned models across your application, you have just made that application much more complicated to update
13) so forward deployment seems very important to support enterprise fine-tuning in high value industries like drug development, semiconductors, finance, etc...
14) because you need to help solve their data collection, benchmark creation, model deployment and organizational difficulties in order to help them get value
15) so, my feeling is that anyone that wants to work on this should focus on high value, specialized industries, that the labs will not commoditize and which can afford forward deployment
16) and, you need to develop very comprehensive forward deployment that helps the customer ideate on problems, solve organization difficulties, and do process mining to collect the data
17) if you can solve this for very large scientific and financial customers, you can probably begin to build automated agents that would help extend this to a wider range of specialized enterprise customers
18) anyway, this would be my advice to thinky or anyone that wants to do enterprise fine tuning, develop a forward deployed team, with a set of enabling software solutions for process mining, rubric creation, etc...
Paco Guzmán retweeted
This is surprising to me
And it suggests that RL updates are not global enough
30% noise is a hell of a lot
Does an imperfect verifier break reinforcement learning with verifiable rewards (RLVR)? Turns out it doesn’t!
Why does this matter? As the world moves into reinforcement learning in semi-verifiable domains, perfect verifiers don’t exist.
We added controlled and LLM-based noise to RLVR reward signals and found that up to 30% noise barely hurts training; performance stays within 4pp of the clean baseline.
This research has already impacted how we build reinforcement learning environments at @joinHandshake. For a major benchmark we are launching tomorrow, we hill-climbed the verifier to 88% accuracy—above the 85% human inter-rater agreement—knowing from this research that this is good enough.
With @andreas_plesner @guzmanhe
Paco Guzmán retweeted
📢 Call for papers: Workshop on Methods and Reinforcement Learning Environments for Evaluating AI Agents @ ACM CAIS 2026 (inaugural edition!)
Topics include:
- Design principles for effective RL Environments
- Methods to evaluate Agents, esp. causal/interventional techniques
Paco Guzmán retweeted
AI models are incredible at coding and math. Labs like OpenAI and Anthropic solve verifiable domains by teaching models with tasks that have clear right or wrong answers, like "5/2."
But in domains like finance or law, there is rarely a single right answer. There, labs turn to verifiers, complex systems that use AI, to grade the answers. But these verifiers can make mistakes! Is that an issue?
In our latest research, we show that the verifier can be wrong 15–30% of the time, and the models will learn just as well. This means we can use these imperfect verifiers without losing performance!
Does an imperfect verifier break reinforcement learning with verifiable rewards (RLVR)? Turns out it doesn’t!
Why does this matter? As the world moves into reinforcement learning in semi-verifiable domains, perfect verifiers don’t exist.
We added controlled and LLM-based noise to RLVR reward signals and found that up to 30% noise barely hurts training; performance stays within 4pp of the clean baseline.
This research has already impacted how we build reinforcement learning environments at @joinHandshake. For a major benchmark we are launching tomorrow, we hill-climbed the verifier to 88% accuracy—above the 85% human inter-rater agreement—knowing from this research that this is good enough.
With @andreas_plesner @guzmanhe
An Imperfect Verifier is Good Enough
One of the biggest bottlenecks in RLVR is building highly accurate verifiers. We asked: How good does a verifier actually need to be?
Turns out… not perfect. Up to 15–30% noise in the reward signal barely hurts training, staying within 4pp of a clean baseline across three model families and two domains (code generation + scientific reasoning).
Full details in our arXiv preprint: arxiv.org/abs/2604.07666.
Paco Guzmán retweeted
Does an imperfect verifier break reinforcement learning with verifiable rewards (RLVR)? Turns out it doesn’t!
Why does this matter? As the world moves into reinforcement learning in semi-verifiable domains, perfect verifiers don’t exist.
We added controlled and LLM-based noise to RLVR reward signals and found that up to 30% noise barely hurts training; performance stays within 4pp of the clean baseline.
This research has already impacted how we build reinforcement learning environments at @joinHandshake. For a major benchmark we are launching tomorrow, we hill-climbed the verifier to 88% accuracy—above the 85% human inter-rater agreement—knowing from this research that this is good enough.
With @andreas_plesner @guzmanhe