Human data for frontier AI. Research and insights from Appen.
Washington, USA
Joined May 2026
- Tweets133
- Following216
- Followers1.1K
- Likes53
An agent can ace the eval and still fail the job. So what was the benchmark actually measuring?
Sergio Bruccoleri @SergioBrucc takes that on at @kaggle Bench Summit in SF, Wed Oct 28, 3 PM PT, on browser agents in enterprise workflows. Bring counterexamples.
A checker can verify 180 million lines of proof. It can't tell you if the spec says what you meant.
Evals have the same shape. For MedTerm-90 we reviewed 12,326 near-miss pairs under one rule: accept only what a clinician would sign off on. 21.2% passed. The rule did the work.
New on our Frontier Red Team blog: using Claude to formally verify software at large scale.
Formal verification can prove that code is free of certain bugs, but historically it has only been feasible for small codebases (or required a lot of expert time).
With AI models, however, this is beginning to change. In collaboration with @theoremlabs, we used Claude to prove security properties in over one hundred real-world open-source libraries over the span of a few weeks.
Read our research in full here: anthropic.com/research/forma…
Stress-test the rule yourself: full method and every judging call.
appen.com/research/medterm-9…
RL tasks are getting more expensive while environments are getting cheaper. That isn't a contradiction. Nobody was really paying for the sandbox.
You pay for the grader: one that rejects answers that look right, still works after the environment is patched, and has had its passing runs read by a person.
How we design verifiers: appen.com/whitepapers/reinfo…
Human vs synthetic labels for alignment evals usually gets framed as either/or. In practice it splits by the kind of question.
Synthetic labels hold up well on clear rule violations: leaked data, a refusal that should have happened, a broken format.
They're much weaker on judgment calls, like whether the agent should have asked first, whether it read intent correctly, or whether it was pursuing the goal or gaming it. Those need people who know the domain, plus a measure of how often those people agree.
55 call centre datasets. 14 languages. ~174,600 hours of audio.
DM us for samples and specs.
We have 55 call centre datasets covering 14 languages, about 174,600 hours in total.
The largest is 30,000 hours of US English automotive calls. Others include 8,000 hours of Indian English telecom, 5,000 hours of Hindi telecom, and 5,000 hours of Tamil ecommerce calls.
Each dataset has a spec sheet covering languages, domain, hours and how it was collected.
DM me if you want a sample.
A polished humanoid robot demo doesn’t necessarily mean the behavior will generalize.
Appen’s Jeanine Sinanan-Singh joined @therobotreport Podcast to talk about the generalization gap, human feedback, RL environments, and what it takes to evaluate robots beyond the demo.
Link in the comments.
What are agent benchmarks actually measuring?
@SergioBrucc VP of Delivery at @AppenResearch, writes in @TheAIInnov about why benchmark performance can fall short of real-world agent performance.
More on reliability, cost, and execution in the full piece.
Link in comments.
What does it actually mean to teach an AI agent to recover?
A correction, contradiction, or tool failure can each require a very different action: edit, clarify, retry, or preserve state.
Part 2 of our recovery-data series looks at why recovery data should be tagged by both trigger + required action.
Link in the comments.
Read Part 2 of the series here: appen.com/blog/ai-agent-reco…
Congrats to the @subquadratic team on the launch of SubQ Snap! Exciting to see what they’re building in long-context decision models. 🚀
Introducing SubQ Snap, the first long-context decision model for real enterprise work.
Snap is the only decision model or LLM where compute per token stays constant as context grows.
• 384K tokens of multimodal context, 12x larger than Jev.
• 61.7 on Decision Index v0.2.1 vs Jev’s 57.9, outperforming on 13 of 16 enterprise benchmarks.
•Processes 100K tokens and returns a decision in under one second.
•$0.05/MM input tokens. Output is free.
Available today to anyone.
The first 10,000 sign-ups get 100M free input tokens.
We’ll likely hit capacity fast and will add users as we add compute. Compute offers welcome 🙂.
Latency will be variable as we tune the platform to this new model and category of users.
Model speeds are 80ms for minimal context, 250ms for 32k, 660ms for 100k, and 2.5 seconds for 384k.
Visit: subq.ai/
Thanks to everyone who joined @AppenResearch at our #COLM2026 @COLM_conf happy hour in SF! Lots of great conversations with researchers and builders about evals, LLM-as-judge, post-training data, robotics video, and multilingual AI. See you at the next one 🍻
@AppenResearch is featured on AI Circle @joinaicircle’s Multilinguality Map, highlighting teams working across multilingual data, speech, evaluation and low-resource languages.
As models expand globally, dialects, code-switching and cultural context matter more than ever.
Map: ai-circle.org/multilingualit…
More on the work: appen.com/multimodal-ai
appen.com/multilingual-ai-tr…
Appen Research retweeted
Nice to see @SergioBrucc on this list
I asked people: "who have you met in the AI data and post-training space that you find most insightful?"
5 votes: Daniel Chang (@dnnssl2)
3 votes each:
- Abhijay Rana (@abhijaymrana)
- Ryan Wexler (@RyanMWexler)
- Sean Cai (@SeanZCai)
- Will Brown (@willcb)
2 votes each:
- Amy Tam (@amytam01)
- Jesse Hu (@huyouare)
- Nikhil Namburi (@nikhilvnamburi)
- Thomas Liao (@distributionat)
91 people were picked. The other 82 got 1 vote each and are in the next two posts.
People couldn't pick themselves. I didn't vote.
We get to work with @SergioBrucc every day. This checks out.
I asked people: "who have you met in the AI data and post-training space that you find most insightful?"
5 votes: Daniel Chang (@dnnssl2)
3 votes each:
- Abhijay Rana (@abhijaymrana)
- Ryan Wexler (@RyanMWexler)
- Sean Cai (@SeanZCai)
- Will Brown (@willcb)
2 votes each:
- Amy Tam (@amytam01)
- Jesse Hu (@huyouare)
- Nikhil Namburi (@nikhilvnamburi)
- Thomas Liao (@distributionat)
91 people were picked. The other 82 got 1 vote each and are in the next two posts.
People couldn't pick themselves. I didn't vote.
We’re bringing the AI research community together in San Francisco during @COLM_conf
Drinks, bites and good conversations with researchers and builders working on frontier AI.
Oct 7 · San Francisco
Join us: luma.com/yhcfodng
30 years of @AppenResearch and a lot has changed along the way.
CEO @kolln_ryan000 Ryan Kolln joined the @VantagePointNWR Vantage Point Podcast to talk about Appen’s journey and the work we’re doing today with customers around the world.
Spotify: open.spotify.com/episode/14F…
YouTube: youtube.com/watch?v=KAx8FOOO…