Andrew Brackin

Introducing RecruiterBench, the people-search benchmark most aligned with recruiting workflows. At Wrangle, when developing our internal models, we realized that all existing people-search benchmarks (that we’ve seen) are flawed: - They only score 10 candidates Real recruiting workflows require sourcing through 100 or more profiles to make a hire. Evaluating accuracy across 10 people doesn’t provide an adequate picture of how a tool will perform on a req. RecruiterBench measures precision over 200 returned candidates for each role, which we believe accurately represents search capabilities and performance. - The queries are overly simplistic and contrived Accuracy over “Product managers in Austin” doesn’t matter when you’re looking for someone with a specific skillset. We built RecruiterBench on 290 complete active reqs on Ashby, spanning 58 unique industries. This also allows us to find unique insights, such as what industries and role types each tool excels or struggles with. For example, some strong points we identified are: Wrangle deep on AI infrastructure: p@50 of 60.4%, versus Metaview 39.6%, Wrangle Turbo 52.2%, Exa 21.8%, and Treg 9.6%. Wrangle deep on food/agriculture: p@50 of 79.4%, versus Wrangle Turbo and Metaview at 56%, Exa 24.6%, and Treg 9.2%. Juicebox on legal: on the same three roles, p@50 was 82.7%, ahead of Turbo 76.3%, Metaview 65.7%, Exa 55.7%, and Treg 11.7%. Wrangle Deep still led at 93.3%. Metaview on Advertising/marketing tech: p@50 was 78.2%, leading Wrangle Deep's 77.8%, Wrangle Turbo’s 73.8%, Exa’s 45.6%, and Treg’s 17.6%.. - They evaluate with a single LLM judge, without nuance about must-have, dealbreaker, and nice-to-have criteria RecruiterBench uses a two-part LLM scoring system to first disqualify candidates based on must-haves, dealbreakers, location mismatches, and over/underqualification. Then we use a second judge to determine relevance and give no/partial/full credit. We’re excited to continue developing more accurate benchmarks, and to build the best recruiting tool in the world! Check out RecruiterBench on Github: github.com/john-c-10/Recruit….