Keenable AI

Keenable AI Open-Sources NEEDLE: A Live Search Benchmark That Rebuilds Its Query Set Every Hour Most search benchmarks are a fixed question set with a public answer key. That's not a benchmark for agents — because an agent with a fetch tool can just download the key mid-evaluation. Keenable open-sourced NEEDLE — a live search benchmark that rebuilds its query set instead of freezing one. News queries regenerate every hour from RSS feeds and Google Trends. Finance, scholar, and legal gold regenerates daily from public registries, SEC XBRL, arXiv, Europe PMC, CourtListener, and the eCFR. Rare-entity queries are sampled daily from public agent-trajectory logs. Here's what's actually interesting: → Queries are archived publicly after every run, so the defense isn't secrecy — it's that each run scores engines on queries newer than the models being tested → 18 engine configurations from 14 providers, one protocol: same query text, same 2,000-character evidence cap, one call at a time, judge blind to engine identity → Every score is read against ultimate, a pooled oracle engine that separates "this engine ranked badly" from "nobody found better." On rare-entity queries from real agent logs, the leader sits at 0.533 of that ceiling → There's an index-independence audit. Agreeing on good results proves nothing, so it flags a pair when they share a below-relevance URL, or share three-plus URLs on one query. In the published overlap data, Brave and You dot com are flagged on 89% of comparable queries Latency is treated as a quality metric — agents call search dozens of times per task, so p50 to p95 lands inside the user's wait. Full analysis: marktechpost.com/2026/08/31/… Technical Details: keenableai.github.io/needle/ @KeenableAI