Plurai

If you use LLM-as-judge, this one is for you. (bookmark it) Most teams validate their agent's outputs by calling a frontier model as the judge. It works, until it doesn't. Three problems stack up fast: → Cost: you're hitting a frontier API on every turn, every tool call, every response. In production that burns millions. → Latency: bigger models, remote calls, slow reasoning on every check. → Blind spots: frontier models don't actually know your domain. In finance, insurance, or healthcare, they miss the keywords and principles your work depends on. So I walk through a different approach: train your own small LLM judge. Instead of a giant model, you start with a small one and let the system generate the training data for you. It decomposes your domain, samples synthetic examples, runs them through a debate arena where judges reach consensus, then trains on the refined set. The result is a judge that's cheaper, faster, and more accurate on your data than Gemini, Claude, or GPT, with an OpenAI-compatible endpoint you can even deploy on-prem. I show the whole thing end to end, using a Claude Code plugin and a web interface, with a real insurance RAG grounding evaluator as the example. You can get the plugin here: github.com/plurai-ai/plurai-… Here's the full breakdown: 00:00 - Intro 00:12 - Three problems with using frontier LLMs as judges 01:05 - A different approach: train your own small judge 01:31 - How it works (synthetic data and a debate arena) 02:50 - Installing the Claude Code plugin 04:03 - Defining your task with /eval 04:34 - Example: an insurance RAG grounding evaluator 05:51 - Kicking it off and giving early feedback 06:26 - Choosing labels, domain, and strictness 08:30 - The web interface and dashboard 09:52 - Bringing your own example data (optional) 10:26 - The finished model: endpoint, accuracy, and speed 11:16 - Control, on-prem deployment, and interpretability 11:57 - Benchmarks vs frontier models and the GitHub repo 12:30 - Outro I worked with the @pluraiAI team on this. Thanks for sponsoring the video.

Vibe train your AI agents. There's a new method that could replace LLM-as-a-judge for production agents. Most teams rely on a giant LLM as a judge to evaluate and guard their agent. But it has two major drawbacks: - It's slow and expensive at inference time - It often misses domain-specific failures Vibe training flips this. Researchers at Plurai distill a small language model that's specialized for your agent's exact use case. The SLM becomes your evaluator and your runtime guardrail, both in one. The training data isn't hand-curated either. They spin up a swarm of adversarial agents that debate and stress-test every use case your agent is supposed to handle. That synthetic interaction data trains the specialized SLM. So the judge actually understands what "wrong" looks like in your specific domain. The reported gains vs. standard LLM-as-a-judge setups: - ~8x faster inference - ~50% fewer evaluation errors Smaller, faster, and more accurate because it's specialized for the job. The SLM-for-agents thesis is playing out in a very concrete way. If LLM-as-a-judge is your current evaluation layer, this is worth benchmarking against. Paper link in the replies.