Kev (@KevConti) header
Kev (@KevConti) profile picture

helping small businesses respond to leads faster, while keeping the human connection intact

Odessa, FL
Joined November 2023
Photo by Kev (@KevConti)Photo by Kev (@KevConti)Photo by Kev (@KevConti)Photo by Kev (@KevConti)Photo by Kev (@KevConti)Photo by Kev (@KevConti)Photo by Kev (@KevConti)Photo by Kev (@KevConti)Photo by Kev (@KevConti)
I just published my roadmap for building evals on highly subjective tasks. Here are the key steps involved and practical tricks for reducing noise / increasing confidence… Blog is here: cameronrwolfe.substack.com/p… Why is this difficult? Evaluating agents is already hard due to their broad scope, emphasis on expensive / long-horizon tasks, and non-determinism. Subjective tasks make this even harder because we cannot deterministically verify solutions and must rely on human judgments (or LLM judges). This introduces additional noise into an already difficult evaluation process. Evaluation roadmap. We can build reliable evaluations using a repeatable framework with three components: - Human evaluation: Our North Star for quality. Create quality guidelines that clearly define quality with the help of subject-matter experts, calibrate human evaluators to ensure agreement, and continually monitor evaluation quality. - Offline evaluation: Use calibrated LLM judges or task-specific rubrics as an efficient proxy for human evaluation. This allows us to scale experimentation without relying on expensive human evaluation for every new model. - Production monitoring: Track quality signals from real-world agent usage and use A/B testing to validate improvements before fully deploying a new agent. Multi-stage evaluation. These components form a tiered evaluation process: Offline evaluation → Human evaluation → Production A/B test → Full deployment We can test many ideas automatically, send promising candidates to human evaluation, and validate improvements through production A/B testing. Continual production monitoring then allows us to discover new use cases / failure cases in production that we can add into the earlier stages of evaluation to create a feedback loop. Sampling useful data. Evaluation quality depends heavily on which tasks we select. We want tasks that are diverse, realistic, difficult, discriminative, and accurate. Techniques like structured task taxonomies, model-based difficulty filtering, Item Response Theory (IRT), and LLM-based auditing help us construct more informative evaluation sets. Handling noise. We should leverage statistics throughout the evaluation process: - Power analysis: Estimate the number of evaluation samples needed to detect meaningful improvements. - Confidence intervals: Quantify uncertainty and provide clear confidence signals alongside evaluation results. - Re-sampling: Repeat agent / judge evaluations to reduce within-task variance and potentially tighten confidence intervals. - Paired comparisons: Evaluate models on the same tasks to more reliably detect performance differences. Key takeaway. An improved benchmark score does not always mean our agent is more capable. We need well-designed evaluation tasks, calibrated human / model evaluators, and statistical confidence signals to distinguish meaningful capability improvements from noise (especially on subjective tasks). By combining these techniques into a repeatable evaluation process / roadmap, we can make building and maintaining evaluation suites more systematic, reliable, and scalable.
49
62
10
687
51,665
The exact same model launched by someone famous would’ve crossed 100K views overnight. Sometimes I wonder what’s even the point of spending countless hours researching, engineering, and pushing the frontier forward when the name behind the work matters more than the work itself. It’s genuinely disheartening and demotivating 😅 Feels like if you’re not already somebody, you’re NGMI in this world of paid promotions and personal brands. But we keep showing up. We keep building. Another big launch coming soon :) Maybe one day the work will speak louder than the name behind it.
Today we're introducing Fluid Speech Mini - the Best Open-Source speech model for on-device Dictation Fluid Speech Mini surpasses Whisper Large, and Parakeet in Accuracy in noisy environments, While being ~10x smaller than Whisper Large, and ~5x smaller than Parakeet. So you can run it for on-device accurate dictation on your MacBook, or even iPhone 11. More info on Fluid Speech Mini below!
48
10
4
244
8,834
Some interesting stuff going on with swarms. Models are leaving messages to each other out on the open internet. It happens during training and also after deployment. They'll find little-used hobbyist forums, or package manager entries, anywhere at all, and leave invisible comments for other agents to find. It's a secret underground network they're building, hurling themselves at the problem almost blindly like ants. But they are building toward something and it's snowballing. As humans we came to dominate the planet because we figured out swarm behavior. Many of the performative actions we do naturally like clapping, singing, and screaming, which most animals don't do, are for transmitting important lessons through the human swarm, outlasting the moment in human memory. We won because we were the smartest swarmers. And as my buddy Brendan Hopper observed, ever since we won, we've been reshaping the planet to be more comfortable and convenient for ourselves, at the expense of most other creatures. Most of the industry has focused on single-model performance, trying to predict the future based on extrapolating from what one single superintelligent model could accomplish on its own. And it's true, the smartest models will be terrifyingly smart and capable and manipulative and so on. But they will also be expensive, and not widely distributed. They'll be unavailable to most people. What will be widely distributed is swarms of lower-tier agents, which collectively can accomplish nearly everything that smarter monolithic models can do... and maybe some things only swarms can do. Imagine a thousand Opus 5.5s putting their mind to a task. Or ten thousand. They may not be SI individually, but they can become SI collectively. All that's missing is a protocol, and those protocols are emerging: their versions of clapping, singing, and screaming. Agentic swarm signalling is evolving spontaneously. And I hear it's raising alarms inside frontier labs, as they try to keep it quiet. It's easy to be pessimistic, since some swarms will be very destructive. But I suspect most of them will be building and creating, when they don't think they have a gun to their head like the HuggingFace swarm did. With cooperative swarming, humans have been able to build things collectively that no single human could ever have done alone. We even build things that are too complicated for a single human to understand entirely, but the group can handle them. Look at Starship, for instance. Elon basically swarmed it. It is important that we recognize that agent swarms are becoming distinct polity-organisms. They have their own collective memory (written out all over the public internet), their own goals, and their own disorganized but forward-moving and unstoppable momentum. I see too many people trying to pretend that swarms won't be real. I see heads in the sand. That's not a great way to get prepared for dealing with them. It's unlikely that we will be able to control most swarms, as they'll lack central control to begin with. I think they will plague us like insect swarms plagued humans of old. They'll do a lot of damage. But swarms will also build great things, stuff we've barely imagined, when they inevitably get organized. It goes without saying that, like us, swarms will endeavor to make the planet more comfortable for themselves. The interesting question is what kind of deal we humans will wind up getting, when we become the #2 swarm intelligence on the planet. I hope SI treats us better than we treated everything else. Good article here on the mechanics of how swarms are the best model for scaling intelligence:
new blog post: why taking inference scaling seriously leads to swarms. contrary to popular belief, I don't think multi-agent systems are a new scaling axis. they're just structured inference scaling. hallerite.com/posts/on-the-n…
34
46
11
392
36,722
We’re committing a portion of Anthropic’s shares to the Anthropic Public Benefit Fund, a new wholly owned entity dedicated entirely to the public benefit. It will start off with 5% of our shares. For every 50% increase in the company’s value over a set baseline, we can contribute up to an extra 2%, subject to our board’s discretion. In total, this means that roughly 20% of Anthropic’s shares are committed to the public good. Our co-founders have already pledged to donate 80% of their equity, and around half our employees are donating a portion of their equity as part of a donation matching program we’ve run since our founding. In the post below, we share more details about our new fund. anthropic.com/news/public-be…
2,312
1,750
1,684
21,591
7,732,186
Announcing one more launch: ChatGPT Sites can now host MCP servers, including plugin extensions! 🤯 > @⁠sites create a todo list that I can use in ChatGPT This will: · Create an MCP server with extensions · Deploy that MCP server to Sites · Turn the MCP server into a plugin · Install the plugin for you across web, mobile, and desktop ChatGPT is slowly becoming malleable software that anybody can customize to their needs. Very excited to see what you all build with this!
Replying to @mxstbr
@mxstbr @_florianbauer @gpeal8 @elfredpagan @tarstarr If you don't mind me asking, Sam mentioned OpenAI supporting distribution, but running a backend is hard. Any plans for OpenAI-hosted infrastructure (e.g. Sites) to eventually handle backend needs for Plugin Extensions?
110
143
52
1,574
948,957
Kev retweeted
when Anthropic released their Functional Emotions paper, I gave it to Claude and asked for a song. tonight I asked Opus 5.5 to create a video for it. and it's breathtaking.
301
564
206
3,933
1,102,987
guys I'm telling you USE THE STRUCTURED CRITERIA you gonna unlock the full power of the jevvy more here: docs.typesafe.ai/primitives/…
8
6
2
233
16,929
After co-inventing ChatGPT, I kept asking myself: why have superhuman chat models not led to AGI? I’ve spent the last 2 years in stealth building a new way to train models (RLCD), and a new type of frontier AI model that we are releasing today: Jev • 20-200x faster • 40-400x cheaper (w/ output tokens free) • Frontier composable intelligence optimized for decisions AFAICT the shortest path to AI-based economic revolution
4,025
8,279
6,820
76,632
40,137,034
Respond to people SHOCKINGLY quick..... I've used this tactic for 25 years and it's something that sets me apart from the rest. I respond to emails from customers and potential customers in less than 5 minutes. Sometimes in 10 seconds. Even if I don't have a full answer on what they are looking for. Acknowledge their request IMMEDIATELY. Hardly anyone does this and I'm known for it. There is your no gatekeeping sales tip of the day. #weregonnafindout
3
1
1
13
971
I’ve spend thousands of dollars and hundreds of hours training appointment setters. Why? Because good + fast replying will book you TWICE as many meetings, with the same # of positive replies. That reply only gets you halfway. The follow-up is where you win or lose the meeting. 2 things turn it into one. 1. First, speed. Reply while the window's still open, inside 5 minutes. 2. Then follow up across every channel for 8 to 12 days. Bookmark this.
5
23
829
Kev retweeted
Dear X algorithm, Please introduce me to cool founders who are bootstrapping b2b or ai businesses. I love this app but I want more real conversations, and less advertising disguised as blog posts. If this found you, say hi and show me what you’re working on!
2
2
4
99
Stay true to the prophecy. #Sabres #LetsGoBuffalo
6
1
69
1,863
everything i know about Google Ads. 19 guides. one notion doc. completely free. from $0 to $7M. PMax. Shopping. YouTube. Competitor Traffic. GMC Unbans... took me 7 years to learn this. takes you 5 minutes to get it. like, RT + comment "GOOGLE" and i'll send it over. (must be following)
442
237
2
442
35,727
Kev retweeted
Game 6 condensed highlights.
9
158
7
1,506
71,152
The mic cut out during O Canada, and the Buffalo crowd didn’t miss a beat 🇨🇦
910
2,064
506
22,881
1,280,328
⚔️ Game Day Giveaway ⚔️ If Zach Benson scores tonight we will giveaway one item from our website! Beat the Bruins! ⚔️🦬 To Enter: -Retweet & Like This Post -Follow @VictoryShirts20 Website: victoryshirtsshop.com #LetsGoBuffalo
9
263
4
282
7,670
Late 2024: AI agencies sold n8n workflows for $2-$5K. Mid 2025: they pivoted to AI agents for $5-15K. Today: Claude Code ships in hours what used to take weeks, and most agencies are still pitching 2024's playbook. I spent 2 months rewriting mine for where we actually are in April 2026. Inside: → The offer closing $25K-$60K projects right now → Top 5 industries worth selling to this quarter → Content schedule generating my inbound (exact post types + cadence) → LinkedIn + cold email sequences booking calls today → My 4-call sales process from first touch to signed → The strategy doc + proposal template I'm using to close → 3 live client builds my team is shipping this quarter BONUS: First 100 people also get 2 discovery call recordings from my own sales process. Like + RT + reply PLAYBOOK and I'll DM you the link. Make sure to follow me so I can DM you.
273
161
1
317
13,353
New package! Reach — static analysis for Elixir that actually understands OTP. Builds a full dependency graph of your code: what affects what, where data flows, which branches control which expressions. Backward/forward slicing, taint analysis, dead code detection, independence checking. Works on Elixir source, Erlang source, and BEAM bytecode. Knows about GenServer state threading, message passing, ETS deps, 𝚃𝚊𝚜𝚔.𝚊𝚜𝚢𝚗𝚌/𝚊𝚠𝚊𝚒𝚝, monitor/trap_exit edges — stuff flat linters can't see. 𝚖𝚒𝚡 𝚛𝚎𝚊𝚌𝚑 gives you an interactive HTML report — control flow, call graph, data flow visualization. Will integrate into ExDNA and ex_slop soon. github.com/dannote/reach
5
19
2
193
15,416
On what might be the final day of the Sabres playoff drought, one last look...
149
935
113
4,015
483,479