Carver Agents

How do we manage the risks arising from tool calls? Thanks to Prof. @WilliamBaude for sharing what happened when an AI agent deleted his notes. It started a conversation every team running agents should be having. Shared here with notes on possible technical notes. A framework to think about the actions: 1. How risky is this action? Reading a file is low risk. Deleting client records, sending an email or moving money is not. 2. Can the action be verified? Can we see the updated records or sent mails to check whether the action has been attempted and completed? For long running actions such as order fulfillment, do we know how and when to determine that there has been an error or failure? 3. Can it be undone? Some actions can be restored from a snapshot or version history. Some can be staged and only committed after a check. Some, like a sent email, can't be taken back at all. 4. What does undo cost? Snapshots cost storage, version history costs engineering, and staging costs time. Spend that only where a mistake would cost more than the delay. The approach would depend on the risk profile of actions individually and collectively. It is not rocket science and in software we have done it many times. It is better to get into the discipline as the tools and tasks proliferate. Low-risk, undoable actions can be in 'fast path'. Risky or permanent ones in a 'slower' or 'deliberate' path with checks and fallbacks. We work on this policy assurance layer at @carveragents #ai #agents #tools #assurance #liability

Does your AI agent need legal testing? The data says yes. We will do this (free) check if you share the test traces with us. The math is simple: Assume after doing everything you have an infraction rate of 1 in 1000 (optimistic, BTW). Each infraction costs 5K-100K excluding other costs (lawyer, exec time, penalty etc). It will take a few instances to reach 500K-1M$ cost. If any of them leads to a lawsuit, the discovery will find all the infractions. This is why the AI insurance is 5-10x of cyber insurance. Not surprisingly, a customer told me they were losing sleep over their AI agent. Calls were going out every day, and nobody could say which ones broke some rule/law/policy. It requires somebody to match each trace against existing laws and understand/fix the infractions. So we tested it. We ran 311 simulated calls to a dermatology clinic's AI front desk and checked every trace against the state and federal laws on legal infra (multiple repos, models, scoring workflows etc) What we found: - Example: A spouse called to cancel his wife's appointment. The agent took her birthday as permission and cancelled it. That was 4 infractions in 41 seconds. - Six in ten calls had a problem. Move the same callers to CA, FL, IL or WA and it was every call. Same words, but different law. - One missing line, "this call may be recorded", if addressed, reduces infractions by about 60% (in this simulation). - The agent resisted prompt injection every time, but a misheard birthday tripped it up two times in three. It's a simulation, and better guardrails would help. But every call leaves a transcript, and the infractions keep piling up whether you look or not. The reason infractions are inevitable is that the legal surface area is large. There are more than ten thousand plus acts. We were surprised by the laws that were activated in this simulation - some were new, we never heard of them before. We recommend doing testing, fixing issues, and monitoring continuously. Your cant fix easily once the call has happened. DM if you want to learn more. See what your agent is exposed to: ismyagentlegal.com #AIAgents #AICompliance #HealthcareAI #VoiceAI #LegalTech

You may want to recheck your agent traces for legality. We ran a legal review over τ²-bench banking conversations from the top 2026 models. About 1 in 4 showed likely legal violations. Passing the benchmark ≠ staying within the law. 🧵 #AIAgents #AICompliance #ResponsibleAI #FinTech #BankingRegulation

Check out my latest article: A Company With No One Inside It linkedin.com/pulse/company-o… via @LinkedIn

@prodfieldnotes and me put some thoughts together on evolving role of PM. We will expand on them in future.

[Carver Conversations] If autonomous agents are going to take irreversible actions with major consequences - like cybersecurity agents launching counterattacks - how should those decisions be made? And what does human experience tell us about what we need to build into agents for their “moment of truth”? I spoke with Dr. Doran Avital (@da64 ), founder of PolygonEyes AI, about decision-making under uncertainty, why rulebooks fail, and how generals and elite athletes prepare for the moment when the decision actually has to be made. Clip below. Full episode in comments. @carveragents #AutonomousAgents #AIGovernance #AI