🎙️ New Deployed episode with @ryancarson, "Builder in Residence" at @AmpCode. It's a great convo about building code agents and building as a solo founder at the same time, and it's refreshingly honest. Case in point, on evals for coding agents: "We do not have a set of evals that we programmatically run to decide if GPT-5 is better than Sonnet-4 yet... everybody in the trenches building these agentic coding tools understands that it is so dynamic, and so frustrating." Great conversation about building with AI, developer experience, and what happens when everyone can code. Check it out. 👇

I had a great conversation with @kellyschaefer on the latest Deployed podcast about how she and her teams build AI products at Google Labs. Things like NotebookLM, Jules, etc. LOTS of good alpha on product strategy, team dynamics, PM career advice... This clip's about getting started with evals. Lots of folks need to hear this. 👇

A few of us from @freeplay_ai are in SF this week for @aiDotEngineer. Come say hi in the expo hall and check out our talks if you're around. 🙌 Both talks are focused on the product and ops realities to ship AI products that work — how to navigate bringing AI products to life in the Fortune 500, and how to run a tight data ops process in sync with engineering to delivery high quality AI products in production. 🎙️ From Hunch to Handoff: How AI PMs Can Help Turn Ideas Into Shippable Features Quickly 📅 Wednesday, June 4 @ 2:40 pm With: Jeremy Silva & Eliza Cabrera 🎙️ The Build-Operate Divide: Bridging Product Vision and AI Operational Reality 📅 Wednesday, June 5 @ 11:00 am With: Jeremy Silva & Chris Hernandez

The next Colorado AI Builders event is happening in Denver on Wednesday June 25th! Sign up now before it books up. 🙌 The last one had over 600 RSVPs for a space that maxed out at 300... Link below for this one. At the last event we got to see: * A preview of the new Jules coding agent from @Google (the week before it was fully released at I/O!) * Amp, the newest coding agent from @Sourcegraph * @zapier Agents * How @Spekit 🐙 is using AI to help sales teams * How Nephrolytics uses AI to improve kidney care This one will feature a mix of builders from startups and big companies, covering a range of topics and types of AI products. All Colorado-based presenters. Huge thanks to our sponsors at @SiliconVlyBank, @Zapier, @Sourcegraph and Technical Integrity for making the event happen.

SaaS is blocked by security. OSS is too much overhead for SRE/DevOps... @freeplay_ai's new Bring Your Own Cloud option is the perfect path for enterprise AI teams: ✅ Full Freeplay stack in your VPC ✅ Zero sensitive data egress ✅ One-line deploy ✅ Automated updates, always in sync Run evals and AI observability safely, without headaches.

🎯 Fresh conversation with Box CTO Ben Kus about building AI agents for the enterprise. Good timing with the @aiDotEngineer Summit on agents this week! Box has 100K+ customers and manages petabytes of unstructured data, and generative AI gave them superpowers to help customers make sense of all that data. They've invested in generative AI search and summary features from early on, and they've been getting into more complex agentic behavior for a while now. Ben shares a bunch of good highlights and practical lessons learned from their experience that will be interesting to Freeplay customers and anyone else building agents at scale: 🎯 How they got started with generative AI, built teams, and decided what to build 🤖 What's going on with agents these days, and what's involved to bring them to true enterprise customers 💬 Moving beyond chat and conversational UX to apps that just do the work for you 🛠️ Foundational tools they've invested in, including the essential role of LLM judges to build high quality products (especially since they never look at customer data) Check out the link below to get the full episode on Spotify, Apple or YouTube.

🎙️Solid new podcast for your weekend! @a_a from @SierraPlatform talks about how they build enterprise agents that actually work. We started Deployed so that other builders could learn from leaders who are building AI systems at scale. Sierra's at the front of the pack with agents and Arya talks about: * Bringing AI Agents to Life With Customers * The Agent Development Stack * Building a Comprehensive Evaluation System * The Essential Role of Domain Experts * Designing Effective Feedback Loops 👇 Check it out, full episode links are here: freeplay.ai/blog/building-en…

🎙️ How does a 13-year-old SaaS company with 12,000 customers transform itself into an AI-native business? @helpscout founder Nick Francis posted today on LinkedIn about how they've done this, including changing their whole pricing model. We recently got to talk about this question and more on the Deployed podcast with Nick and @lacion, Help Scout's VP of Engineering. Check it out! What makes their story fascinating isn't just that they've added AI features – they've completely restructured their business to account for AI capabilities. It's changed how they think about product strategy, how their engineering team works, and what they deliver to customers. Key highlights from our conversation: ✅ The story behind their decision to eliminate seat-based pricing after 13 years ✅ Their journey to build AI expertise within their engineering & product organization ✅ Why they created an "AI Quality Lead" role to bridge domain expertise and AI development ✅ How they shifted engineering culture to embrace the experimental nature of AI products Check out the full episode below. Thanks Nick and Luis for sharing your journey with everyone. 🙌 freeplay.ai/blog/how-help-sc…

The new @Deloitte Tech Trends report is out! And it's all about the impact of AI on business. I'm proud that @freeplay_ai is mentioned in it. 🔥 Every year the Deloitte Office of the CTO publishes the report to help businesses think about the biggest trends that they need to think about in the coming 18-24 months. From their intro, the big idea: "We propose that the future of technology isn’t so much about more AI as it is about ubiquitous AI. We expect that, going forward, AI will become so fundamentally woven into the fabric of our lives that it’s everywhere, and so foundational that we stop noticing it." www2.deloitte.com/us/en/insi…

New episode of Deployed! 🎙️ Kyle Nesbit's been an engineering leader at Google for 17 years. He drops a ton of practical knowledge about how to build AI products that actually work — and how to build engineering teams that consistently deliver better AI products. Example: the need to constantly iterate on evals.

The AI Builders meetup is going on the road! Come join us in LA for Tech Week on 10/14. We'll be hosting with our friends @ThisIsArtium and @SiliconVlyBank. There will be demos from local startups & big companies -- come see what folks are building in LA. 🌴🤖 These events are always packed, RSVP below. lu.ma/bej1uk00

"Just look at the f*@!ing data.” 🔥 On the latest episode of Deployed: The AI Product Podcast, I sit down with my friend & fellow Boulder founder, @vagabondjack. Mike is a co-founder at @brightwaveio. We dig into what it takes to build an AI-powered financial research tool and lessons learned along the way: * LLM limitations (large context windows aren't perfect) * Smart summarization techniques * Creating a useful eval suite * Building AI engineering teams

🧵 Everyone's looking for signal in all the AI noise... There’s no better way to learn than from teams who are already doing it! So today we’re launching our new podcast to tell these stories. 🎙️ Episode 1 of Deployed features our friend, @jeffseibert, CEO & Co-founder of @digits. 🎉 Here's a quick preview: Why it's worth it to review lots of data by hand.

THE Clancy. IYKYK 😉

One of the hardest problems with model-graded evals in AI products is tuning them to agree with human experts. At @freeplay_ai made it radically easier to create custom auto-evals you can trust and tune them over time. Demo below, how it works in thread. 🧵

For all the chatter on Twitter about how "LLM-as-a-judge" evals don't work... We just spent 45 min as a team hand-labeling data to measure alignment with LLM-as-a-judge for a text-to-SQL project. Every time our human reviewers disagreed, it turned out the LLM was right. ✨