The hardest thing about building @harvey is doing what’s best for our customers despite immense pressure to do what’s easy. The easy thing would have been to force our customers onto consumption pricing before they were ready and serve them worse models to protect our margins. We chose to help our customers transition on a timeline that works for them and give them the best models in the meantime, even though it hurt our margins. This meant optimizing our product through routing, harness improvements, and post-training so we could serve frontier intelligence at an affordable price. It also meant building the infrastructure for customers to monitor and manage spend: usage dashboards, per-matter cost attribution, spend caps, and ROI reporting. As a result, we improved our gross margins from -50% to positive in a single quarter despite usage doubling month over month and continuing to serve the best models. Our philosophy is simple: do what’s best for our customers, even when it’s painful, hurts our margins, or draws criticism from competitors, X, and the press. We believe the most important part of building a company is earning and keeping your customers’ trust. You do that by doing the hard thing for them, even when it costs you.

In July, 20% of our inference spend was review tables. The most expensive review table queries cost $20k. Today we’re excited to share work we did with @appliedcompute that will reduce this cost by 50% by post training a review table model. Our review table product allows layers to upload thousands of contacts or emails and extract or analyze flexible fields. Historically we optimized this system via prompt caching, packing (multiple fields per model call), routing, batching, etc. Once we hit the limits of gradient free approaches we wanted to see if post training could improve on an already strong baseline. Review tables turned out to be a perfect candidate for post training because verification is easy and we’ve built large datasets to do so. Awesome work by @vtrengarajan, our vault and labs team as well as applied compute. Detailed write up below.