Zhihao Zhang

2,100+ tokens per second per user on DeepSeek V4.1 Flash, at full model quality, on eight AMD Instinct MI355X GPUs. In our benchmarks, the Lithos engine delivers: → Up to 3.4× faster inference for DeepSeek V4.1 Flash → Up to 2.5× faster inference for Kimi K3 Compared with tested open-source engines on the same eight GPUs. The key challenge is reducing the coordination tax: time spent waiting between kernels and across GPUs. Our approach combines persistent megakernels, overlapping execution, and communication inside computation. Our technical deep dive covers: - Decode speed, cost per token, and latency up to 1M context. - How Lithos engine reduces launch overhead and overlap computation with communication across GPUs. - How MI355X’s register capacity and direct instruction-level control enable hardware-specific optimizations. - What our roofline analysis reveals about opportunities for further speedups. Read the full analysis: lithosai.com/blog/ultra-fast…

DeepSeek-V4.1-Flash-1017 is now live on LithosAI at up to 1,500 tok/s 🚀 Great model upgrade from @deepseek_ai

Optimizing local inference for ultra-low latency is both easier and harder than optimizing for a cloud service. The unavoidable hardware variety, memory constraints, and thermal throttling automatically make each optimization more complicated. But knowing which challenges to tackle first is easier because you’re dealing with users who: 1) know exactly what they’re looking for from inference performance for their setup and 2) are predisposed to build in the open and share their findings Here’s how we know: @JiaZhihao announced that we were open-sourcing lithos-metal, our local inference engine for Apple silicon, 5 days ago. In that time, we’ve made a ton of progress because everyone who let us know they tried lithos-metal (or even just explored the code or blog) gave us the platonic ideal of technical feedback: immediate, specific, actionable. Thanks to all of you who shared your questions and suggestions, including insights into your actual use cases and testing approach. Our day job is still primarily building ultra low-latency agent infrastructure on standard GPUs, but it’s important to us to keep lithos-metal going and build the community. Internally, we talk a lot about taking a co-design approach to AI efficiency. That means simultaneously thinking about how the hardware and software layers impact each other. Only focusing on one area for improvement leaves performance gains on the table. That logic applies to where your inference engine lives, too. Whether it’s in the cloud, on-prem, or on your laptop, we’re excited to keep working to solve agent latency for our users.

LithosAI is pushing the Pareto frontier of agent inference speed, latency, and cost, and our latest results on Artificial Analysis proves it. Unlike most providers, offering more tiers improves our utilization. Read the full story: