Pinned Tweet
Three years. Three products. Each one solved the problem the last one created.
Xinference started with a simple frustration: deploying an open-source model privately shouldn't take weeks of infrastructure work. So we built a platform that collapses it to one command. Your cloud, your GPUs, your data — never leaves your environment. Same OpenAI-compatible API. 9.4k+ GitHub stars. 8M+ downloads. Enterprise teams running everything from RAG pipelines to real-time trading agents.
Once teams had private inference running, the next question was obvious: "How do I build things with these models?"
So we built Xagent. Describe what you want an AI agent to do. It builds it. No code, no workflow YAML, no DAG editors. Enterprise-grade agents deployed in minutes on your own infrastructure, powered by the Xinference engine underneath.
Then came the problem I didn't expect.
Teams weren't running one model. They were running ten or twenty — different sizes, different specialties, different cost profiles. Sending everything to the biggest model is expensive. Sending everything to the smallest is unreliable. Hard-coded rules break the moment traffic shifts.
So we built XRouter. Smart model routing that learns which model handles which request most cost-effectively — in real time. Least cost. No quality degradation. Teams are saving another 30–50% on top of the savings from running open-source models privately.
The lesson: the real value in AI infrastructure is compound control. Not one clever feature — but owning the full path from request to response and optimising every layer.
Sovereignty → Automation → Cost intelligence. One stack.
Still a small team but growing fast. Visit us at xinference.co/.
8
832
See how many users one GPU can serve before replies slow down. Run a benchmark from the Xinference Web UI and get throughput, time to first token and time per output token at each concurrency level.
"Can we get results like ChatGPT without sending everything to one AI provider?" It's the question we hear most, so Paul Towers and Joel Brilliant answered it in 18 minutes of plain English. Free on , slides included.
Open-weight models: 19% of enterprise AI usage, just under 5% of LLM API spend (Menlo Ventures, 2026). DeepSeek and GLM, two of the top three by spend, run on the Xinference Model API, hosted in Australia.
Most of what makes LLM serving hard isn't new. Distributed data systems spent a decade on the same mistake: deciding how to split the work before you can see it.
Chris Qin wrote a short book about it. Four ideas from Inference at Scale:
1
502
4. A metric that only feeds a dashboard is observability. A metric that feeds the scheduler is adaptation.
Worth asking which one your serving stack has.
1
11
Nearly 3 in 4 AI tokens in public developer usage data now go to open-weight models, up from about 1 in 4 a year ago (Futurum Research, Oct 2026). Most of the spend still goes to closed models.
📍 Day one at the AI Infrastructure & Enterprise Systems Summit, Sydney. Find the Xinference team in the Grand Ballroom today and tomorrow.
94
📣 Tomorrow and Tuesday at the AI Infrastructure & Enterprise Systems Summit, InterContinental Double Bay. Find the Xinference team in the Grand Ballroom.
100
Thanks to everyone at DevOps Summit NSW 2026 at ICC Sydney. Paul Towers hosted our roundtable on the true cost of running AI in production, and the team ran the booth on the Model API: Kimi K3 and DeepSeek, hosted in Australia.
140
Most model launches go up and to the right. DeepSeek's V4.1 Flash-1017 chart aims top and to the left: pass rate against cost per task. Judge models on what a finished task costs, not price per token.
75
🧩 Next week at the AI Infrastructure & Enterprise Systems Summit: open models like DeepSeek and Kimi behind one API, from per-token pricing to dedicated GPUs. Find us in the Grand Ballroom, InterContinental Double Bay.
76
📍 Live today at DevOps Summit NSW, ICC Sydney. Find us at the Xinference stand, and at 1:10 pm for our roundtable on the true cost of running AI in production.
📣 Tomorrow at DevOps Summit NSW, ICC Sydney. Find Paul, Joel, Andrew and Summer at the Xinference stand, and join our 1:10 pm roundtable on the true cost of running AI in production.
3
99
🎙️ Wednesday at DevOps Summit NSW: when does a managed API stop being the cheaper option? Paul Towers hosts our roundtable on the true cost of running AI in production, 1:10 pm, ICC Sydney.
67

















