Today, we're proud to announce @inferact, a startup founded by creators and core maintainers of @vllm_project, the most popular open-source LLM inference engine. Our mission is to grow vLLM as the world's AI inference engine and accelerate AI progress by making inference cheaper and faster. The Challenge Inference is not solved. It's getting harder. Models grow larger. New architectures proliferate: mixture-of-experts, multimodal, agentic. Every breakthrough demands new infrastructure. Meanwhile, hardware fragments: more accelerators, more programming models, and more combinations to optimize. The capability gap between models and the systems that serve them is widening. Left this way, the most capable models remain bottlenecked and with full scope of their capabilities accessible only to those who can build custom infrastructure. Close the gap, and we unlock new possibilities. And the problem is growing. Inference is shifting from a fraction of compute to the majority: test-time compute, RL training loops, synthetic data. We see a future where serving AI becomes effortless. Today, deploying a frontier model at scale requires a dedicated infrastructure team. Tomorrow, it should be as simple as spinning up a serverless database. The complexity doesn't disappear; it gets absorbed into the infrastructure we're building. Why Us vLLM sits at the intersection of models and hardware: a position that took years to build. When model vendors ship new architectures, they work with us to ensure day-zero support. When hardware vendors develop new silicon, they integrate with vLLM. When teams deploy at scale, they run vLLM, from frontier labs to hyperscalers to startups serving millions of users. Today, vLLM supports 500+ model architectures, runs on 200+ accelerator types, and powers inference at global scale. This ecosystem, built with 2,000+ contributors, is our foundation. We've been stewards of this engine since its first commit. We know it inside out. We deployed it at frontier scale—in research and in production. Open Source vLLM was built in the open. That's not changing. Inferact exists to supercharge vLLM adoption. The optimizations we develop flow back to the community. We plan to push vLLM's performance further, deepen support for emerging model architectures, and expand coverage across frontier hardware. The AI industry needs inference infrastructure that isn't locked behind proprietary walls. Join Us Through the open source community, we are fortunate to work with some of the best people we know. For @inferact, we're hiring engineers and researchers to work at the frontier of inference, where models meet hardware at scale. Come build with us. We're fortunate to be supported by investors who share our vision, including @a16z and @lightspeedvp who led our $150M seed, as well as @sequoia, @AltimeterCap, @Redpoint, @ZhenFund, The House Fund, @strikervp, @LaudeVentures, and @databricks. - @woosuk_k, @simon_mo_, @KaichaoYou, @rogerw0108, @istoica05 and the rest of the founding team

Great work! FWIW, vLLM is moving in the same direction. We've put a lot of work into isolating model-specific code, and it works really nicely now. We've also been considering moving more of the stack to Rust, and your work makes a strong case for it!

FIFY

VLLM and SGLang both just said they match DeepSeek's new DeepInfer engine. Here's what you need to know. DeepSeek open-sourced DeepInfer on Oct 20 under Apache 2.0. It is the in-house inference engine behind DeepSeek-V4.1-Flash, and it supports NVIDIA GPUs and Huawei chips. It was originally built on vLLM. Now the two big open serving engines are claiming parity with it. vLLM posted first on Oct 25. It says it matches DeepInfer on DeepSeek's decode configuration across batch sizes 8 to 256. It does that at DEP32, against DeepInfer's EP64DP64. vLLM did it by integrating DeepSeek's open-source kernels and heavily optimizing prefill and decode. The work was led by the Inferact team, with help from NVIDIA and SemiAnalysis' AgentX benchmark. SGLang says it matches DeepInfer on both prefill and decode. It first brought in DeepSeek's Mega kernels (MegaMoE, MegaAttention, Mega-mHC, MegaGate), SWA bounded replay and Engram optimizations. Then it added more kernel fusions and breakable CUDA graphs for prefill. Everything is in SGLang main, with an end-to-end reproduction guide. SGLang also points out gaps in DeepInfer's release. It leaves out a KV cache system, prefill/decode disaggregation and load balancing. SGLang ships all three. Its team also says some prefill comparisons mix GB300 results with DeepInfer's published B300 numbers, and argues that makes a fair comparison hard. Key numbers: - vLLM decode: matches DeepInfer at batch sizes 8 to 256, using DEP32 vs EP64DP64 - vLLM on a GB300 NVL72 rack: 954K tokens/s per GPU, 196 tokens/s P90 per user, 7.9x its single-node result - SGLang prefill: up to 1.04x throughput on 4x GB300 and 4x B300 - SGLang decode: up to 7.4% more throughput on 64x GB300 The result is that DeepSeek's release set a performance bar, and the two main open engines reached it within days.

vLLM matches DeepInfer, DeepSeek's own engine for DeepSeek-V4.1-Flash, on DeepSeek's decode configuration across batch sizes 8 to 256, and does it at DEP32 against DeepInfer’s EP64DP64. DeepInfer was originally built on vLLM. By integrating DeepSeek's open-source kernels and extensively optimizing prefill and decode, vLLM now matches DeepInfer's performance curves.

We just had a really tough inference bug that 4x parallel Opus 5.5 agents couldn't figure it out after more than 24 hours. Our Agent V's agent collaboration infra running 16 self-hosted Kimi K3 agents cracked it under 12 hours. Definitely a win for open intelligence!

Thanks Inferact Day 0 support! xitter.cf/inferact/status/211343…

Today, we are announcing a strategic partnership with @MiniMax_AI to advance frontier production inference for open-source models. The Inferact Platform has served MiniMax M3.1 Flash in production since the day it launched. Requests through the MiniMax platform are routed to our stack alongside MiniMax’s own capacity, after we met MiniMax’s vendor verification requirements for accuracy and reliability. The Inferact Platform serves this agentic workload with guaranteed SLAs and auto-tunes the endpoint against live traffic, while MiniMax keeps control of how its model is served. Inferact also leads day-0 @vllm_project support for each MiniMax release, so the model serves at production quality the day the weights ship. The engineering from both efforts is upstreamed to vLLM. Thanks to the MiniMax team for the partnership. Read the full announcement: inferact.ai/news/minimax-par…

🚨🚨🚨 vLLM is all you need to reproduce DeepSeek’s performance. With Wide-EP and EPLB, vLLM can match DeepInfer’s decode performance within 6% and generate $39/gpu/hr in revenue at DeepSeek’s off-peak pricing, at 200 P90 TPS on the AgentX workload. Specifically: • Under DeepSeek’s decode configuration (DeepSeek-V4.1-Flash, 131K context, nextn=3), vLLM on GB200 achieves throughput within 6% of DeepInfer’s published results across batch sizes 8–256. • On AgentX, vLLM pushes the Pareto frontier further on GB300, reaching up to 1M tokens/s/GPU and generating $38.96/gpu/hr in revenue at 196 P90 TPS, based on DeepSeek’s off-peak API pricing. • Also, vLLM extended low-latency kernel support for DeepSeek-V4.1-Flash, achieving up to 826 TPS per user on AgentX.

Our Kimi K3 result running @vllm_project TPU v7 reaches 391 tokens/s per user on @SemiAnalysis_ AgentX. This result uses our TPU megakernels in vLLM TorchTPU and holds up on real, long-context, production-level agentic workloads. The results are publicly viewable on InferenceX, and the megakernels are open source. 1/2

Grateful to @deepseek_ai for releasing DeepInfer, initially built primarily on @vllm_project, under Apache 2.0 and crediting the open-source community! We've worked hard on DeepSeek V4.1 Flash inference efficiency and will keep improving with ideas from DeepInfer! Stay tuned 🙌

Inferact is an API launch partner for MiniMax M3.1 Flash! 🎉 We're working with @MiniMax_AI on production inference for its latest model, built for multi-step automation, real-codebase engineering and visual reasoning. M3.1 Flash scored 29.6 on Agents’ Last Exam (ALE), alongside 72.5% on AutomationBench-AA and 73.67% on DeepSWE—capabilities that matter when agents need to plan work, use tools and carry tasks through. Our contribution is the serving stack: bringing our vLLM engineering and the Inferact Platform to the collaboration. Thanks to the MiniMax team for the partnership. Explore the model and API integration: platform.minimax.io/docs/gui…