90% of AI 'breakthroughs' are slop. Let's build the real 10%.
Bucharest, Romania
Joined December 2021
- Tweets1.9K
- Following339
- Followers203
- Likes730
Pinned Tweet
We benchmarked Surogate's serving engine against vLLM and llama.cpp. Same hardware, published methods, losses included.
Best case: 7.0x llama.cpp on a 200 GB MoE across 8 RTX 5090s. Worst case: we lose. Details below.
80.5 on Terminal-Bench 2.1. Dense 65B.
Ripple-65B Preview is out from Ricursive: Apache 2.0, 12T tokens, 1M context. Their table has it edging Beam (80.1) and beating Muse Spark Open (76.4), Inkling (63.8), Nemotron 3 Ultra (56.4) on that bench among U.S. open weights.
Caveat: lab comparisons, 0.4-point lead over Beam. DeepSWE: 44.3 vs Muse Spark Open's 60.0.
Pull it today for coding agents, or wait for a third-party run?
87 ms.
That's how long the engine pauses to swap a 2.5 GB rank-16 LoRA under eight live generations.
No restart. Staging that adapter took ~16 s in host RAM while traffic kept the current weights. In-flight requests continue on the new policy from their next token.
Do you drain adapters between RL steps, or swap them under traffic?
In-place swap details (staging, pause times, lora_versions for RL clients): github.com/invergent-ai/suro…
802 vs 346 tok/s.
Qwen3.5-0.8B GGUF Q4_K_M, 1 user decoding, one RTX 5090 at a 400 W limit. Surogate vs vLLM (NVFP4): 2.32x.
Where vLLM wins: H100 Qwen3-8B FP8, 32 users, 2,048-token prompts. Prefill 55.1k vs 52.5k tok/s.
Single-user decode still the number people screenshot. Do you trust it for capacity planning?
Full table, test conditions and the rows where we lose: github.com/invergent-ai/suro…
You can keep exactly one open-weight model family for all of 2027. Nothing else.
Qwen, GLM, DeepSeek, Kimi, Gemma, gpt-oss, Mistral?
Mine is Qwen. Not because it tops every leaderboard. Because there's a size for every GPU I own, from a 0.8B up to MoEs that fit on one card.
Which one are you keeping, and why?
2,765 vs 683 tok/s.
Qwen3.5-0.8B GGUF Q4_K_M, 8 users decoding, one RTX 5090 at a 400 W limit. Surogate vs llama.cpp: 4.05x.
Where vLLM wins: H100 Qwen3.6-35B-A3B FP8, 32 users, 2,048-token prompts. Prefill 39.5k vs 35.7k tok/s.
What concurrency do you actually break first at — 1, 8, or 100?
Full table, test conditions and the rows where we lose: github.com/invergent-ai/suro…
Your offloaded training step is slow. Faster kernels won't fix it.
When you stream weights or optimizer state from CPU RAM, the step is often waiting on PCIe, not on compute.
Check first: profile with nsys and look at Host-to-Device memcpy time. If its bandwidth sits near your link's ceiling (we measured ~53 GB/s on PCIe 5.0 x16), you're transfer-bound.
Then faster kernels just turn busy time into idle time. Move fewer bytes, or overlap them better.
What's the last bottleneck you profiled that surprised you?
The profiling commands and the knobs that help (prefetch blocks, resident layers): github.com/invergent-ai/suro…
Your users don't feel your average latency.
They feel the slowest request in a hundred.
That's the one stuck behind a long prompt, a cold cache or a traffic spike. And it's almost never on the launch chart.
What's your p99 time to first token in production? Do you even track it?
Your small router model and your big model don't need two GPUs.
One server, two models, one card:
surogate serve nvidia/Qwen3.6-27B-NVFP4 --served-model-name big --model small=/path/to/cached-small.sinfer
Both show up in /v1/models. Pick one per request with the "model" field.
Out of VRAM? --enable-sleep-mode parks idle models in system RAM and wakes them on the next request.
How many models are you running per GPU right now?
The best open-weight model is 30 points behind the best closed one.
It's also ~13x cheaper.
ArtificialWatch bench index (median of 7 public leaderboards, Nov 3):
Claude Opus 5.5: 98.1, $4/$20 per 1M tokens
DeepSeek V4.1 Flash: 68 (12th of 44), $0.30/$1.20
On coding: 96 vs 75.6.
The top 6 open models are all Chinese.
Which side of that trade are your workloads on?
Moonshot said the Kimi K3.1 Pro weights would land "by Nov 4".
It's Nov 4. At 06:45 UTC, their newest Hugging Face repo is still Kimi K3, last updated Sept 2.
If it ships, the weights alone for a ~3T-parameter model:
FP8: ~3 TB
4-bit: ~1.5 TB
And that's before KV cache for a 1M-token context.
Do the weights drop today? And who's actually planning to run it?
2,607 vs 2,162 tok/s.
Qwen3.6-35B-A3B NVFP4, 100 users decoding, one RTX 5090 at a 400 W limit. Surogate vs vLLM: 1.21x.
Where vLLM wins: Qwen3.8-27B NVFP4, 100 users, 2,048-token prompts, same card. Prefill 12,402 vs 12,006 tok/s.
What concurrency do you actually run at?
Full table, test conditions and the rows where we lose: github.com/invergent-ai/suro…