(follow @thinkinstant instead)

A single RTX 3090 just pushed Qwen3.8-27B to around 381 tok/s. 24GB of VRAM. And no, this isn’t a normal-chat benchmark. The crazy number comes from changing how speculative decoding works. The progression is already impressive: ~82 tok/s: initial single-user setup ~114 tok/s: optimized MTP ~138 tok/s: DFlash2 + lookup drafting ~381 tok/s: longer verification blocks + context lookup The hardware stayed the same: 1x RTX 3090 24GB VRAM ~250W The stack is doing most of the heavy lifting: Qwen3.8-27B, optimized vLLM, DFlash2 speculative decoding, lookup-augmented drafting, prefix caching, quantized KV cache, heads and activations, plus 16-token verification blocks. Here’s the clever part. DFlash2 normally proposes around 7 tokens for the target model to verify. But the developer realized the verification block doesn’t necessarily need to stop there. If the model is answering from information already sitting inside the prompt, the system can use that context to help fill the remaining draft positions. Instead of verifying roughly 7 tokens, the target model can verify up to 16 at once. On a roughly 25K-token document reproduction task: DFlash2 alone: ~260 tok/s Longer verification + context lookup: ~382 tok/s The reported acceptance rate was around 15 out of 16 tokens per verification step. That’s the real explanation behind the 381 tok/s number. The model isn’t magically generating 381 original tokens every second on arbitrary prompts. The system is finding ways to predict tokens from information it already has, then getting the target model to verify large blocks efficiently. And the distinction matters. On ordinary real-world chat prompts, the same setup reportedly reaches around 133 tok/s. That’s still extremely fast for a dense 27B model running on a single RTX 3090. The ~381 tok/s mode is particularly suited to workloads where the answer is heavily grounded in material already present in context: RAG and document Q&A Coding agents applying edits Document rewriting Quoting existing material Information extraction from long prompts There’s another optimization that makes this even more useful for RAG: prefix caching. Ask a second question about the same 25K-token document and reported TTFT drops from: 22.4 seconds → 0.56 seconds The reason is straightforward. The model doesn’t need to process the entire document from scratch again. The cached prefix can be reused. So this isn’t really a “381 tok/s chatbot” story. It’s a story about designing an inference system around the workload. The developer combined speculative decoding, context lookup, larger verification blocks and prefix caching to exploit a simple property: The answer is often already contained in the context. That turns the context itself into part of the drafting system. And that’s a fascinating direction for local inference. It comes from making the GPU compute less.

This is what 12GB of VRAM can build in 2026. An RTX 3060 with 12GB of VRAM just spent five hours building an entire playable game locally. The setup: RTX 3060 12GB Bonsai 2 27B, based on Qwen3.8-27B 5.95GB weights Hermes Agent 125K context ~50 tok/s fresh ~22 tok/s average By the end of the session, the agent had generated: 328K tokens 8 JavaScript files 2,368 lines of code A complete playable game Zero handwritten code The 5.95GB weight file is possible because Bonsai 2 uses aggressive ternary compression to squeeze a 27B-class model into a fraction of the memory normally associated with models this size. But the model size isn’t actually the most interesting part. Generating 2,000 lines of JavaScript isn’t particularly impressive anymore. The hard part is maintaining a software project. The agent has to create files, remember what it wrote, inspect its own work, modify existing code, fix mistakes and keep making progress without constantly losing the thread. That’s where the 125K context window becomes important. A long coding session can easily accumulate thousands of lines of source code, tool outputs, errors, decisions and previous attempts. Once the model loses track of those things, the project starts drifting. In this run, Bonsai 2 was able to maintain enough context to keep working on the same project for hours. And it did all of this locally. No API calls. No cloud coding model. No handwritten implementation. The speed isn’t spectacular either. ~22 tok/s average across a five-hour session isn’t going to beat a high-end cloud setup for raw responsiveness. But that’s not really the use case. A 12GB GPU doesn’t need to compete with a data center. You can start an agent, give it a task, leave it running and come back later to inspect the result. That’s a very different way of thinking about local AI. And the MTP version makes the hardware look even more capable. The original setup was around 26 tok/s. The packaged MTP version reached roughly 50 tok/s fresh, approaching a 2x improvement on the same class of hardware. The MTP head and prefill improvements were released publicly, with the release reportedly passing 4,000 downloads in its first three days. The RTX 3060 is also one of the most common consumer GPUs around. So this isn’t some exotic workstation demonstration. It’s a 12GB gaming card running a 27B-class model with 125K context and a real agentic coding workflow. That’s the part worth paying attention to. Local AI isn’t only about buying more VRAM. It’s increasingly about what developers can squeeze out of the hardware people already have. Better compression. Better inference engines. Better caching. Speculative decoding. Longer context. Better agent harnesses. Put enough of those pieces together and a GPU that was never designed for a 27B model suddenly becomes a machine that can build software for five hours while you do something else.