KKoilbase

Qwen3.8-27B reportedly shrank from 55.6GB to just 5.9GB while retaining 98.2% of its aggregate benchmark performance. That’s an approximately 89.4% reduction in model size, according to PrismML, the team behind Bonsai. Here are the reported numbers: Original Qwen3.8-27B: 55.6GB Bonsai 2 compressed version: 5.9GB Aggregate benchmark performance retained: 98.2% Previous Bonsai 27B version: approximately 95% retention Evaluation coverage: 20 benchmarks The compression technology behind this is ternary quantization. Traditional model weights use numerical values represented at different levels of precision. Ternary quantization takes a much more aggressive approach, representing weights using three possible values, typically -1, 0 and +1, alongside scaling factors. This drastically reduces the number of bits needed to store the model. But shrinking weights is only half the challenge. The real difficulty is preserving the information they contain. Push quantization too far and the model can lose important distinctions that affect reasoning, coding, instruction following and other tasks. Bonsai 2 reportedly improves on the previous version in areas such as: Coding agents Multimodal reasoning Long-horizon tool use The agentic capabilities are particularly interesting. An AI coding agent might need to inspect an entire codebase, understand dependencies, edit several files, execute tests, interpret errors and repeat the process until the task is complete. Maintaining those abilities after such aggressive compression is more significant than simply preserving performance on short question-and-answer benchmarks. Why this matters for local AI A 5.9GB model is much easier to distribute and store than a 55.6GB version. It could make larger models more accessible to people running local inference on consumer GPUs, laptops and other memory-constrained systems. There is an important distinction, though: 5.9GB is the reported model size, not the total memory required to run it. Runtime memory also depends on the KV cache, context length, activations and inference-engine overhead. The 98.2% figure also represents aggregate performance across 20 benchmarks. It doesn’t mean the compressed model performs identically on every task. Some capabilities may retain more accuracy than others, which is why independent evaluations matter. Still, moving from approximately 95% retention in the previous Bonsai version to 98.2% with Bonsai 2 is a notable reported improvement. If independent testing confirms these results, this could make models that previously required substantial hardware much more practical to run locally. The real test for compression is not how small the model becomes. It’s how much of its useful capability survives the reduction.

Qwen3.8-27B reportedly reached 381 tokens/sec on a single RTX 3090 with 24GB of VRAM. The same GPU and model previously ran at around 82 tokens/sec. That is approximately 4.6× higher throughput without upgrading the hardware or replacing the model weights. Developer TeksEdge achieved the reported result by changing how inference is performed, combining speculative decoding, DFlash2, context lookup and optimized verification. Here is the progression: Baseline: 82 tok/s Optimized MTP: 114 tok/s DFlash2 + context lookup: 138 tok/s Optimized lookup verification: 381 tok/s The jump from 138 to 381 tok/s is particularly interesting. It suggests that the cost of verifying proposed tokens was a major bottleneck in this particular setup. The model itself did not suddenly become more efficient at reasoning. The inference pipeline became better at producing and validating output without paying the full cost of ordinary token-by-token decoding for every token. Here is how the pieces fit together. 1. Speculative decoding reduces sequential generation Most language models generate text autoregressively. Each new token depends on the preceding tokens, so ordinary decoding requires a sequence of dependent operations. Speculative decoding changes that workflow. A faster draft mechanism proposes several tokens, and the main model verifies those proposals. If the predictions are accepted, the system can advance through multiple tokens during one verification step. This reduces the number of sequential generation steps needed to produce the same output. DFlash2 is designed to improve this process by making speculative decoding more effective. However, the benefit depends on draft acceptance, verification overhead and the workload. A draft mechanism that proposes useful tokens consistently can deliver substantial gains. One that frequently produces rejected tokens may deliver much less. 2. Context lookup reuses sequences already present in the input Some tasks require the model to reproduce material it has already seen. Think about copying a document, transforming structured text or generating code that contains long sequences matching the supplied context. Instead of predicting every token individually, context lookup searches for reusable sequences in the existing input and proposes matching text for verification. TeksEdge reports processing blocks of up to 16 tokens and accepting roughly 15 tokens per block in its setup. On a document reproduction task involving 25,000 tokens, the reported throughput increased from approximately 260 to 382 tok/s. That result makes sense for a workload with substantial input-output overlap. The engine has opportunities to reuse existing sequences rather than generate them entirely from scratch. Open-ended conversation is different. New explanations, unfamiliar code and original reasoning may contain fewer sequences that can be copied directly from the context. Consequently, context lookup can be extremely effective on some tasks without delivering the same gains on every coding or chat workload. 3. Verification efficiency can determine the final speed Generating candidate tokens is only half the job. The inference engine still needs to verify them against the main model. If that verification process is expensive, the system can lose much of the advantage gained from proposing multiple tokens at once. TeksEdge’s reported progression from 138 to 381 tok/s after optimizing lookup verification points to the importance of this stage. It also illustrates why inference optimization requires attention to the entire pipeline. Faster draft generation alone is not enough if verification becomes the next bottleneck. The reported stages represent successive configurations. Their individual throughput figures should not be treated as independent multipliers that can simply be combined. 4. Prefix caching addresses a different bottleneck