I like to shrink dem models 🤏 ML/AI Perf @dropbox Prev. Co-Founder & Principal Scientist @mobius_labs (acquired by @dropbox) PhD @inria
Berlin, Germany
Joined November 2023
- Tweets655
- Following123
- Followers733
- Likes1.2K
Confirmed that 3090 unlock is successful
with 360watt power wall, I got 75T -> 135T (measured in Pytorch)
Which is insane
I used github.com/Zagitta/gpu-unloc… with commit a9b0c75
Well well well
🚀New blog: Accelerating Long-Context and Agentic Inference with NVFP4 KV Cache
4-bit KV cache is here! NVFP4 KV in SGLang packs ~1.78× more context into GPU memory and speeds up long-context decode by up to 78%.
Together with @Alibaba_Qwen and @nvidia, we brought NVFP4 KV cache to Blackwell:
- NVFP4 stores KV in just ~56% of FP8's footprint per token
-️ Decode throughput jumps +37% / +58% / +78% at 32K / 160K / 1M context
- Near-lossless accuracy: matches FP8 on GPQA-Diamond & AIME 2025 (Qwen3.5-397B-A17B)
- Higher cache hit rate keeps AgentX throughput scaling where FP8 drops off
The whole recipe combines NVFP4 two-level scaling, in-kernel dequantization on decode, and paged KV cache, and you can enable it in SGLang with a single flag --kv-cache-dtype nvfp4
mobicham retweeted
Today, we’re announcing Ternary Bonsai 2 27B.
Based on Qwen3.8 27B, Bonsai 2 27B is 9x smaller than its full-precision counterpart while retaining 98.2% of its aggregate benchmark performance.
Two months after the first Bonsai 27B release, the biggest change is quality. The footprint remains 5.9 GB, but the gap to full precision has narrowed materially, with particularly strong gains in agentic coding, multimodal reasoning, and long-horizon tool use.
Ternary Bonsai 2 27B is available today under Apache 2.0.
Gemma 4 with image inputs is very slow with the Triton attention backend because it doesn't prune the sliding window.
Small fix, 3-4x faster E2E inference 🫡
github.com/vllm-project/vllm…
mobicham retweeted
Not all work happens in writing. Teams that work with photos, videos, and audio need AI that can search within multimedia content too.
Hear more from Dropbox machine learning engineer Appu Shaji—along with machine learning engineer Hicham Badri—on Working Smarter.
github.com/dropbox/gemlite/t…
You can now easily load various pre-quantized models (block FP8, NVFP4, AWQ/GPTQ/HQQ, certain GGUFs) via a vLLM plugin! You can also run on-the-fly quant as well, easy to use: 1 or 2 flags to enable!
There's a very simply one-shot trick that seems to improve the quality of low-bit weight quantization by quite a bit in some cases: simply reordering the rows.
It doesn't require changing the matmul kernel, only reshuffling the activations.
What kind of FP4 format does the new TPU8 use? MXFP4 quality is pretty poor, and NVFP4 is specific to Nvidia, so I'm guessing it uses a smaller group size (<32) to achieve better quality ? 🤔
Some great on-device multi-vector work at Dropbox, check it out!
Open sourcing something fun from @Dropbox: Witchcraft.
It's a local search engine built in Rust with no API keys or vector DB required.
Think: ColBERT / late interaction style retrieval, but packaged to run locally (perfect for coding agents).
Let's dive in👇
Friendly reminder that QuaRot had a full 4-bit LLM solution back in 2024: weights, activations and KV cache, all in INT4, running on Ampere with int4 mma with actual massive speed-up, before 4-bit KV cache was cool.
