aggressively quantized pretrain is exciting news!
Today, in the first of a series of blog posts we’re calling “Behind the Grove”, we’re offering a look into some of the technical work we do at DeepGrove.
Our first post is about our pretraining infrastructure, Grove. Grove is built around 2 backends: Grove-FSDP and DDP. We detail how we optimized both backends, culminating in an over 2x speedup over public baselines.
A production-scale pretraining run of Maple 20B-A1B (deepgrove.ai/maple-preview) on 4T tokens takes a mere 2 weeks on 64 H100s.
We view infrastructure as the backbone of our work, and as one of the many pieces that help us build toward the frontier of efficiency and performance.
To read the full post, visit deepgrove.ai/infra.
Introducing Flash-MSA, the world’s first open source sparse attention training kernels optimized for extreme context lengths. 4 simple kernels, 400%+ speedup over dense flash attention at long context ⚡
To verify kernel correctness, I ran sweeps of comparisons to eager MSA, for example this table sweeps over Q/B/S in bf16. I also implement Flash-MSA to be flexible- it supports various main and proxy attention configs, variable top-k, and the proxy dense warmup phase.
9/
blog: nanduruganesh.github.io/flas…
code: github.com/nanduruganesh/fla…
very open to collab on flash-msa. I’m excited to see what people do with it, and I hope it encourages folks to bring their wild architecture ideas into the open, where the community is eager to optimize them 🚀
10/10





