Co-Founder / CTO @spiraldb | Creator of @vortexdotdev
London, England
Joined February 2021
- Tweets61
- Following361
- Followers289
- Likes127
This is exactly why Spiral is focusing on AI data infrastructure. Training on the right data is a surprisingly outsized lever
Pretraining progress seems to be coming mostly from data improvements.
@who_is_jerbear and I pretrained combinations of year-representative open model recipes and data corpuses across 2019 to 2025 at various small scales.
Data improvements contributed 3.24x as many compute multipliers as model improvements did (12.0x vs 3.7x).
And the gains stack independently - a better dataset helps every architecture about equally, and vice versa.
Here are full results, plus what we think this means for the future of AI progress:
dwarkesh.com/p/pretraining-p…
You heard it here first folks - @vortexdotdev is the solution to the world's hardest problems
The group produced an analytical proof and Lean formalization that via Navier-Stokes dynamics a fluid can develop a singularity in finite time.
The solution is a vortex, a spinning swirl of fluid, that spirals inward and gets increasingly elongated, like spaghetti.
cdn.openai.com/pdf/32d9f210-…
I thought I'd test out my theory by vibe coding a Vortex plugin to read Parquet files...
It works!
I’ve written enough file formats to know the bytes are the easy part.
I/O, pruning, scheduling, pushdown eat the weekends.
I built Vortex so I never have to do this again!
spiraldb.com/blog/vortex-the…
I’ve written enough file formats to know the bytes are the easy part.
I/O, pruning, scheduling, pushdown eat the weekends.
I built Vortex so I never have to do this again!
spiraldb.com/blog/vortex-the…
Write the pipeline you want. Let Spiral choose the physical plan.
Materialize RGB, keep native YUV, or fuse preprocessing into the model. Let the optimizer decide.
See how @SpiralDB avoids the 9× footgun:
spiraldb.com/blog/when-video…
Fusion doesn’t always win though. At 64 output channels, the matched benefit falls to 1.2×: the fused schedule repeats preprocessing, while materialized RGB provides reuse.
We ran the 4K, 3→8 first operator three ways on an NVIDIA H100 80GB HBM3:
composed: 0.499 ms
same conv, split: 0.107 ms
same conv, fused: 0.056 ms
The complete physical plans are 8.9× apart.
At 4K, the normalized RGB FP16 tensor is 47.5 MiB.
Preprocessing writes it. The first model operator reads it again.
That is a 94.9 MiB boundary.
But the tensor APIs also fix the physical plan:
NVDEC → NV12 → RGB8
→ resize + FP16 + normalize → RGB FP16
→ first model operator
The decoder and preprocessing each materialize an RGB tensor for the next stage.
Four innocent lines of PyTorch hide a video preprocessing footgun.
In our H100 benchmark, the resulting physical plan made the model’s first operator run 8.9× slower.
The code is clean. The physical plan isn’t. 🧵
Ohhhhh, they were running the frontend off an OLAP database. Makes sense now.
We launched Rippling Data Cloud today - an all-in-one rebuild of the modern data stack, with AI deeply integrated throughout.
Why would you want an org-and-employee-centric data stack? Well, here’s how I used Rippling Data Cloud to help with token burn and cut AI slop. 1/
xitter.cf/parkerconrad/status/20…
SmithDB is the perfect example of how far performance can be pushed by having full control over the storage layer. The DataFusion + @vortexdotdev stack seems to be emerging as THE way to build next generation databases.
#ParquetIsForFloors
We built SmithDB: the database purpose built for agent observability workloads that now powers many parts of LangSmith.
Agent observability presents a challenging data problem. Agent traces can contain tens of thousands of intermediate spans and large, unbounded payloads. These characteristics are a direct result of agents running for longer time horizons and LLM context window sizes growing.
Traditional data infrastructure was not built to handle the complexities associated with storing and querying this data.
SmithDB brings LangSmith up to 12x performance improvements across access patterns most important for agent observability. I’ve been working on SmithDB directly with an amazing team over the past few months, and I’m incredibly proud of the results we’re seeing.
I wrote a bit more about the story and engineering challenges behind SmithDB in this blog.
Additionally, if you’re a systems engineer interested in building the future of agent observability, please reach out!
Nicholas Gates retweeted
Love seeing Naomi Osaka honor the CLRS Algorithms textbook at this year's Met Gala
Vortex is BOTH 38% smaller and has 10–25x faster scans than Parquet+ZStd for TPC-H SF10
We implemented BtrBlocks-style cascading compression in Vortex that recursively tries codecs like ALP, FSST, and bit-packing, letting the data pick the best encoding
spiraldb.com/post/cascading-…
