MTS @ Datology | @ICMEStanford PhD | @dartmouth
San Francisco, CA
Joined April 2021
- Tweets69
- Following394
- Followers175
- Likes968
don't miss out!
Fellowship applications close in 10 days.
Datology Labs Fellows get funding, compute, and mentorship to spend a semester on the science of data. Paid, San Mateo/NYC, starting January.
Apply by Nov 15: labs.datologyai.com/fellowsh…
Kaleigh Mentzer retweeted
Fireworks serves >40T tokens a day of "specialized models." I'd define this as any post-trained model for a customer.
Kaleigh Mentzer retweeted
1/7 New Labs Research: Follow the Gradient
SafeTulu filters benign-looking data that weakens LLM safety, cutting attack success up to 22% relative while largely preserving capability.
🤗 Space: huggingface.co/spaces/Datolo…
Data: huggingface.co/datasets/Dato…
Paper: datologyai.com/blog/follow-t…
A backtest-able model with real modern synthetic data is huge alpha
Kaleigh Mentzer retweeted
When a model rephrases documents the knowledge cutoff guarantees are lost. It can introduce information its sources never had. This is catastrophic for backtesting. So we built a way to generate synthetic pretraining data with measurable, controllable knowledge cutoff guarantees
Kaleigh Mentzer retweeted
Tomorrow I'll be on Datology's Curation Studio webinar discussing what went into measuring our 6x compute multiplier.
Come ask about scaling ladders, baselines, and quantifying data quality!
10am PT w/ @arimorcos @hurrycane @josh_wills: events.datologyai.com/curati…
guaranteed to be a good watch
Data quality is the ultimate compute multiplier.
Curated data matched our largest baseline with 6x less compute. A 12B-A1.4B model beat 30B-A3B on a fifth of the pretraining compute.
Live Tuesday 10am PT with @hurrycane @lukemerrick_ @josh_wills: events.datologyai.com/curati…
As a researcher, I cannot understate how much easier this determinism makes my life. No more mystery "maybe it's just distributed computing?? 🤷♀️" results from my dataloader
Zephon’s TorchTitan integration preserves identical global batches when training resumes on a different GPU count.
In our latest blog’s 1B-parameter model test, training resumed from a 16-GPU checkpoint on 8, 32, and 64 GPUs, and the SHA-256 hash of every global batch matched the original run.
Zephon tokenizes, packs, shuffles, and mixes data inside the training job, avoiding a preprocessed copy of the corpus.
The data, tokenizer, seed, mixture, context length, and global batch size must stay unchanged when resuming. The lane count must divide evenly across data-parallel replicas, and each optimizer step’s batch must split evenly across lanes. Training losses can still differ because floating-point reduction order depends on GPU count.
✍️ @datologyai @josh_wills @MaxiBoether, Ties Robroek (@datologyai)
Read more: pytorch.org/blog/bringing-el…
Kaleigh Mentzer retweeted
Applications are open for the January cohort of the Datology Labs Fellows Program.
Funding, compute, and mentorship to work on the science of data.
Paid · One semester · San Mateo/NYC
Apply by Nov 15: labs.datologyai.com/fellowsh…
Fresh ideas, mentorship, compute, engineering support - what more could a researcher want!
Kaleigh Mentzer retweeted
"Not all data is equally valuable" - often yapped about, but Datology have done the work to quantify that and put it into in practice
Kaleigh Mentzer retweeted
Tbh I just found out that Thomson-1 beats GPT 5.6 Sol
That is insane!
That’s our curation is allowing a 400B model to beat the best model in the world at the time on it’s specialized domain: legal
Owning your own intelligence is more critical than ever, but harnessing your data to build a model that can beat the frontier has required deep data curation expertise.
We built @datologyai Curation Studio to change that. Curation Studio enables teams of all sophistication levels to customize curation to their use cases, delivering a massive compute multiplier that fundamentally changes the economics of model ownership.
We applied Curation Studio to the best publicly curated datasets from Nvidia, HuggingFace and others, and were able to deliver a 6x compute multiplier out of the box, all through better use of existing data.
This has allowed us to deliver huge wins for our customers. For example, we worked with @thomsonreuters to help them build Thomson-1 for only $450K in compute, which outperformed GPT 5.6 Sol in head-to-head comparisons and did so for 100x lower cost per task.
Curation Studio brings those capabilities to everyone.
This is a big moment for us at Datology, but it's just the beginning. We'll be expanding Curation Studio support to more modalities and verticals. And stay tuned for more research releases coming very soon!
Kaleigh Mentzer retweeted
We believe intelligence should be democratized, not just controlled by one or two labs on the planet, and we're getting there through data.
Kaleigh Mentzer retweeted
Wow this is awesome! If you train & own your own models you guarantee information security and IP priority - while cutting token spend
You get to own your alpha!
@thomsonreuters got frontier legal performance - for ~$500k in compute!?!
Which pharma is first? @EliLillyandCo?
Kaleigh Mentzer retweeted
A data loader is only as useful as the data it loads. This is why this week is so exciting: we're sharing our Curation Studio and how we use it to generate the best tokens (or raw strings to be tokenized and packed on-the-fly) possible. Check out our release announcement!
This result is super impressive ESPECIALLY considering the "baseline" here is crazy sophisticated
Kaleigh Mentzer retweeted
Look at him, that’s my 1000x Engineer
Your what?
My 1000x Engineer! My throughput whisperer. Look at him, you notice anything different about him? Look at his face
I’ll give you a hint, his name is Josh. He put H100 in his name before it was cool. Yeah, I’m sure Zephon scales.
Data Loader (n.): The part of your training stack that is too ML-y for the systems folks and too systems-y for the ML folks.
It's the work I love best, and I had a ton of fun building a new one with the team here at @DatologyAI. My three favorite things about Zephon:
Kaleigh Mentzer retweeted
Data Loader (n.): The part of your training stack that is too ML-y for the systems folks and too systems-y for the ML folks.
It's the work I love best, and I had a ton of fun building a new one with the team here at @DatologyAI. My three favorite things about Zephon:
At @datologyai we run thousands of data experiments. Two things keep biting us:
- Same run on 16 vs 8 GPUs → different evals
- Every pipeline change → re-materialize the dataset
Introducing Zephon, the data loader we built to process data on the fly, deterministically. 🧵