writing as a rehearsal of language. language as a rehearsal of logic. pretraining research
Joined October 2022
- Tweets2.4K
- Following1.1K
- Followers2.2K
- Likes10.5K
Pinned Tweet
I spent the last six months trying to deconstruct Taalas's patents. We think it can be better: 100x fewer memfetches than their bitROM, better software by better quantization than their hardware team could.
So we wrote a compiler that:
> takes a huggingface checkpoint,
> quantizes the model
> descends the weights down metal layers to RTL and GDS, do your DRC, Yosys, PEX, make the electrical waveform execute a mat-vec from your huggingface checkpoint (thanks Cambricon tech papers)
in ~7000 lines of human-readable code.
we're looking for someone with contacts with a foundry/access to 7nm PDKs or contacts at a cheap EuroPTW shuttle? I'm broke and unemployed.
@itsclivetime this is what i think is the future of perplexity per picojoule. @zerohedge @zephyr_z9 you guys wanna see 40,000 tokens/sec?
Progyan retweeted
Mustafa Suleyman’s blast radius continues to increase.
The End of AI Frontiers hunch.net/?p=13764006 .
There's a bit of a train wreck here that's painful to express. Nevertheless, it's a significant event for me and quite a few other folks involved so I wanted to memorialize a bit.
wait till the marketers find a new word for breakthrough to separate their breakthroughs from everyone else's breakthroughs.
"here's your scientific breakthrough of the day! and here's why it's a breakthrough and not some random result!"
Under 3 days! And someone said less than 68 hours for a 600B pretrain? That's incredible, congratulations! It is possible to scale so much further!
plugyawn.com/distributed-tra…
been saying since jan heh
Progyan retweeted
I guess 'tis the day i complain about over-hype on this app.
This does not show a quadrillion param model, nor how to train one.
The claim (which is only in the blog and this thread, the paper is ok) is based on wild extrapolations, but i promise you that the techniques studied in the paper will not continue carrying more than ~one OOM.
In fact you can are already see here (figure) that reality starts detaching from prediction. It will continue detaching more.
(The study is on 25M-400M unique tokens, 30M-400M params)
Also, just because i can't resist sprinkling a little bit of Schmidhubering, how to do many epochs is something vision folks studied a long time, and the "increase wd with epochs" is a finding were published in S4L paper in 2019. It's of course worth revisiting in LLMs but i generally find things carry over.
Humans are ~1,000,000x more sample efficient than LLMs, and there are no good explanations.
We’ve found that a big part of the answer is that LLMs are *massively* underparameterized by many orders of magnitude. The brain has 100T synapses learning from ~200M tokens of language. Chinchilla prescribes a 1T-parameter model on 20T tokens. That’s 7 OOMs apart in parameters per token.
After a lot of painstaking work, we present new data-constrained scaling laws and learning algorithms to scale to (extrapolated) quadrillion-parameter transformers.
w/ @akshayvegesna, @EthanBaron78553, @asatheesh6561
Full paper led by @EthanBaron78553, with @ShmuelBerman, @andrewgwils
Progyan retweeted
totally fraudulent. this is retarded
today, we introduce a little mischief to the world.
for four years, we’ve been brewing real brain cells on electrodes to understand how they compute. now, we’ve turned those discoveries into a video model anyone can use.
from 0.5¢ a second, mischief 1 is at least 5x faster than its base and the cheapest image-to-video model on @fal and @OpenRouter. soon to be on @awscloud marketplace.
it’s not a delusion anymore. biocompute is real, commercial and this is just the beginning.
happy halloween.
Progyan retweeted
This one seems like an obviously helpful no-brainer.
Multi-Token Prediction (MTP) can be done much better.
🚀 Turn out,you can recover the gradient for KL self-distillation to next-token prediction (NTP) for free without ever computing the KL. You just need a one-line code change:
h_ntp += h_mtp.detach() - h_mtp
Soft Multi-Token Prediction is pretty much a free lunch for MTP.
📜 Paper: arxiv.org/abs/2610.38331
💻 Code: github.com/JaydenTeoh/Soft-M…
🤗 HuggingFace: huggingface.co/papers/2610.3…
ALT Soft Multi-Token Prediction Illustration
Progyan retweeted
Multi-Token Prediction (MTP) can be done much better.
🚀 Turn out,you can recover the gradient for KL self-distillation to next-token prediction (NTP) for free without ever computing the KL. You just need a one-line code change:
h_ntp += h_mtp.detach() - h_mtp
Soft Multi-Token Prediction is pretty much a free lunch for MTP.
📜 Paper: arxiv.org/abs/2610.38331
💻 Code: github.com/JaydenTeoh/Soft-M…
🤗 HuggingFace: huggingface.co/papers/2610.3…
ALT Soft Multi-Token Prediction Illustration
Progyan retweeted
Foundation models are trained in stages: pretraining, midtraining, and post-training. But is this separation fundamental?
We introduce unified training: a single algorithm for the full pipeline. This gives
⚙️ Simpler infrastructure
🔬 A framework for better science
(1/n)
Yet another lovely paper.
arxiv.org/abs/2610.38331
Improving MTP by only keeping self-distillation of next token prediction in the loss term.
