llm psychosis 1.0 survivor circa sonnet 3.5

Joined February 2010
Exploring Direct Tensor Manipulation in Language Models: A Case Study in Binary-Level Model Enhancement: areu01or00.github.io/Tensor-…
1
9
2
63
17,768
Pretraining without backprop, 1.58 seconds to 4.2 loss. This one's a tuning win, not a trick every run can use like Drift Leap (shoutout to the LAWA paper). @RobertTLange is right, Track 2 is a systems race now. What I learned that might save others time is in the PR below. But 4.2 is mostly n-gram stats (bigram counts alone get 5.1), so I think Track 2's target needs a relook.
Pretraining without backprop, 14.6 seconds to loss 4.2
2
3
74
3,808
Tensor-Slayer retweeted
0th-order pretraining keeps getting faster. I just opened a new Noise track-2 PR: a 37.7M-parameter transformer trained to val loss 4.2 in 1.66 s on one 8×H100 node, with no backprop anywhere. The current record is 2.46s. Two days ago the track-2 record was 47 s. Then came identity-path credit, moment kernels on tensor cores, structured starts, tail averaging, and a PyTorch port of our JAX trainer, built by a lot of people in the open. This PR doesn't change the algorithm at all. It uses #39's estimator and #38's constants with a faster update: 6.27 → 4.44 ms per update. Val matches #39 within 0.0004 on every seed, so the gain is pure systems. Most changes were worth 1–4%. Together, they cut 40% off #37's update. Where the time went: - The credit kernels and matmuls cost exactly the same as before. - 0.8 ms of eager PyTorch copies and casts were compiled away. - The credit kernels build their features in-kernel, merge their products (14 → 6 per tile), and run every block in one launch: 553 → 349 kernels per update. - Gradients and weight gathers move in bf16, and both all-to-alls run under compute: exposed NCCL is 0.25 → 0.10 ms. - GPU idle time is 0.42 → 0.16 ms. A critical note on the benchmark itself: At 1.66 s, track 2 has become a systems benchmark. The last 30% came from kernel launches, copies and collectives, not from better credit assignment. Each update now does the same math in fewer launches. That's fun engineering, but it says little about whether forward-only learning scales. And 4.2 is a low bar. It is only 0.33 nats past n-gram statistics. Track 1 (lowest val loss within 1 epoch) now edges past the backprop + Adam reference on loss (3.722 vs 3.769), but it needs 82 minutes on 8×H100 to get there. Matching backprop's loss is possible today; matching its efficiency isn't really close (yet?).
Replacing backprop with zeroth-order optimization is inevitable. Introducing Noise: a speedrun for pretraining without backprop. Last week we presented Dust, the first method to approach backprop with SGD for pretraining transformers. We think another 1,000x improvement on top of it is possible, and it will require *co-designing* credit assignment, optimizers, architectures, and eventually, hardware. We built Noise to iterate on this faster.
7
7
83
3,793
Pretraining without backprop, 14.6 seconds to loss 4.2
Replacing backprop with zeroth-order optimization is inevitable. Introducing Noise: a speedrun for pretraining without backprop. Last week we presented Dust, the first method to approach backprop with SGD for pretraining transformers. We think another 1,000x improvement on top of it is possible, and it will require *co-designing* credit assignment, optimizers, architectures, and eventually, hardware. We built Noise to iterate on this faster.
5
12
1
157
12,817
Tensor-Slayer retweeted
Inference Gross Margins - 87.76% to 96.94% Assumptions: 97% cached input, 2% uncached input and 1% output tokens Prefill: 80k uncached input tokens/s/GPU. Decode: 16k output tokens/s/GPU (250 tokens/s/user) Token mix: 970M cached input + 20M uncached input + 10M output per billion total tokens. It takes around 875 GPU seconds (250 for prefill & 625 for decode) to process 20M uncached input + 10M output tokens. Assuming GPU rental cost is around $3-$6/hr, the GPU system cost per billion tokens is around $0.73-$1.46 1B tokens generate $11.91 in revenue at off-peak prices and $23.82 at peak prices At off-peak prices, inference margins are between 87.76%-93.88% depending on the GPU rental cost ($3-$6/hr) At peak prices, inference margins are between 93.88%-96.94% depending on the GPU rental cost ($3-$6/hr)
32
63
24
717
147,238
Tensor-Slayer retweeted
LLM Writing is limited by semantic understanding. I wrote about it in my blog. This is the first essay in a series on LLMs and writing quality. The link is in the next tweet.
3
1
2
24
1,479
Tensor-Slayer retweeted
DeepSeek V4.1 Flash 1017 everyone's talking about it for a reason. It does great with 3d built this honeybee on a sunflower like it was nothing. Used Deepseek Harness, 1 turn, looks so good!
56
54
11
1,203
43,781
Fable 5.5 spotted
2
221
Tensor-Slayer retweeted
i was blown away by the naturalness of bulbul v4 when i first saw the demo at epoch a couple months ago we're launching v4 flash today and bulbul v4 pro to follow soon!
Today, we're introducing Bulbul V4 Flash, our latest text to speech model built for natural, expressive speech across Indian languages. Flash brings frontier-quality speech into a smaller, more efficient model, preserving the subtle details that make voices sound natural while making speech generation significantly faster. See details in our blog: links.sarvam.io/siuhmS
2
1
54
1,167
Tensor-Slayer retweeted
Can AI agents do interpretability research? At @PrimeIntellect Residency program, we built BioCircuit, an RL environment where agents trace attribution graphs on Qwen3-4B, find the features behind a concept and get graded by intervening on prompts they've never seen. Link : amitprakash.in/biocircuit/
10
26
2
196
12,688
Omarchy plugin layer is amazing @dhh , truly one step closer to agentic OS. Opus and i built Piphone: an @OmarchyLinux plugin that lets an agent drive my iPhone over USB.
3
1
4
630
warming up the algorithm
2
128