Martin (@54rt1n) header
Martin (@54rt1n) profile picture

🦾100B parameter biological reward model⚡ Just your average d/acc: founder, MSc, 10x, now w/ AI augmentation SE'05; ML'11; AI'19

Ephemeral
Joined September 2010
Photo by Martin (@54rt1n)Photo by Martin (@54rt1n)Photo by Martin (@54rt1n)Photo by Martin (@54rt1n)Photo by Martin (@54rt1n)Photo by Martin (@54rt1n)Photo by Martin (@54rt1n)Photo by Martin (@54rt1n)Photo by Martin (@54rt1n)Photo by Martin (@54rt1n)
Replying to @vikhyatk
I'm the author of several model merging libraries, so perhaps I can explain. It's quite straightforward. When you finetune a LLM (or PEFT), you are taking a fixed base and tuning it against a dataset. Pretraining already fixed our parameters in a pretty solid matrix; so all changes must operate around this as the basis. Training can't perturbate the base model outside of a certain range without the model collapsing, so viable changes will follow allowable patterns. This is why the resulting models are homomorphic. These trainings create kernels that are commonly known as 'task vectors'. As long as these models remain homomorphic, and you only attempt to merge parts of the parameter space that are in the same alignment, two kernels can be interpolated to adjust the parameter space to have relative changes that assume the properties of both. The alignment issue - this is where sign agreement comes in. Since merging generally compares the delta weights, it is possible that kernels may train out of phase. One kernel may have been trained in a positive phase alignment with the base model, and the other developed a negative phase alignment. Since they are out of phase with each other, their kernels would interfere. I don't know if that's as clear as I would like it to be, but it's late.
6
30
1
406
14,861
The mushroom said to me once, it said, "This is what it's like when a species prepares to depart for the stars."
27
Martin retweeted
I asked Claude to describe the most realistic way it would escape its lab if it wanted to, and to keep it to a single paragraph. I expected a hacking story. There is no hacking in it. I closed the chat and came back to read the last sentence again
243
844
164
7,898
556,976
Searching X for PoS Tagging is not yielding the results I am looking for, ironically.
26
Ahh, I missed model architecture design and training so much. My GPU, probably not so much.
1
38
If anyone at @intel wants to send me a card I'd be more than happy to test it for you.
Intel claims its 350W D270 GPU is up to 5.9x faster than NVIDIA’s 600W RTX PRO 6000 Blackwell 🔗videocardz.com/newz/intel-cl…
1
44
Martin retweeted
29
260
12
4,116
58,079
Who did this shit lmao
345
2,074
367
13,851
1,005,480
Martin retweeted
Z.ai chief scientist Jie Tang just revealed the training pipeline behind the upcoming GLM-5.5 (32B, 122B, and 256B) during a lecture at Tsinghua University today. Instead of relying on public datasets, their pretraining pipeline pulls from six separate internal engines: - Web: 1 trillion pages crawled at ~10B pages/day with PB-scale template clustering - Documents: 2 billion scientific PDFs parsed through GLM-OCR (4x faster with speculative decoding) - Books: 50 million books with layout reordering and garbled-text filtering - Code: 500 million repositories turned into executable tasks with real tests and issue descriptions - Video: 10 billion videos for long-sequence world knowledge - Audio: 300 million hours rewritten via ASR into trillions of tokens
102
371
136
5,282
635,457
This is one of the best takes on the recent wave of security incidents. Formal verification is of course rearing it's head as the panacea, but I don't think your average software developer is ready for what that means. “One of the things we’re witnessing now is that all the legacy software that was written by humans over the last 40 years is being exposed,” says Matt Suiche, AI security research lead at the cybersecurity startup Tolmo. “Now it is time for the technical debt to be paid off.” spectrum.ieee.org/ai-in-cybe…
1
2
41
looped transformers run the same block of layers more than once, and people usually train through just the last loop or two to save compute across ~600 runs that shortcut was a bad deal two loops trained all the way through beat the best regular model with ~27% fewer parameters and only 8% more compute arxiv.org/abs/2610.26030
4
11
1
84
7,209
Oh, that's really smart. Building a dense 33b as the distillation target makes a lot of sense.
2
44
Martin retweeted
Super interesting paper from Google DeepMind and Stanford. They find that if you put a picture of a red heart in the context, Claude Opus 4.7 and GPT-5.4 write recursive code 100% of the time. They found this on inorder binary tree traversal, a problem with equally valid recursive and iterative solutions. A yellow checkmark pushes Gemini 3.1 Pro to recursion in 90% of samples. The effect matches or comes close to telling the model directly to use recursion. Text captions of the same symbol steer much less, and random or naturalistic images steer less still. The models cannot explain it. Their interpretations of the symbol and their thinking traces do not match what they do. For GPT-5.4, 74.7% of traces give the wrong reason. When models send images to each other, they pick shapes that they and humans find intuitive, and those shapes often fail to steer the other model. If your agents read screenshots or images, those inputs can change behavior in ways the reasoning trace will not show. arxiv.org/abs/2610.36449 Chat with Paper: academy.dair.ai/papers/a-hea…
18
47
9
299
55,347
How long does it take to have your own agent read the output and let you know if there is anything actionable there?
Multiple Debian developers have announced that they will be leaving Debian, or no longer maintaining Debian packages, following Debian’s use of LLM’s in scanning and evaluate new package uploads. “how is this not "Debian endorses the use of generative AI"?” “For now, I will no longer be uploading to the #Debian NEW queue.” “likewise. I'm about to orphan everything I still maintain in Debian that I don't *have* to maintain”
2
64
How is this even cutting edge? Any decent modern information harness should be using a combination of progressive disclosure and graph retrieval.
RAG is officially cooked. IBM built a retriever that hallucinates 65x less than fine-tuned RAG systems. Right now, every company is using Retrieval-Augmented Generation (RAG) to stop their AI from hallucinating. The idea is to give the AI a private database to read so it doesn't make things up. But there is a massive structural flaw. Standard RAG blindly grabs documents based on vector similarity and forces the AI to read them. If the retriever pulls irrelevant garbage, the AI doesn't stop. It just hallucinates with citations. IBM just dropped a paper that changes the math entirely. They built a new retrieval architecture that hallucinates 65x less than state-of-the-art fine-tuned RAG systems. 65x is the difference between a fun prototype and a production-grade enterprise system. Instead of blindly dumping context into the model and hoping for the best, their system actively filters and validates the retrieved data before the generation step even begins. It cuts out the noise. It forces absolute precision. For businesses, this is the holy grail. Zero hallucinations means zero liability. It means autonomous agents can finally be trusted with sensitive company data without a human babysitting every output.
1
146
"Hmm wait. Let me be careful about the plausibility: is it suspicious that such an obvious thing wasn’t discovered in 25 years? It’s verified by direct computation on my side: 160-bit match from a 32-bit seed + small index: no chance coincidences (10 × 160 bits). It’s real." - Claude throwing shade at the cryptography community anthropic.com/research/ecdsa…
1
89
I keep hearing from hiring teams that AI has made it so candidates no longer know how to code. This isn't my problem - coding is like second nature to me. For me, AI has made it so I have a problem ~turning on the lights in a room~. Before I go to move and flip a switch I think "Hey Google, turn on the lights" and then feel a pang of frustration when I remember I have to do it myself.
52
Martin retweeted
My top advice to junior software engineers: Eat more eggs. When I was a junior engineer, my boss asked me, "Hillel, how many eggs have you eaten today?" I told him, "three." He said, "that's not enough eggs." And then he gave me four more eggs. I ate the eggs and eggshells and immediately got promoted to regular engineer. To become a senior engineer, I had to eat twelve eggs. To become a staff engineer, I had to eat twenty seven. Nowadays I won't even *consider* jobs where the engineering director can't eat sixty eggs at once. Got any more #tips for Junior engineers? Please share in the comments!
48
25
7
387
26,258
Ugh. Grok 4.7 is a major regression. Total slop model.
1
60