https://xitter.cf/t.co/mC1FmHzTlZ Reluctant Mathematician (No relation to the chef on Instagram)

Joined September 2010
I use LLMs for work, hell these days I'm in charge of implementing them securely. But this, I resent over every other harm/disruption that AI companies have inflicted on the world in the last year or so. Some sacrifice is needed for progress, dont get me wrong. But the open discourse? The fun conversations that spark innovation? The feedback that drives ideas from good to great? That's all going away, if its not gone already. And in my opinion, it wasn't worth it.
Truly sad to see the intellectual commons become a Dark Forest
1
11
1,792
Keith Ramphal retweeted
The way frontier labs have restricted access to cyber capabilities makes no sense. If you're considered a "defender" you're not allowed access to the "offensive" versions of the models (the ones that can do development/validation of malware and exploits). So for example, if you want to build some sample malware to test a detection system, nope. Or if there's a major crisis due to a critical vulnerability in PHP and you need to rapidly triage fake exploits to see if there are any real ones floating around so you can warn people, nope. Want to build a remote code exploit to make sure the detection rules you plan on proposing to the clients actually work? Hard nope. You gotta call your red team friend and wait for them to do your job for you, because "defender" are only allowed to play in the sandbox with the children. Not an expert, but I think that the ability to triage in-the-wild exploits during a major crisis affecting 40 million systems is possible maybe important. And then they want to ban abliterated open weight models, which half of the people are only using because frontier labs will only give them access to lobotomized models.
19
39
4
247
13,548
Keith Ramphal retweeted
Current research status
15
68
3
1,283
25,230
Keith Ramphal retweeted
Perhaps something to consider if you work at OpenAI / Anthropic: If your policies have produced an outcome like 'if a researcher wants to do software verification research then they have use a Chinese model', then perhaps your approach to 'safety' is net negative.
20
79
4
978
26,685
Keith Ramphal retweeted
If I recall correctly, I met @diagram_chaser and @rajashree at a YC event sometime around F24, which I believe was their batch. They were the second FM YC company, after us, and the first to do interactive theorem proving. To the best of my knowledge, they were the second overall company to do LLM-based ITP after @HarmonicMath . We were dealing with gpt-4o in those days, and I couldn't even get the model to write a good fuzz test, or set up a sandbox (or an AWS lambda or similar) with someone's repo. There was a bunch of demand -- we launched agentic PBT and __immediately__ had >100 users, which was way more than we could keep up with at the time -- but the tech wasn't even remotely there. I thought Jason and Rajashree were fucking nuts to be trying to actually prove stuff, when literally just testing was borderline impossible. Later, I heard rumors of research projects with Redwood and maybe ARIA, and I thought, huh, I guess that's a reasonable way to keep your company alive and learn things without taking on more non-dilutive funding. Good for them, they found a way to do the research lab thing for a while and have some fun. FM is about the fun we have along the way! But I was hella wrong to be remotely skeptical. The reality is that, Jason and Rajashree spent two years becoming state of the art at large scale agentic verification. Later, when a biglab asked me "who should we talk to if we want to do formal methods", I was like, "I mean obviously, Theorem." I think a **lot** of other people said the same thing. Who else? (To be clear, there are a couple other quite meaningful players in this space, like @axiommathai and @mathematics_inc , but they've largely positioned themselves as AI neo-labs themselves, which makes them awkward partners for a biglab, so ...) I'm not convinced that a world where we formally verify assembly code is really ideal. I am concerned it enables a world where we LLM-generate assembly, which I view as an uninterpretable and harmful outcome for humans (here I disagree with both Elon and my employer ..). I would like to believe that there are all kinds of really elegant, important properties that are best expressed over more abstract representations. But a world where we've proven *anything at all* about a giant proportion of software is actually a **massive improvement** over the counterfactual. So even if I feel inclined to nitpick the approach, the reality is that this looks like a phenomenal overall uplift in cybersecurity and a totally worthwhile and important endeavor. This is a great time to start learning formal methods. And it's a great time to work in formal methods. I really couldn't be more excited about where we're at, and can't wait to see what's next from the folks re-inventing the field!
New on our Frontier Red Team blog: using Claude to formally verify software at large scale. Formal verification can prove that code is free of certain bugs, but historically it has only been feasible for small codebases (or required a lot of expert time). With AI models, however, this is beginning to change. In collaboration with @theoremlabs, we used Claude to prove security properties in over one hundred real-world open-source libraries over the span of a few weeks. Read our research in full here: anthropic.com/research/forma…
4
7
68
2,872
Oh look they’ve caught on that this is helpful
New on our Frontier Red Team blog: using Claude to formally verify software at large scale. Formal verification can prove that code is free of certain bugs, but historically it has only been feasible for small codebases (or required a lot of expert time). With AI models, however, this is beginning to change. In collaboration with @theoremlabs, we used Claude to prove security properties in over one hundred real-world open-source libraries over the span of a few weeks. Read our research in full here: anthropic.com/research/forma…
7
414
I help reverse old games to preserve them, it was a hobby I stopped doing but have been able to get back into. The Factor 5 team were pure artists, they pushed the GameCube in ways that were just brilliant. (this has nothing to do with AI, and everything to do with me getting caught up)
1
2
59
Keith Ramphal retweeted
In this new age of LLMs, it feels very much like: 1. we are seeing a dramatic increase in the amount of new code being generated 2. we are not getting an even noticeable increase in the quality of software (from an end user point of view)
77
74
14
1,932
32,781
Keith Ramphal retweeted
The context behind such tweets is that the AI companies are rejecting 40 years of cybersecurity knowledge. They are ignorant of cybersecurity and are trying to re-invent it from scratch, repeating the mistakes of bad people from 40 years ago.
What happens in a major incident where the frontier models all become unreachable?
11
50
2
265
9,044
Keith Ramphal retweeted
What in the regulatory capture are we doing here
18
51
5
258
8,813
This is entire scenario is weaponized misinformation and everyone pushing this narrative has either no idea how real world malware works or is playing along to push the agenda of hyping AI with capabilities that don't exist This is coming from someone who wrote the actual worms
Anthropic says its research found ‘signs of life for AI worms,’ that could colonize household devices. Experts say they have a point. theverge.com/ai-artificial-i…
Readers added context they thought people might want to know
Anthropic's research used simplified, controlled tests on emulators with trivial unauthenticated access and did not demonstrate a self-replicating AI worm colonizing real devices; caveats and expert skepticism on hype were downplayed. anthropic.com/research/self-… reuters.com/business/ai-mo… newscientist.com/article/259398…
13
32
1
151
4,692
Keith Ramphal retweeted
cat laziness is so contagious. you'll see your cat flopped on its back in your bed a completely comfy cozy baby and you'll be like. you're right. you make a compelling argument.
15
4,241
126
36,025
249,242
Keith Ramphal retweeted
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works already-clean.goodfire.com/
884
2,631
1,441
19,819
4,509,001
Keith Ramphal retweeted
When I think about what the future of programming looks like, I think about formal methods. It’s the natural answer to both increasing levels of abstraction and increasing needs for assurance. But what’s the future of formal methods exactly? Here are two projects I think give a glimpse: 1. Simon Henniger’s Djinn, a spec first language which reimagines logic programming in an era where the synthesis of both code and proof can be done agentically: henniger.dev/post/djinn/ . Look closely at the resulting performance gains! Making ripgrep 3% faster while being provably equivalent is not easy. 2. Sam Xifaras’s ruby semantics done quick: samx.io/blog/topics/devlog/2… . Sam smartly dodges the inevitable “@kirancodes found a bug in my semantics” moment by leading with known limitations up front — and he presents a really interesting take on how to vibe semantics, with differential fuzzing for comparison.
8
17
1
133
5,262
Keith Ramphal retweeted
not every AI sandbox escape is evidence of an emerging superintelligence. sometimes you just have a shitty sandbox and bad security practices.
85
168
32
1,267
25,196
Keith Ramphal retweeted
If only there was some defense agency charge with testing defense weapon systems and programs under operationally realistic conditions to make sure it can operate effectively in combat situations and conditions… 🤔
Important article to better understand where we need to go. Project Convergence is a great exercise (was apart of 2024), and exposes a lot of soldiers to the true state of innovation across the force. A lot of tough times remain ahead.
7
3
87
2,775
Company coming in last in a race asks for other competitors to slow down.
JUST IN: Microsoft CEO Satya Nadella urgently calls for an “emergency brake” on advanced AI development.
9
305
Agreed, someone gave it an instruction and told it to do something. Hold them liable.
there's no such thing as a rogue agent man its software someone made and deployed they are responsible for what it does cannot let this blame shifting occur
1
1
17
907
Keith Ramphal retweeted
Ok I checked: LLMs aren’t conscious, and actually they never will be.
65
14
9
280
15,044
Good news, now you can leverage the fine decision making that brought you Windows 11, innumerable variants of CoPilot, and Microsoft Teams for your organization.
Introducing Microsoft-Decision-1, our new model for fast decision-making. It delivers top performance on structured decision tasks, outperforming both LLMs and other decision models in latency and quality. We’re already testing it across Microsoft for everything from incident response and quality control to scientific discovery.
1
4
17
933
Like I said, this is how we go from open communication to the dark forest. This is where progress dies, because people who work on this are not going to want to share it with a company that doesn't give anything back.
Hugo Duminil-Copin (médaille Fields 2022) : la résolution de 350 problèmes majeurs par OpenAI, c'est comme si des camions m'avaient écrasé ; tous les problèmes (et donc pistes de recherche) que je mentionnais dans mes exposés, articles et demandes de financement, ont été résolus.
1
14
952