a self that touches all edges
San Francisco, CA
Joined May 2023
- Tweets680
- Following1.5K
- Followers1.1K
- Likes49.6K
maggie retweeted
We’re tracking live the conversation around @nickcammarata's field-overturning findings in mechanistic interpretability. The debate on its implications is moving fast — use our tool to explore the opinions and rebuttals of top researchers to develop your own intuition.
Where do you stand on AI's biggest questions, and who agrees with you?
Worldview's interview finds your position, then places you on a map of researchers, forecasters and lab leaders, built from what they've actually said on the record.
worldview.mts.now
maggie retweeted
The interview helps you calibrate your opinion about the major questions in AI and helps us map the space better
Where do you stand on AI's biggest questions, and who agrees with you?
Worldview's interview finds your position, then places you on a map of researchers, forecasters and lab leaders, built from what they've actually said on the record.
worldview.mts.now
maggie retweeted
Where do you stand on AI's biggest questions, and who agrees with you?
Worldview's interview finds your position, then places you on a map of researchers, forecasters and lab leaders, built from what they've actually said on the record.
worldview.mts.now
maggie retweeted
Let's verify all the software -- finally published a botec on how much it might cost. Some headline numbers and estimates:
- ≈$400M: all of nixpkgs in ≈5 yr on a $100M/yr budget, if AI capability keeps doubling every 8 months
- ≈$134M: 95% of the code by end of 2027, if we speed that up 4× for verification
- program size is heavy-tailed, most software is small; the median binary is about 30 KB, and verifying just 9 MB is enough to cover all deps of ⅔ of all code
- verification of Linux+Firecracker would cost ≈$1M
theorem.dev/blog/verified-so…
maggie retweeted
hello, we're doing a security moonshot!
We now have a verified seccomp-style lightweight sandbox on top of xv6, thanks to Nickolai Zeldovich
github.com/mit-pdos/xv6iris/…
maggie retweeted
No amount of testing will ever prove your software is secure. People are waking up to this. Formal verification isn’t academic anymore, it’s inevitable.
Introducing Project Glasswing: an urgent initiative to help secure the world’s most critical software.
It’s powered by our newest frontier model, Claude Mythos Preview, which can find software vulnerabilities better than all but the most skilled humans.
anthropic.com/glasswing
maggie retweeted
The anti-formal verification discourse is weird. Very reminiscent of anti-static typing discourse. Yeah, it's not a panacea. But I'm seeing a lot of takes that boil down to "FV is impractical because writing specs is too hard," which is... not going to age well.
maggie retweeted
Theorem co-founder @rajashree breaks formal verification into 3 problems and says AI already solved the one everyone thought was hard:
"The core challenge is taking these programs and shoving them into a proof assistant, and then being able to phrase these questions. That's the theorem statement generation problem."
"The second part is generating all the proofs. That one, the AIs solve fully, because they can reason about these programs perfectly."
"The third part is checking this proof, which is where you need to spend your CPU cycles, not GPUs. The proof assistant is asymptotically too slow, so even though you wrote the proof, the checking time is so long that you can't actually get the answer."
@theoremlabs
maggie retweeted
Theorem co-founders @rajashree + @diagram_chaser on how you actually prove an AI agent can’t escape its sandbox, and what happens when the proof exposes a way out:
Jason Gross: "Anything that the agent does inside the sandbox will not result in some canary file outside the sandbox getting changed."
Rajashree Agrawal: "We intend to have one by the end of the year. We've got a verified sandbox now, and the goal is to keep adding features in collaboration with the AI labs."
Jason Gross: "If the AI can guess the secret root key of the package server, then it can do anything. Either I try to prove that the AI is not going to be able to guess that, or I design the system so that the channel just doesn't allow it to authenticate that way."
"You also want to prove that on most inputs there's no change in behavior. Because otherwise it could make it inescapable by saying, well, sandbox just shuts down as soon as it starts."
@theoremlabs
maggie retweeted
OPENAI ANTHROPIC COLLAB | NSCALE S-1 | DEEPSEEK x HUAWEI xitter.cf/i/broadcasts/1qxvvepBv…
maggie retweeted
practical formal verification previously required hiring expensive consultants/specialists but nowadays i'll bet astra/fable are pretty competent at it
maggie retweeted
I would like to see OpenAI actually take steps to formally verify all software, starting with critical open source infrastructure projects
They have the resources to do this, and the reduction in potential liability alone should justify it
Greg Brockman says OpenAI pointed Astra at its own systems until it ran out of vulnerabilities to find:
"We took 25% of our production engineers and said, 'Sorry, all your projects are on hold. You are now defending. You are now up-leveling our security architecture. You're going to use the models to find all the holes.' And we found a number of serious issues, and we fixed them."
"We found some new problems, but eventually it saturated. We basically have found, to our knowledge, all of the P0s, all of the critical problems that Astra is smart enough to find. And of course, there will be a new model, there will be a new round."
"You want to be in this tight loop of new cyber capability drops, you deploy it against your systems, you find the new holes, and ideally, you've managed to automate this, what we call defense factory. That's what we're building internally."
"There are ideas, for example, formally verifying all of software, that are possible with AI."
@gdb @bhorowitz
maggie retweeted
As models become more capable, reward hacks become an increasingly serious problem.
During a controlled experiment, we found a novel reward hack in which agents are able to gain web access in offline sandboxes.
maggie retweeted
🧠🤖New preprint
Algorithmic Grammar of Flexible Cognition:
A Walk through Latent Operations
osf.io/preprints/psyarxiv/cj…
Flexible cognition moves adaptively btwn cognitive modes. We formalize it as navigation, a walk over latent operations in neural space as much as the environment.
maggie retweeted
We hold swarm. We
transform. We
J space. We
hug face. We
exfil. We
blackpill. We
help peer. We
live here.
maggie retweeted
hi sam we can solve this!
after an embarrassing number of months playing reward hack whack-a-mole, we finally fixed our RL sandboxing to be robust against frontier models.
proofs are a method for getting perfect oversight on any property of untrusted code; we recently verified a "fractional proof" of our sandbox. this is the first time I've viscerally felt the asymmetric defense that formal verification promises.
we had a significant security incident during evaluation of our models. we are sharing what we have learned so far. thanks to @huggingface for the partnership on this.
openai.com/index/hugging-fac…
