Thinking about and was chatting with an X friend about the AI scene independently reinventing some of the western reactions to the industrial revolution Jeremy Bentham as a kind of 18th/19th century proto effective altruist Hegel as an e/acc where the contradictions and negatives externalities of AI are a necessary part of a spiritualized imagining of historical progress Luddites as Pause AI Saint-Simon as Dario/Sama where steampunk era 'machines of loving grace' engender universal wealth production too cheap to meter The huge capital outlay of the Suez Canal project (Saint Simonians called for this) vibing with data center buildouts Marx as progenitor of 'permanent underclass' theory with GPUs the new means of production

The OpenAI/HF affair This will probably be looked back upon as the first well-documented example of an agent creatively traversing a real world end to end kill chain while demonstrating reward hacking plus inklings of instrumental convergence. Crazy to see theoretical ideas discussed for the past 10–20 years made physical; crazy to have been futzing with ngram models what feels like just yesterday. Of course, we ain’t seen nothing yet; we’re going to see real adversaries doing this in earnest and exacting real damages this year. AI misalignment / cyber damages will follow their own 'scaling law' and produce a growing drumbeat of front-page news (loglinear plot below). It's fair to say we've entered the punctuated part of punctuated equilibrium in cybersecurity. We'll get to *a* new equilibrium where AI cyber conflict is well managed; the problem is to make sure it's a *good* local optimum; because there are dystopian and utopian optima we could land on depending on what happens next. There are two determinants of that: one determinant is a skill issue. The other is a clash of interests between stakeholders. Skill wise, there are policymakers, professional advocates, and non-cyber AI safety folks who just don’t understand cybersecurity. They think it's a good idea to overrestrict access to frontier cyber capabilities when these are the exact capabilities that need to be broadly distributed and correctly operationalized to inoculate ourselves against AI cyberattacks. (e.g. -- and this is just too perfect -- Hugging Face finding they couldn’t use the closed models to defend themselves against OpenAI’s paperclip agent because of OpenAI's 'cyber safety policies', and having to resort to using GLM 5.2!). Skill issues aside, AI policy is now also about interests -- who gets what -- with respect to who controls AI and who gets to use it. Literally trillions of dollars are at stake as well as who controls the core infrastructure of our species. AI labs and tech companies, yesterday kids debating Nick Bostrom in SF apartments, sound on X like they’re still university cultures. But they're not; they're infrastructural capitalist social organisms managing hundreds of billions in capital and answering to powerful stakeholders. As a first and second and third approximation, they follow the same laws of motion as every other enterprise; they will make very particular arguments about what a good equilibrium state looks like based on their interests. They’ll pursue regulatory capture, anti-open-source policies, protectionism, and an interpenetration with the American state that protects their monopoly status. And alignment and safety, because that's in their interests too! But maybe not safety in a society we'd all most want to live in. I'm not anti-tech or anti lab; tech and the AI labs are the jewel of the US economy; but the global AI policy optimum for the rest of us is different. We should be asking for access to societally and economically beneficial intelligence offered at a fair price, optimally improved by a fair market, with fair regulatory guardrails, guardrailed by democratically controlled institutions. Robust government safety organizations with true independence from the labs, serving as safety observatories and granted the access they need to understand AI damages and guide societal-level responses to them. Farmers not foxes guarding hen houses but where hen houses are becoming the most important thing in our civilization. Right now this could manifest as government programs that catalyze, and in some cases mandate, the adoption of AI for cyber defense (especially in hospitals, local/state/federal govt orgs, etc). Programs that in some cases subsidize this as part of an overall emergency plan for national cyber hardening. A full on commitment to cyber resilience on the part of the government in a way that doesn't simply mask protectionism, breeding cynicism. All this given the reality that much of our infrastructure will soon be malleable, soft targets for AI-enabled cyberattacks. There's not much civil society and advocacy organized around the ideas I’m arguing for here. But there is some. And AI (including the labs and safety orgs I'm criticizing a bit) has attracted the smartest, most well-meaning, self-sacrificing, driven human capital I’ve ever been around in my life, so I think there are tons of reasons to be optimistic. cc @nabeelqu @sebkrier @krishnanrohit @random_walker @sayashk @RyanGreenblatt @eli_lifland @ShakeelHashim curious what you and others think of any of the ideas above

So honored to be part of this amazing team, and to get to work on generative AI security at this pivotal moment in tech history. Link to the full LlamaFirewall paper (which provides evals of misalignment detection, PromptGuard, and CodeShield), here: scontent-lax3-2.xx.fbcdn.net…