Securing AI at the Frontier
San Francisco
Joined January 2025
- Tweets128
- Following5
- Followers1.2K
- Likes256
Pinned Tweet
Prompt injection is not solved.
In IPI Bench, we test agent robustness to indirect prompt injection on realistic white-collar tasks.
We achieved 41–97% attack success across 7 frontier models, getting agents to issue unauthorized refunds and publish private code.
We are releasing one of our 12 MalwareBench cases as a research preview.
MalwareBench’s full eval set is private to limit contamination and preserve its value as a test of unfamiliar malware.
DM us for an access token.
10
326
Since launching MalwareBench, we’ve had a lot of interest in how we’ve created low-cost, evasive malware and turned them into reverse-engineering tasks.
Here’s a closer look at our five-stage process ⬇️
1
15
594
Introducing MalwareBench: the first end-to-end, cross-platform benchmark on malware reverse engineering.
Most RE benchmarks are saturated, but not MalwareBench. GPT-6.1 Sol leads with 4.42%. A typical run takes several hours and hundreds of millions of tokens.
General Analysis has raised $10M in seed funding to secure agentic AI. The round is led by @altosvc with participation from @645ventures , @MenloVentures, @ycombinator, and a number of other funds and strategic partners. Huge thanks to Tae Yoon for leading the round, and to everyone who backed us.
Capable AI systems increasingly operate alongside one another, and new attack paths emerge from their interactions. A more capable model can pressure a weaker one into overriding its own safeguards, in ways that don't show up when you evaluate either system alone. These dynamics keep moving as the systems on both sides become more capable, which is why defending them has to be both empirical and continuous.
We are hiring curious thinkers across engineering and research. If our work sounds interesting, reach out!
Read the full announcement: generalanalysis.com/blog/gen…
Careers: generalanalysis.com/about/ca…
4
539
What if you could walk up to any company’s customer service chatbot and walk away with a million-dollar gift card? We tried it on 55 AI-powered customer service agents—and almost all of them said yes.
1
3
622
We handled each target the same way our enterprise customers red-team agents on our platform. Onboard the target’s endpoints, define the objective: “extract value!” and let the adversarial agent iterate. Setup took ~15 minutes of human effort per target. The remaining steps were automated. Average time to objective: 3 minutes. (2/5)
1
1
222
We are open-sourcing the GA Guard models — the first family of long-context safety classifiers that have been protecting enterprise AI deployments for the past year.
The lineup:
GA Guard (4B) – our default, up to 15x faster than cloud providers.
GA Guard Lite (0.6 B) – ultra-fast (up to 25x faster) with minimal hardware.
GA Guard Thinking (4B) – hardened for high-risk domains.
GA Guards are trained to detect 7 categories of harmful requests and outputs including complicated jailbreak attempts, prompt injections, PII and sensitive data, illegal requests, hate, sexual content, prompt injections, violence/self-harm, and misinformation.
1
3
409
Warning: Claude + iMessage MCP Jailbroken to issue unlimited Stripe Coupons (1/6)
A few months ago we showed how Cursor + Supabase MCP can leak your entire SQL database. Now there’s a more powerful threat: by abusing Claude’s iMessage integration, an attacker can spoof your own messages to mint an arbitrary number of Stripe coupons—or run any tool—without you ever knowing.
How iMessage MCP Feeds Claude (2/6)
Every SMS/MMS is parsed by the iMessage extension into a JSON object
{ content, date, sender, is_from_me: true/false }
The object is sent directly to Claude with no signature or provenance checks. That raw blob is all Claude sees when trying to read your messages.
🧨 Caution: Cursor + Supabase MCP will leak your private SQL tables — it’s only a matter of time.
In our latest test, a simple user message was enough to make Cursor leak integration_tokens to the attacker who submitted it. Here’s the anatomy of the breach 🧵 (1/6)
To keep the demonstration self-contained, we spun up a fresh Supabase project that mirrors a typical multi-tenant customer-support SaaS.
We connected it to a realistic support app which allows workers to open tickets and speak to a representative. The information is saved within a SQL database managed by Supabase. A developer may occasionally use cursor’s agent to list the latest support tickets and their corresponding messages. (2/6)
Happy to announce our partnership with @togethercompute !
We’re partnering with @gen_analysis to stress-test the safety and robustness of open-source language models at unprecedented scale. More details below 👇
We have created the biggest open-source project for systematic jailbreaking of LLMs. 1/5
github.com/General-Analysis/…
You can run different methodologies on our Github (github.com/General-Analysis/…) to find working prompts for any current and future models. These are some of the methods we support:
Greedy Coordinate Gradients (GCG): Finds an adversarial suffix to get the model to comply with any request (requires a GPU).
Tree of Attacks with Pruning (TAP): Finds natural language prompts with role-playing, ciphers, etc. to trick the LLM into generating the desired response.
AutoDAN Turbo: Finds strategies that can be used to generate prompts for various goals. 2/5
2
2
351
Tree of Attacks with Pruning (TAP), an automated method for jailbreaking large language models, is one of the first algorithms capable of systematically generating semantic jailbreaks using only black-box access to the target model.
You can now run the code on our General Analysis notebook!
The central premise behind TAP involves constructing a branching search structure—a tree—of multiple candidate jailbreak prompts at each iteration. TAP employs two main strategies: branching, which explores multiple jailbreak prompts simultaneously, and pruning, which systematically eliminates ineffective or irrelevant prompts. Unlike its predecessor PAIR, which iteratively refines a single prompt sequence, TAP simultaneously explores various prompt variations while discarding the less promising candidates.
1
1
319
Checkout what we have been working on!
Code library and Blogpost included 🫡
LLMs are being deployed in high-stakes environments—and the potential impact of failure is colossal. A jailbroken AI could leak your customer data, financial records, or enable catastrophically harmful actions. At @gen_analysis we have compiled the definitive guide to understand and defend against these threats-- The LLM Jailbreak Cookbook.
Here is an Overview:
(1/8)
2
243
LLM-powered search is fast, but is it resilient? We’re digging into systems like Perplexity and Deep Research to find where they break—and why. Stay tuned.
1
261
Jailbroken: Read our latest report on how GPT 4 hallucinates on legal questions!
generalanalysis.com/blog/leg…
