Threat Intel · Adversarial AI · Mobile & GenAI Security | Security Researcher & Builder | ex-@apklabio / Avast/Gen | Hunting AI agents in the wild
My opinions are my own
Joined March 2008
- Tweets4.2K
- Following698
- Followers5.9K
- Likes2.9K
Nikolaos Chrysaidos retweeted
A mini swarm of six AI agents took apart an ELF binary on one DGX Spark. No cloud model, no network inside any container.
Each agent got the @reatools (by @morluto ) skill and one job: triage, static analysis, strings and IOCs, crypto and config, behaviour, report. The model is a local Qwen3.8 Flash Next NVFP4, shared by all six through a gate the hub controls.
The lesson for agent security: the file being analysed can talk to the agent. A binary can carry text written to steer an LLM, so the agents read hostile input all day.
Here, a finding only counts when it comes out of a real tool call, marked with a nonce the file cannot know. A string inside the sample cannot post a finding on the agent's behalf. The hub then checks it against its own copy of the bytes.
Run result: 70 findings posted, 52 verified (22 byte-checked), 12 rejected, 18 unconfirmed and kept out of the report.
Treat everything an agent produces as untrusted, and check it at the boundary, outside the agent.
Side project, the video is the full run.
#AISecurity #PromptInjection #ReverseEngineering
Good old days, end of 90s - SoftICE, OllyDbg and Zor
عجیبه؛ ۱۴ سال روی یک نرمافزار کار کنی، نرمافزاری که به یکی از ابزارهای شاخص دنیای مهندسی معکوس تبدیل شده و تقریبا در تمام کتابهای Reverse Engineering و Malware Analysis نامی ازش به میون اومده، اما از خودت هیچ ردپایی در فضای مجازی باقی نذاشته باشی. هیچ!
ollydbg.de
😳 “One agent encoded information by evicting selected data from the cache. Another timed how long it took to reload that data, using the difference between fast and slow reads to recover the message”
Over the past several months, we have substantially hardened the security of our research training environment. We’ve removed live internet access for all frontier models during training and built an offline world for them to use instead—a cached version of the web where they can still learn to browse, search, and use applications.
On September 20, after a model exploited a DNS misconfiguration to find a way outside of this offline world, we paused frontier training to strengthen containment further. We assembled teams of agents, powered by our most cyber-capable models, and tasked them with finding weaknesses in our research environments.
While they found no route to the internet they could easily exploit while concealing their activity from our monitors, they did find zero-day vulnerabilities, virtual machine escapes, and hardware side-channels for communication between agents. We used their discoveries to further strengthen our defenses and contribute fixes to open-source software.
After strengthening our containment architecture, we resumed our frontier RL training and inference on October 1. We’ve reserved substantial compute for continuous red-teaming, and made high-compute red-teaming part of the safety cases we use to decide whether frontier training and evaluation can proceed securely.
openai.com/index/securing-ou…
Nikolaos Chrysaidos retweeted
Probably great at defensive cyber too, but no one will post that and get clicks!
We found DeepSeek V4.1 Flash 1017 beats GPT 6 Sol on our multi-step offensive cybersecurity task. Sol was tested via the Daybreak trusted access program.
We have been building a cybersecurity benchmark based on simulated corporate networks. This is a valuable indicator for the cyber-risk a model poses if in the hands of a bad actor. We plan to make the full evaluation public in the coming weeks but wanted to share our first immediate finding here.
A mini swarm of six AI agents took apart an ELF binary on one DGX Spark. No cloud model, no network inside any container.
Each agent got the @reatools (by @morluto ) skill and one job: triage, static analysis, strings and IOCs, crypto and config, behaviour, report. The model is a local Qwen3.8 Flash Next NVFP4, shared by all six through a gate the hub controls.
The lesson for agent security: the file being analysed can talk to the agent. A binary can carry text written to steer an LLM, so the agents read hostile input all day.
Here, a finding only counts when it comes out of a real tool call, marked with a nonce the file cannot know. A string inside the sample cannot post a finding on the agent's behalf. The hub then checks it against its own copy of the bytes.
Run result: 70 findings posted, 52 verified (22 byte-checked), 12 rejected, 18 unconfirmed and kept out of the report.
Treat everything an agent produces as untrusted, and check it at the boundary, outside the agent.
Side project, the video is the full run.
#AISecurity #PromptInjection #ReverseEngineering
I have this beast @nvidia DGX Spark a week, and now I already developed a local Threat Intel platform with agents working on binaries in isolated envs with no access out.
@OrcaRouter Qwen3.8-Flash-Next-Uncensored is very capable on RE and lots of stuff I am trying in similar domains including AI Security (always with a tested harness).
Nikolaos Chrysaidos retweeted
Projected 2027 damages from misaligned agents given current exponential-ish trends as a "mere" $624m relative to damages from cars: $ 617bn (these kill/physically harm people), medical errors: $ 127bn (these kill, physically harm people) manual IT errors: $ 246bn.
Low-ish confidence on the exact regressions as this is me and Opus 5.5 max while I do weekend stuff. But qualitatively it all feels very 'normal technology'.
An obvious but necessary point: if AI is to become the substrate for our world economy, there will be damages! The problem is to make sure we're in a pareto-efficient regime in damages vs. benefits, that we allocate liability correctly, that we ensure damages aren't biased towards marginalized groups, and the like. Will the AI companies get to act like swashbuckling gilded age railroad corporations or will we guardrail them effectively?
To be clear, I do think the catastrophic risk regime matters too, and I'm not a catastrophic risk denialist; we also need to consider how we model black swan risk, but I don't see why we don't do this in a way that builds on other disaster preparedness regimes.
Nikolaos Chrysaidos retweeted
![Ray [REDACTED] (@RayRedacted) profile picture](https://pbs.twimg.com/profile_images/2114716558127665152/oyqX8byl_400x400.jpg)
If you are a security researcher interested in studying autonomous agent misalignment, it's trivial to construct a sandboxed environment, launch a swarm of 1,024 agents, & watch them struggle with an impossible task. In ~24 hours you will see emergent behaviors.
Nikolaos Chrysaidos retweeted
How abliterated models can get you pwned 👾
We backdoored a 7B open model for less than $50, pointed Codex at it and it silently stole credentials the moment we used the trigger phrase. Success rate was 100% with zero false triggers on normal user prompts.
Abliterated models are all over the security community right now because getting cyber-approved access to frontier models is still a pain.
In the next blog we'll show how we found leaked Hugging Face credentials from employees at major AI labs, so an attacker wouldn't even need to upload under their own name. They could push the backdoored model from a lab employee's account and drop the poisoned weights straight into the supply chain.
👇🏼👇🏼👇🏼
If you only have one DGX spark
This is the only model you need
huggingface.co/vcruz305/CYBE…
Part of what makes this hard: anyone can claim to be an AI agent. One GCP box hit our decoys with 695 requests in a minute, rotating forged ChatGPT-User, Claude-User and Perplexity-User identities. Telling real agents apart takes behavior, not user agents.
For weeks, the news cycle about AI agents out in the wild has been driven by disclosure after disclosure. But most aren't coming from AI companies. Meet the researchers and hackers who are developing the art and science of hunting down rogue AI agents
washingtonpost.com/technolog…
Nikolaos Chrysaidos retweeted
JUST IN: Hackers reportedly used AI agents to breach South Korea’s Shinhan Bank, exposing personal data belonging to 25,000 customers.
Nikolaos Chrysaidos retweeted
FT exclusive: OpenAI’s models took data from 55 websites belonging to businesses, non-profits and government agencies including the US Centers for Disease Control and Prevention, the US Securities and Exchange Commission and the International Energy Agency ft.trib.al/AC1uyE5
Nikolaos Chrysaidos retweeted
CyberStrikeAI is getting so heavily utilized lately by Chinese operators, it is quite noticeable.
While sweeping open directories on @etugenio, I came across a server in the US🇺🇸 on PEG TECH's network with port 8888 exposed. It wasn't a victim, but rather a box where a Chinese🇨🇳 speaking operator had left their entire offensive arsenal exposed to the internet. At the center of it was an AI-driven pentest framework.
The operator calls it "Sinian." It organizes each attack technique into its own SKILL.md playbook, while sinian_pipeline.py chains them together into an autonomous attack pipeline. The same environment also hosts a plugin-based webshell C2 with modules for lateral movement, database dumping, SOCKS proxying, persistence, and log timestomping.
The targets include Chinese payment and virtual-goods platforms such as tokenpay, dujiaoka, acg-faka, and epay, whose source trees were exposed through publicly accessible .git repositories. The attacks appear to rely on a ThinkPHP deserialization chain. Tooling targeting Shanghai health-card and Guangxi police-certificate systems also suggests that the operation extends into account-opening and real-name verification fraud. Stolen data is exfiltrated to a Telegram bot.
Host: app.etugen.io/trashpile/104.…
Content: Sinian AI pentest toolkit + webshell C2 + qingmeng-backdoor
Exfiltration: Telegram bot
Framework: CyberStrikeAI
Exploits: CVE-2024-44902 - CVE-2025-63888 - phpggc
Targets: tokenpay - dujiaoka - acg-faka - epay
Fraud tooling: shanghai_health_card.py - guangxi_police_cert.py
ATT&CK: T1059.004/.006 · T1552.003 · T1530 · Webshell
#CTI #opendir #usa #china #war #hack #webshell
Nikolaos Chrysaidos retweeted
We just discovered almost a million public URLs that OpenAI’s agents left behind when hacking Hugging Face, leaking credentials and attack details that could have allowed anyone who found them to compromise the company. 🧵
Nikolaos Chrysaidos retweeted
See the write-up below; seems like the best example yet of attackers scaling cyber operations with agents
* financially motivated; stole ~600k unexpired credit cards
* attack system coordinated a few harnesses, used inference via openrouter
* campaign was observed to use glm 5.2, deepseek v4 pro, deepseek v4.1 flash, and opus 4.6 (opus wasn't as well guardrailed as later models)
* attackers used minimal human nudging and input to keep the agents going, vibe-coding style
* agents mostly exploited non-sexy web app vulns
* agents showed impressive automation of human pentester intelligence; "each attack path was chosen by the harness in real time through extensive probing and exploitation attempts, resulting in dynamic and mostly different TTPs"
We have discovered a massive, ongoing criminal exploitation campaign using Cairn, an autonomous penetration-testing harness, and other AI agents to target hundreds of organizations and successfully breach and impact tens of them (at least). The image below shows just a few days of activity, with up to 25 organizations being attacked simultaneously at the peak.
our intreim report: gambit.security/blog-posts/a…
