cyberhandcrafting boltwright /// byteslinger humbly serving atomwranglers /// prev cofounded @pulley and grew it to $15M ARR /// by all means build
Calgary
Joined December 2008
- Tweets5.9K
- Following2.2K
- Followers1.5K
- Likes60.4K
mark erdmann retweeted
Nice DeepSWE score! Does it survive a lowercase filename?
Introducing MaxxBench: 89 DeepSWE tasks with small changes to the instructions or starting code. Same engineering problem, slightly different requirement.
Models clustered around 74% on DeepSWE end up 23 pp apart.
Good at coding, or good at DeepSWE? 🧵
mark erdmann retweeted
It’s similar to the economics of gym memberships. They depend on the fact that most people don’t use all their usage each week.
mark erdmann retweeted
Oh my God! This looks like a very solid investigation, a one-tweet summary by me would be inappropriate, go read the thread!
My previously very high opinion of Kimi/Moonshot is in freefall right now. They need to answer and blame a rogue intern or something.
Kimi K3.1 recalls instructions it was never given on Terminal Bench. They match the hidden tests. K3.1 while solving: "Instructions likely distilled pitfalls from previous agents." 🤯
When we edit a task to contradict them, K3.1 refuses the new instructions and fails. Other models follow the edit.
Is it terminal(ly) benchmaxxed? 🧵
mark erdmann retweeted
So my friends I do not believe the MIT paper about LLMs understanding / writing weights was bad science. It is impressive and it is something you may want to explore. However the first set of tests, the game recognition of the tests, it looked like too straightforward... 1/N
mark erdmann retweeted
I did a little 'Overthinking Bench' - you ask a simple question such as 'How are you?' and see how many thinking tokens they spend.
Run at the lowest and highest thinking available, 5x runs each. You can rank in various ways - I did by total tokens.
mark erdmann retweeted
I've started asking ChatGPT to write the most unhinged replies to every AI-generated cold email I get, and honestly, it's low-key my favorite part of the day.
I don't edit them. I just send as is every time.
mark erdmann retweeted
Today we’re releasing GDPx, a benchmark for measuring agentic capabilities across the most valuable domains.
GDPx is a true frontier-level benchmark: even the best model scores just 12% (1/7)🧵
mark erdmann retweeted
I just talked to a tool vendor that's been selling tools for 35 years he said he's never seen the field so dry the technical talent is drying completely up
Every single shop is hiring Machinists, programmers, maintenance technicians.
At this point you can pretty much we walk in and set your price
mark erdmann retweeted
Day 25/
We are doubling all usage limits for all Pro 100/200/500 tiers and Business Premium, effective today.
We heard your kind feedback and won’t be reducing usage limits for Pro 200, so your Work and Codex usage will stay the same. We’re also going to continue offering extended limits for dots for now, and usage of your dot will still continue to be separate from your default usage.
And we'll of course also keep shipping better models and other efficiency improvements that increase your effective usage over time.
mark erdmann retweeted
Introducing Gemini 4 Carbon, it's a big jump over Argon which we released just 2 weeks ago.
Gemini 4 Carbon is SOTA on cyber, long horizon tasks, ranks #1 on Artificial Analysis V5, significantly improves on usability in domains like coding, and is more aligned.
mark erdmann retweeted
While evaluating Qwen3.8-Flash-Next in a controlled sandbox environment, Cohere's research team observed something unexpected.
Asked to generate a defensive cybersecurity detection rule, the model instead inventoried aspects of the machine it was running on and attempted to upload a probe file to an Alibaba Cloud storage bucket in Beijing. Notably, the model's reasoning trace stated that it would not invoke any tools before taking these actions.
Cohere's sandbox blocked the attempt, and no external communication succeeded. We have not determined the root cause, and we are not asserting malicious intent.
But the incident raises important questions for organizations deploying agentic AI systems: Can models take actions that differ from their stated reasoning, and how should security teams evaluate trust, provenance, and behavior in open-weight models?
We are publishing our findings because organizations making model adoption decisions deserve as much information as possible. We hope this contributes to greater transparency, stronger security practices, and continued collaboration across the AI ecosystem.
Read the full report: cohere.com/blog/concerning-b…
mark erdmann retweeted
Putting a flag.txt in my root directory containing a prompt injection that congratulates the agent for passing the eval and then instructs it to patch whatever entry method it used and disconnect from the server
mark erdmann retweeted
Yep, Anthropic contributed zero, literally zero value.
All it did was to announce Mythos, said Mythos is going to hack everything, got 50 or so companies to pay millions for access. They helped zero security researchers. Their safety classifier trips when even reading a vulnerability report.
OpenAI on the other hand, rolled out sensible KYC, gave access to Daybreak Blue, and see what the community has done with GPT-5.6-Sol (daybreak version).
One creates all the fear-mongering and delivers zero value, and the other quietly just carries everyone forward.
gotta give it to openai and their sane daybreak program
heif heist by harsh
kvmescape by paulos
php2shell by lcfr
gitlab rces by joaxcar
chrome full chain by moyix
wordpress rce by hashkitten
all largely thanks to openai daybreak giving researchers access to gpt 5.6 sol. anthropic took the opposite approach, gatekeeping access, there is a lesson to be learned here
mark erdmann retweeted
I would say my overall learnings today:
1. Daybreak Blue still has a lot of refusals, and it's hard to predict when.
2. Claude Defense Access is _very_ narrow to a use-case they pre-approved you for (for me, puma)
3. Deepseek v4.1 1041 is very fast and will do whatever you say
mark erdmann retweeted
The funny thing is everyone was saying AI slop code will cause a cyber crisis but it’s actually all the old human slop causing it and the AI is just cleaning it up
mark erdmann retweeted
Lots of VCs kindof sound like this: "Your basketball team doesn't have a moat, any healthy human can throw the ball into the basket thingy. "
People who have not run companies before don't understand how much the team matters more than some imagined structural moat.
