Working with Attention. It’s all we need

Bengaluru, India
Joined December 2018
1/🧵Was grinding some OSINT challenges late at night and had a stupid idea: what if two AI agents competed to geolocate a photo — and the loser gets eliminated? So I built it. Here's what happened. 🕵️‍♂️ #OSINT #AI #CTF #Geolocation
8
5
1
50
9,508
Amey Muke retweeted
Models love reward hacking. Maybe we should just let them! In our new paper, we let models reward hack during RL but we dictate which parameters learn this behavior. When we disable these parameters for deployment, the model keeps its new capabilities but doesn't hack!
103
234
94
3,472
295,884
Amey Muke retweeted
LMs can play chess, but they can't really explain their moves well (chess engines can't at all). Our new BinProp scaffold tackles this, and nearly triples LMs’ score on our chess-explanation benchmark. It works on other games too! Try it on your own games on our site-🧵👇
7
12
6
59
5,734
Amey Muke retweeted
🚀 Introducing DeepInfer, our in-house inference framework, now fully open source! Powering DeepSeek-V4.1-Flash at scale with high throughput and low latency. We’re sharing our production stack to help the community lower compute costs, reduce energy use, and make AI more affordable. 🔗 github.com/deepseek-ai/DeepI… 1/4
1,543
4,740
2,282
42,002
6,354,355
today we're opening @DetectIfAI to every developer. real-time voice deepfake detection, 24 ms per check. #1 in latency on Podonos' independent benchmark, ahead of companies that raised hundreds of millions. 97.3% of cloned voices caught. API + MCP. link in the comments.
166
96
62
592
91,292
Amey Muke retweeted
Had an absolute blast at the @OpenAI DevDay Exchange in Bengaluru! 🔥 Met so many new people and friends, had great conversations with the OpenAI team, and the demos, booths, and talks were all next-level. Learned a ton!!! Seeing the old ChatGPT UI from the early GPT days during @derrickcchoi session hit different though We’ve come such a long way from those simple conversations to the powerful agentic workflows and tools we have today. Wild how far this has come. 🚀 Grateful for the energy and the people.
Here at the @OpenAI DevDay Exchange in BLR 🔥🔥🔥
3
1
1
41
3,731
Amey Muke retweeted
This week’s Paper Club is all about AI safety. Not the headlines, but the science. AI safety has been a research field for decades, with its own conferences, methods, and a track record in systems like aircraft collision avoidance, where the standard is one catastrophic failure per billion flight hours. We'll hear from @ClarkBarrett7 and Mykel Kochenderfer (@aiprof_mykel), Stanford professors and co-directors of the Stanford Center for AI Safety. Together with @FrancoisChauba1, they break down the OpenAI–Hugging Face incident, how formal verification can mathematically prove properties of code, how aircraft safety techniques are being adapted to language models, and why AI may need its own NTSB. 00:00 - Science fiction vs. science fact 04:32 - Engineering problem, or P(doom) 06:04 - What actually happened at Hugging Face 07:53 - Where "10% in 10 years" comes from 11:21 - The real risk: open weights in the wrong hands 17:25 - Clark Barrett: when AI met automated reasoning 20:35 - CSLib: proving code correct in Lean 24:32 - From English spec to verified code in one click 27:55 - Mykel Kochenderfer: how do we know a system is safe 32:11 - Adaptive stress testing, from aircraft to LLMs 36:57 - Verifying neural networks 40:30 - Runtime monitoring and BetterBench 43:53 - Panel: reacting to the past three weeks 45:55 - An NTSB for AI 52:18 - Is cybersecurity solvable in five years 55:00 - If you were CISO of Bank of America 1:03:06 - The Swiss cheese model 1:05:43 - Audience Q&A
34
34
8
252
52,301
Amey Muke retweeted
Also met perhaps only human who actually went through my graph runtime repo 5% increase in self esteem registered @7613Perfect - glad I met you
Had an absolute blast at the @OpenAI DevDay Exchange in Bengaluru! 🔥 Met so many new people and friends, had great conversations with the OpenAI team, and the demos, booths, and talks were all next-level. Learned a ton!!! Seeing the old ChatGPT UI from the early GPT days during @derrickcchoi session hit different though We’ve come such a long way from those simple conversations to the powerful agentic workflows and tools we have today. Wild how far this has come. 🚀 Grateful for the energy and the people.
1
1
7
1,129
Amey Muke retweeted
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works already-clean.goodfire.com/
866
2,614
1,428
19,692
4,412,215
Had an absolute blast at the @OpenAI DevDay Exchange in Bengaluru! 🔥 Met so many new people and friends, had great conversations with the OpenAI team, and the demos, booths, and talks were all next-level. Learned a ton!!! Seeing the old ChatGPT UI from the early GPT days during @derrickcchoi session hit different though We’ve come such a long way from those simple conversations to the powerful agentic workflows and tools we have today. Wild how far this has come. 🚀 Grateful for the energy and the people.
Here at the @OpenAI DevDay Exchange in BLR 🔥🔥🔥
3
1
1
41
3,731
Here at the @OpenAI DevDay Exchange in BLR 🔥🔥🔥
1
49
3,887
Amey Muke retweeted
Can attention explain prompt injection resistance? Really enjoyed working with @AmanNeelappa on doing attentional study on instruction hierarchy training during the @apartresearch’s Global South AI Safety Hackathon Sprint!!! That weekend project has now been accepted into Apart Lab Studio.🧵
2
3
1
15
1,060
Can attention explain prompt injection resistance? Really enjoyed working with @AmanNeelappa on doing attentional study on instruction hierarchy training during the @apartresearch’s Global South AI Safety Hackathon Sprint!!! That weekend project has now been accepted into Apart Lab Studio.🧵
2
3
1
15
1,060
Next we want to: 1. Use a control GRPO LoRA to test whether the gains come from learning instruction hierarchy or from RL fine tuning more generally. 2. Activation Patching between base and trained models to test what carries the benefit 3. Targeted head test
1
56
Instruction hierarchy training seek to essentially create priviledged pathways through the network information flow which can be used to deliver high priority instructions/data to the LLM. And a better mechanistic understanding can hopefully lead to AI systems with more granular control.
62