CS PhD Student @GeorgeMasonU
Fairfax, Virginia
Joined September 2014
- Tweets77
- Following459
- Followers281
- Likes545
Pinned Tweet
🚨 New paper: Data-driven Circuit Discovery for Interpretability of Language Models 🚨
Do circuits actually explain how language models (LM) implement a task?
In mechanistic interpretability, the goal of circuit study is to discover a “circuit” that is responsible for implementing a “task”.
But we find that existing methods often discover circuits that are:
❌ not general task circuits: they do not capture the full range of mechanisms LMs uses across the task.
Instead, they find:
✅ dataset-specific circuits: they explain how the model processes the examples used for circuit discovery.
✅ mixed-mechanism circuits: consisting of multiple independent mechanisms mixed in a single circuit.
1/🧵
Daking Rai retweeted
Thanks @AmazonScience for the award!
📣 Amazon Research Awards announces the 34 recipients of the Build on Trainium program, a $110M credit initiative supporting AI research at 30 universities on AWS Trainium: amazon.science/research-awar…
Daking Rai retweeted
Honored to receive the NSF CAREER Award on the topic of "Towards Scalable and Actionable Interpretability of Language Models for Code Generation"! Thank you to @NSF for this recognition and all the support ever since my PhD study, and to my students, collaborators, colleagues, advisors/mentors, and everyone else who has helped me and inspired me in my career!
This idea is a convergence of my research topics started during PhD study at @osunlp (code generation) and after I joined @GMUCompSci (interpretability). The terms "scalability" and "actionability" are widely discussed in the interpretability community today. I'm excited to continue exploring these topics in the coming years!
Finally, special thanks to my students; the award belongs to you as well:
- @DakingRai on interpretability
- @yue_murong21307, @aghzalm, and @salokr_deep on LLMs and LLM agents
- Hao Yan on code generation
(Pictures taken when we celebrated Saurabh's graduation in a lovely Mexican restaurant)
Daking Rai retweeted
🚨New paper: Can Language Model Agents be Helpful Circuit Explainers in Mechanistic Interpretability?
Localizing circuits is automated. But explaining what each localized component does is still manual and hard to scale.
Can an LM agent do it? We build a benchmark to find out🧵
Daking Rai retweeted
Great paper! Happy to see other papers advocating for a data driven approaches in interpretability
🚨 New paper: Data-driven Circuit Discovery for Interpretability of Language Models 🚨
Do circuits actually explain how language models (LM) implement a task?
In mechanistic interpretability, the goal of circuit study is to discover a “circuit” that is responsible for implementing a “task”.
But we find that existing methods often discover circuits that are:
❌ not general task circuits: they do not capture the full range of mechanisms LMs uses across the task.
Instead, they find:
✅ dataset-specific circuits: they explain how the model processes the examples used for circuit discovery.
✅ mixed-mechanism circuits: consisting of multiple independent mechanisms mixed in a single circuit.
1/🧵
Daking Rai retweeted
Trying to find circuits in language models?
Bad news is that existing methods DO NOT give you what you think...
Check out new work with @DakingRai @ZiyuYao where we show fundamental limitations of existing approaches and propose a new data-driven framework for circuit discovery
🚨 New paper: Data-driven Circuit Discovery for Interpretability of Language Models 🚨
Do circuits actually explain how language models (LM) implement a task?
In mechanistic interpretability, the goal of circuit study is to discover a “circuit” that is responsible for implementing a “task”.
But we find that existing methods often discover circuits that are:
❌ not general task circuits: they do not capture the full range of mechanisms LMs uses across the task.
Instead, they find:
✅ dataset-specific circuits: they explain how the model processes the examples used for circuit discovery.
✅ mixed-mechanism circuits: consisting of multiple independent mechanisms mixed in a single circuit.
1/🧵
In our new paper (arxiv.org/pdf/2605.09129v1), we also make similar observations to the paper in parallel --- that 𝐋𝐌𝐬 𝐮𝐬𝐞 𝐦𝐮𝐥𝐭𝐢𝐩𝐥𝐞 𝐝𝐢𝐬𝐭𝐢𝐧𝐜𝐭 𝐜𝐢𝐫𝐜𝐮𝐢𝐭𝐬 𝐟𝐨𝐫 𝐢𝐦𝐩𝐥𝐞𝐦𝐞𝐧𝐭𝐢𝐧𝐠 𝐭𝐚𝐬𝐤.
We find that the existing circuit discovery methods do not discover:
❌ General task circuits: they do not capture the full range of mechanisms LMs uses across the task.
Instead, they find:
✅dataset-specific circuits: they explain how the model processes the examples used for circuit discovery.
✅ mixed-mechanism circuits: consisting of multiple independent mechanisms mixed in a single circuit.
🧵 More in our paper thread: teapaw.org/DakingRai/status/20553…
Does mechanistic interpretability really find the circuit?
Our new paper, "All Circuits Lead to Rome: Rethinking Functional Anisotropy in Circuit and Sheaf Discovery for LLMs," (Accepted by ICML 2026) suggests the answer may be: not always.
A common implicit assumption in mechanistic interpretability is that a model's behavior is explained by the circuit — a sparse, canonical, almost-unique mechanism.
Instead, for the same LLM task, we find multiple circuits/sheaves that are:
✅ faithful
✅ sparse
✅ structurally different
✅ low-overlap
This means a discovered circuit may not be the unique mechanism behind a behavior, but one realization among many possible mechanisms. We call for rethinking how circuit/sheaf discovery results should be interpreted and evaluated.
Huge thanks to my amazing collaborators: @frankniujc, @YutongYin774638, and @zhaoran_wang
Paper: arxiv.org/abs/2605.12671
#MechanisticInterpretability #LLM #AI #MachineLearning
7/8 Our experimental results show that:
1. DCD recovers more faithful and sparser circuits on mixed-task datasets than standard hypothesis-driven methods.
2. It also produces circuits with clearer specialization: different circuits explain different subsets of examples.
The broader takeaway:
1. Human task labels do not always match how language models organize computation internally.
2. To understand models mechanistically, we should let the model’s internal structure help define the scope of explanation.
8/8 Paper: arxiv.org/pdf/2605.09129v1
Grateful to my amazing collaborators: @ZiyuYao , @megamor2 !
We'd love to hear your thoughts — feedback and comments welcome!
Daking Rai retweeted
Neural networks might speak English, but they think in shapes.
Understanding their rich *neural geometry* is key to understanding how they work – and to debugging and controlling them with precision.
Starting today, we’re releasing a series of posts on this research agenda. 🧵
Daking Rai retweeted
#COLM2025 Prof. Greg Durrett presenting "LLM Reasoning Beyond Scaling" @gregd_nlp
Daking Rai retweeted
🚨XLLM-Reason-Plan Workshop is happening right now at @COLM_conf !
Join us at 520F🙌
Daking Rai retweeted
⏰ Only 9 days away!
Join us at @COLM_conf on October 10 for the first workshop on the application of LLM explainability to reasoning and planning.
Featuring:
📑 20 poster presentations
🎤 9 distinguished speakers
View our schedule at tinyurl.com/xllm-workshop.
🚨 New NeurIPS 2025 Paper 🚨
Does 0% accuracy mean a language model (LM) has no correct mechanism for the task? 🤔 We investigated this question on the balanced parentheses task and uncover surprising insights:
1️⃣ Even at 0%, models can contain mechanisms that solve the task with high accuracy—but they’re overshadowed by the faulty ones.
2️⃣ Building on this, we introduce RaSTEER to improve model performance by amplifying the output of sound mechanisms —yielding dramatic performance boosts of up to 100%. 🧵(1/9)
Paper link: arxiv.org/pdf/2507.00322
(8/9) RaSTEER generalizes to arithmetic reasoning.
We consider a two-operand arithmetic reasoning task (+, -, %, x) using three models: GPT-2 XL, Pythia-6.9b, and GPT-3 8b. RaSTEER yields performance improvements across most arithmetic operations for all three models, with the largest gain of 20.25% observed in Pythia-6.9b for multiplication.
(9/9) 🙏 Thanks for reading this far! If you found this interesting, be sure to check out the full paper, and feel free to contact me with any questions or clarifications. A huge thanks to my advisor @ziyuyao and collaborators Samuel Miller & @kevinmoran — this work wouldn’t have been possible without them.
Paper: arxiv.org/pdf/2507.00322