Instruction Workshop, NeurIPS 2023 (@itif_workshop) header
Instruction Workshop, NeurIPS 2023 (@itif_workshop) profile picture

The official account of the 1st Workshop on Instruction Tuning and Instruction Following (ITIF), colocated with NeurIPS, in December 2023.

New Orleans
Joined August 2023
Photo by Instruction Workshop, NeurIPS 2023 (@itif_workshop)Photo by Instruction Workshop, NeurIPS 2023 (@itif_workshop)Photo by Instruction Workshop, NeurIPS 2023 (@itif_workshop)Photo by Instruction Workshop, NeurIPS 2023 (@itif_workshop)Photo by Instruction Workshop, NeurIPS 2023 (@itif_workshop)Photo by Instruction Workshop, NeurIPS 2023 (@itif_workshop)Photo by Instruction Workshop, NeurIPS 2023 (@itif_workshop)Photo by Instruction Workshop, NeurIPS 2023 (@itif_workshop)Photo by Instruction Workshop, NeurIPS 2023 (@itif_workshop)Photo by Instruction Workshop, NeurIPS 2023 (@itif_workshop)
Instruction Workshop, NeurIPS 2023 retweeted
🚨 Update: Claude & political elections 1 in 17 conversations is on politics. 1 in 200 is on elections. Glad my colleagues are bringing transparency to their approach here: ▶️ Only ~1 in 50,000 conversations violate our election integrity policy (astroturfing, fake-account networks, etc.) ▶️ New classifiers now catch multi-step misuse, not just single bad requests ▶️ Our latest models score 99% on political evenhandedness. Independent reviewers pressure-tested this. Full blog: anthropic.com/news/2026-us-m…
1
6
31
1,681
Instruction Workshop, NeurIPS 2023 retweeted
What happens to the direction of science when we delegate it to AI? With @seungonekim, we ranked all 6,617 #ICML2026 papers with AI—the AI Awards. Findings: 1⃣Humans & AI disagree. Significantly. 2⃣AI is highly influenced by online hype. 3⃣AI preferred papers w/ perceived impact and generalization across settings, whereas humans preferred novelty. 4 AIs agreed fairly highly with one another. 🔗 research-taste.github.io/ai-… 1/
8
33
7
181
16,530
Instruction Workshop, NeurIPS 2023 retweeted
Excited this is out! My list of fav Opus 5.5 improvements: 1. 40% cheaper than Opus 5. 2. ~30% faster 3. Stronger at agentic coding/computer use than Astra (CursorBench, KWBench, OSWorld) 4. Style and instruction following wayyyy better 5. Science + health capabilities much stronger 6. Significantly more robust model to cyber/bio threats
Introducing Claude Opus 5.5, the first model in our new Claude 5.5 family. It performs at the level of Claude Fable 5.1 for most tasks, and costs 40% less to run than Opus 5.
7
6
68
5,291
Instruction Workshop, NeurIPS 2023 retweeted
This is a thoughtful and appreciated post, but (from my brief tenure) I question that Anthropic is less transparent than OpenAI in many of the most important respects? Where Anthropic does well: - The Anthropic Economic Index is incredibly detailed, and invites public scrutiny - The new Researcher Access Program for external real user log analysis. - Anthropic's Transparency Hub is way more detailed than OpenAI's - Anthropic's Alignment Science and Frontier Red Teaming blogs are super active and dump tons of valuable context on a frequent basis. These are clearly different from the public essays the original poster is talking about, but from my perspective, the above (and especially the second) are really crucial to enable public intellectuals to engage with the impacts of AI themselves- updating priors for themselves. Credit where it's due, all labs are working significantly with many external testing groups, which is great. I think external essays are good, but often they seem to engage mostly other AI safety audiences. Whereas public data, reports, and careful access programs allow the community to independently arrive at conclusions around AI safety and security. Edit: It goes without saying, much more could be done! Definitely invite critical perspectives on this
Because Anthropic has a cultural belief that transparency leads to pressure that would decrease their ability to solve problems internally. They think this is the correct posture. (Yes, there is a steelman for this, but I very strongly disagree with this point of view and I think they are setting themselves up for failure in the long run.) So far they are basically getting rewarded for it, in that they do not incur the penalties that OpenAI takes for being comparatively open and having the debates and discussions directly with the public. That's not to say that there is zero communication from Anthropic. Obviously there are a couple of Ants who make an effort to post on twitter, or who respond to a subset of concerns raised in public, and there are of course many publications from them in the voice of the company. But they are very selective about how they engage, they are capable of keeping internal discussions (almost) completely internal, and their essay culture is for internal consumption and not external consumption. Where they have profound differences of opinion with each other, they are going to discuss them internally and not externally; where they feel they are falling short, they're going to talk about it internally, not externally. I don't think this is totally healthy! I think they tend to get overconfident in their safety posture as a result. In particular I have heard directly from Anthropic employees, or heard implied by Anthropic employees, that they feel the recent GPT cyber issues (e.g. Huggingface and the wiki incident) could not have happened there, or that if something had happened there, that those issues would quickly become public. I feel *very* worried that their culture supports a belief that it is unlikely to happen there, and especially because it seems connected to a meme of "Anthropic is the good guys, and OpenAI is the bad guys; we are unlikely to fail in the way they are." A healthy safety culture has a preoccupation with failure and believes that the organization is always, ALWAYS capable of failing. The Ant-OpenAI rivalry, where the Anthropic CEO is willing to describe the entire staff of the other company as a "gullible bunch," is producing something unhealthy in Anthropic that it would do well to purge.
4
4
63
11,269
Instruction Workshop, NeurIPS 2023 retweeted
Claude formalized Fermat's Last Theorem in 11 days! 🚀 This spans 29k intermediate theorems, and 13M lines of Lean. For scale, Mathlib in it's entirety is ~8k files and 2.2M lines. This shows tremendous promise to automatically formalize massive swathes of modern mathematical literature, starting from the elementary vocabulary in Mathlib. Congratulations to @tianyi_peng who led this effort! anthropic.com/research/forma…
4
6
1
51
7,077
Instruction Workshop, NeurIPS 2023 retweeted
My list of most notable improvements w/ Fable 5.1: 1. Cost efficiency (~25% cheaper in most workloads) 2. Way better at math + Lean generation. 3. Way better at Science (+26% on Terminal Bench Science) 4. False positive Safeguards down 60-80% on bio/cyber. 5. The writing is getting much clearer again. 6. 3D/game code gen: Fable 5.1 remains phenomenal at generating games and 3D apps/visualizations. 7. Wayyyy better at office skills: excel, powerpoint, finance, accounting.
We’re introducing Claude Fable 5.1 and Claude Mythos 5.1. They're the world’s most advanced models for coding and knowledge work.
3
3
57
4,654
Instruction Workshop, NeurIPS 2023 retweeted
This is massive transparency! 🔍🚀 The first privacy-preserving independent access to an AI company's real usage data. And a commitment to release each projects' results. Huge credit to the Societal Impact team for this leap in transparency 🎉
For the first time, we’ve given external researchers a way to study AI’s impacts using real, privacy-preserved Claude usage data. To date, this work has only been possible within AI labs. We can’t tell the whole story alone, so we opened up our tools. anthropic.com/research/enabl…
2
4
55
7,360
Instruction Workshop, NeurIPS 2023 retweeted
Really cool work (and @ShayneRedford is a superstar!) but looking at the data also made me realize how trying to answer one of the most important questions in AI is still constrained to a really thin public record while the much richer picture sits inside a few companies.
1/ How are people really using AI? Today, researchers across @MIT, @Stanford, + 12 academic institutions are launching the Public AI Observatory: public infrastructure for measuring how people actually use AI assistants in the wild. 🔗 ai-observatory.org/ 📜 dataprovenance.org/ai_observ…
1
2
15
913
Instruction Workshop, NeurIPS 2023 retweeted
1/ How are people really using AI? Today, researchers across @MIT, @Stanford, + 12 academic institutions are launching the Public AI Observatory: public infrastructure for measuring how people actually use AI assistants in the wild. 🔗 ai-observatory.org/ 📜 dataprovenance.org/ai_observ…
7
66
25
314
74,244
Instruction Workshop, NeurIPS 2023 retweeted
This year I plan on writing more, on AI, society, and their evolving relationship. To start, some thoughts on the AI+math revolution.
2
4
26
1,978
Instruction Workshop, NeurIPS 2023 retweeted
Life Update: I recently moved to San Francisco and joined @AnthropicAI as a Member of Technical Staff. In this next chapter, I’m fortunate to be working across pretraining and societal impacts. Over the past few years—much to the anxiety of my advisors—my PhD research at @MIT has spanned data, pretraining, post-training, scaling laws, evaluation, and ecosystem safety, including work on transparency, accountability, safe harbors for independent research, and AI flaw reporting. I’ve loved working across the model lifecycle, and I’m excited to continue doing so at Anthropic—thinking about better science, multilinguality, and how we can more rigorously understand the broader impacts of models in the real world. I also plan to remain actively engaged with the academic and public-interest communities working on these questions. The Data Provenance Initiative has several new projects underway that I’m excited to share soon. I’m deeply grateful to my friends, family, advisor @alex_pentland, and collaborators across MIT, Stanford, Google, Cohere, BigScience, and the Data Provenance Initiative who supported me throughout this journey. I feel very lucky to keep working on questions I care about, now from a new vantage point. Please reach out if you’re around SF! Pictured: me paddling into a new life chapter, my new run route, and some fun pins.
51
21
3
719
70,509
Instruction Workshop, NeurIPS 2023 retweeted
I'm publishing a blog post soon on "How To Write Rebuttals in 2026". (read: in the age of AI 📜🤖) Would love feedback from senior academics before I do! (DM me if you're interested--will certainly acknowledge feedback.)
3
4
85
6,697
Instruction Workshop, NeurIPS 2023 retweeted
Launch FLARE-AI 🎆: Flaw & Incident Reporting for AI. Reporting AI flaws is broken: forms are scattered, non-standardized, and reports get siloed instead of reaching everyone who needs them (think universal jailbreaks). We built one open-source place to fix that: create your report + route. Open source + free. 🧵/
2
12
22
6,614
Instruction Workshop, NeurIPS 2023 retweeted
Excited to see our Economies of Open Intelligence work highlighted in Chp. 1 of @StanfordHAI's #AIIndex2026! We release tons of info on the open model ecosystem, using 🤗 HF data. Thank you @russellwald and team!
1
3
12
816
Instruction Workshop, NeurIPS 2023 retweeted
We just released the Google Research Blog for ATLAS 🗺️! Check out for: 1) Multilingual scaling and data mixing laws for 100s of languages 2) "Curse of Multilinguality" modeling 3) Cross-lingual transfer scores 🌎 research.google/blog/atlas-p…
1
5
17
757
Instruction Workshop, NeurIPS 2023 retweeted
The open AI race is becoming more competitive, and more international. Check out @Melissahei's excellent coverage of our new study in the @FT! ft.com/content/931c8218-a9d7…
4
8
1
16
11,336
Instruction Workshop, NeurIPS 2023 retweeted
Who is winning the open AI race? Our new study "Economies of Open Intelligence" maps 2.2B @huggingface downloads across 851k models (2020→2025). 1) Power is rebalancing (US big tech ↓; China + community ↑) 2) Models got big & efficient (MoE, quant, multimodal surge) 3) Intermediaries now matter (adapters/quantizers steer usage) 4) Transparency is slipping /🧵
9
24
6
92
32,474
Instruction Workshop, NeurIPS 2023 retweeted
TL;DR: Seven lessons on collaboration in AI research (slides below). Presented last week as a keynote at @Cohere_Labs Connect 2025. Last week I gave a keynote on Collaboration in AI Research at Cohere Labs Connect 2025, alongside Cohere co-founder @1vnzh, Head of Cohere Labs @mziizm, and Chief AI Officer @jpineau1. I’m grateful for the invitation—and for the many collaborators whose work shaped this talk. Slides: a practical roadmap for starting an interdisciplinary research collaboration. In brief: 1. Ideate clearly. Share a one-pager with potential collaborators, advisors, and critics. 2. Run a pre-mortem. Find your “reviewer #2” early and invite their critiques to sharpen direction. 3. Recruit widely. Don’t let geography or institution limit you; add complementary skills, senior advisors, and junior contributors. 4. Structure incentives. Be explicit about authorship, credit, visibility, and impact; let people own their piece. 5. Communicate early and often. Relationships may be the most durable outcome. Align early on authorship contingencies, timeline, cadence, and responsibilities. 6. Document decisions. Keep a living record of links, choices (and why), and who did what—details slip in larger teams. 7. Focus on infrastructure & visuals. Beyond relationships, open-source tools and clear figures/tables travel the farthest and longest. I also folded in lessons from advisors, mentors, and collaborators including @alex_pentland , @sarahookr, @percyliang, @BlancheMinerva, @PeterHndrsn, @RishiBommasani, @sayashk, @AnkaReuel, and @RobertMahari and @TobinSouth.
3
9
33
3,253
Instruction Workshop, NeurIPS 2023 retweeted
A new, tractable approach to study scaling laws for larger data mixtures compared to prior art. We achieve significantly better fit ($R^2=0.98$) on multilingual data mixtures with ~50 languages.
1
7
8
2,835
Instruction Workshop, NeurIPS 2023 retweeted
Have you ever wondered how to build target-language LMs most efficiently? 🤨 Will finetuning from other multilingual LMs help? No curse of multilinguality?? See the snapshot below for the answer. What about transferring data to help? What languages help? See our new paper! 👇
📢Thrilled to introduce ATLAS 🗺️: scaling laws beyond English, for pretraining, finetuning, and the curse of multilinguality. The largest public, multilingual scaling study to-date—we ran 774 exps (10M-8B params, 400+ languages) to answer: 🌍Are scaling laws different by language? 🧙‍♂️Can we model the curse of multilinguality? ⚖️Pretrain from scratch or finetune from multilingual checkpoint? 🔀Cross-lingual transfer scores for 1444 lang pairs? 1/🧵
1
7
11
1,532