Aashaka Shah (@aashaka_) profile picture

Optimizing ML systems at Microsoft Research

Joined April 2015
Photo by Aashaka Shah (@aashaka_)Photo by Aashaka Shah (@aashaka_)Photo by Aashaka Shah (@aashaka_)Photo by Aashaka Shah (@aashaka_)Photo by Aashaka Shah (@aashaka_)Photo by Aashaka Shah (@aashaka_)
Aashaka Shah retweeted
Today we're announcing our $1B Series A, led by @kleinerperkins and @clearlake. We believe making frontier AI R&D more accessible is the most direct path to AI that solves humanity's most pressing problems across science and technology.
50
61
57
744
223,025
Aashaka Shah retweeted
Introducing Fern, Mirendil’s first model: pretrained from scratch, with up to 10M tokens of context! That’s ~0.5M lines of code (roughly Megatron-LM scale) or days of an agent’s work, all in context at once. Its first version, Fern 0.1, had a short round of post-training 🚢
133
202
60
2,341
217,925
One person pretraining 🤯
We formally announced Mirendil less than 4 months ago. Today, I'm excited to share an important milestone for: Pretraining at scale without a pretraining team. Mirendil's mission is to solve Recursive Self-Improvement (RSI) and use it to democratize frontier AI development, ultimately accelerating science and technology. At Mirendil, when we think about RSI, we think holistically about the company as an intelligent entity that self-improves toward its goal. From that perspective, the most powerful notion of acceleration is reducing the time and resources (number of human experts, compute, and other costs) needed to achieve ambitious goals. Given that pretraining is perhaps the most important yet most challenging part of frontier AI development, we decided to run an ambitious controlled experiment: Starting fully from scratch, can we significantly reduce the time, number of human experts, and compute resources needed to go all the way to successfully pretraining a strong coding model at scale? Mirendil-Base is the result of this experiment. Our first pretrained model was built by one researcher in 3.5 months, with less than $10M in compute. This includes all the research and small-scale experiments it took to get there, as well as the final training run. The recipe holds up as the models get bigger. For reference, Mirendil-Base is in the 100B-500B parameter range, Mirendil-Base-Large in the 500B-2T range, and Mirendil-Base-XL in the 2T-10T range. We're beginning to see a world where a single researcher, empowered by AI, can take on work that once required an entire research organization. As these systems improve, the bottleneck shifts from the size of a team to the ambition and judgment of the people directing them. This has profound implications for who gets to build AI and, ultimately, who gets to push the frontiers of science and technology forward. We’ll share independent evaluations, post-training benchmarks and API-access sooner than you think so stay tuned!
2
175
My team at Microsoft Research is hiring a senior researcher to conduct research on optimizing ML systems. If you care about exploring performance tradeoffs in LLM inference and post-training and building efficient systems, I strongly recommend you apply! tinyurl.com/3yned3ta
1
1
5
662
We are hiring a Senior RSDE in my team at Microsoft Research. If you have an ML Systems background and are excited about optimizing throughput and latency of large-scale LLM serving and post-training systems, please directly reach out to me! jobs.careers.microsoft.com/g…
215
I'm honored and thrilled to be serving as the PC Co-Chair for ACM SoCC'24 (@ACMSoCC) with @palvaro. While we work on getting the details up on the website, here are the key dates: * Abstracts due: July 8th * Papers due: July 15th Looking forward to your cool submissions!
1
14
1
61
9,657
Thank you so much Vijay. I am grateful to have been a part of the @UTSASLab!
This Post is from an account that no longer exists. Learn more
1
1
15
2,737
I gave a guest lecture in Rachee’s class on Intro to Networking at Cornell today! I talked about some of the work we have been doing at RiSE in @MSFTResearch on optimizing collective communication in large-scale ML systems. Thanks for inviting me @rachee_singh !
2
4
40
4,688
I was also really impressed with the questions that the undergrads asked during the talk - very on-point and informed!
1
2
425
Aashaka Shah retweeted
Chandrayaan-3 Mission: 'India🇮🇳, I reached my destination and you too!' : Chandrayaan-3 Chandrayaan-3 has successfully soft-landed on the moon 🌖!. Congratulations, India🇮🇳! #Chandrayaan_3 #Ch3
67,520
263,142
43,156
805,192
71,326,672
Thank you Rachee. It was a pleasure working with everyone! The black slide says it all - collective algorithms (used in distributed ML communication) need to be topology-aware and data-size-aware.
An excellent talk by Aashaka (@aashaka_) at NSDI 2023. Aashaka presented TACCL (usenix.org/conference/nsdi23…), a system that synthesizes algorithms for collective communication for distributed ML training. Joint work with @madanMus @vj_chidambaram @cowanmeg27 and others not on Twitter
1
1
16
2,584
We build TACCL, which uses a novel abstraction of communication sketches to specify intuitive parts of the communication, around which we then synthesize data transfer algorithms using an interesting stage-wise synthesis technique.
1
1
382
TACCL can synthesize efficient algorithms that outperform state-of-the-art and also provide significant end-to-end performance improvements when used in distributed ML models :)
1
1
349
So happy that I got to give a talk at @DeepMind today!! This is the stuff that dreams are made of :) I talked about my work on memory optimization for deep learning training (MONeT). (Arxiv: arxiv.org/pdf/2010.14501.pdf, Github: github.com/utsaslab/MONeT) Here are the main insights:
6
16
1
251
We can run ResNet50 using 3x lower memory for ~12% compute overhead with MONeT schedules!
1
9
Reiterating a bit, I had been looking forward to this talk with a lot of excitement, and it finally happened 😁
6
On Sat, Oct 30, 10am US/Eastern, to dispel the vast misinformation out there, I will answer questions about applying to US computer science PhD programs. Especially hoping to reach HUGs, students w/out access to research, etc. Please help spread the word! docs.google.com/forms/d/e/1F…
3
82
7
169
Undergrads who are getting started with research often get stuck because they are unfamiliar with various tools. I now point them to this very useful course: missing.csail.mit.edu/
8
245
21
1,064