Today we're announcing our $1B Series A, led by @kleinerperkins and @clearlake.
We believe making frontier AI R&D more accessible is the most direct path to AI that solves humanity's most pressing problems across science and technology.
Aashaka Shah retweeted
Introducing Fern, Mirendil’s first model: pretrained from scratch, with up to 10M tokens of context!
That’s ~0.5M lines of code (roughly Megatron-LM scale) or days of an agent’s work, all in context at once.
Its first version, Fern 0.1, had a short round of post-training 🚢
One person pretraining 🤯
We formally announced Mirendil less than 4 months ago. Today, I'm excited to share an important milestone for:
Pretraining at scale without a pretraining team.
Mirendil's mission is to solve Recursive Self-Improvement (RSI) and use it to democratize frontier AI development, ultimately accelerating science and technology. At Mirendil, when we think about RSI, we think holistically about the company as an intelligent entity that self-improves toward its goal. From that perspective, the most powerful notion of acceleration is reducing the time and resources (number of human experts, compute, and other costs) needed to achieve ambitious goals.
Given that pretraining is perhaps the most important yet most challenging part of frontier AI development, we decided to run an ambitious controlled experiment: Starting fully from scratch, can we significantly reduce the time, number of human experts, and compute resources needed to go all the way to successfully pretraining a strong coding model at scale?
Mirendil-Base is the result of this experiment. Our first pretrained model was built by one researcher in 3.5 months, with less than $10M in compute. This includes all the research and small-scale experiments it took to get there, as well as the final training run.
The recipe holds up as the models get bigger. For reference, Mirendil-Base is in the 100B-500B parameter range, Mirendil-Base-Large in the 500B-2T range, and Mirendil-Base-XL in the 2T-10T range.
We're beginning to see a world where a single researcher, empowered by AI, can take on work that once required an entire research organization. As these systems improve, the bottleneck shifts from the size of a team to the ambition and judgment of the people directing them. This has profound implications for who gets to build AI and, ultimately, who gets to push the frontiers of science and technology forward.
We’ll share independent evaluations, post-training benchmarks and API-access sooner than you think so stay tuned!
2
175
My team at Microsoft Research is hiring a senior researcher to conduct research on optimizing ML systems. If you care about exploring performance tradeoffs in LLM inference and post-training and building efficient systems, I strongly recommend you apply! tinyurl.com/3yned3ta
We are hiring a Senior RSDE in my team at Microsoft Research. If you have an ML Systems background and are excited about optimizing throughput and latency of large-scale LLM serving and post-training systems, please directly reach out to me!
jobs.careers.microsoft.com/g…
215
Thank you so much Vijay. I am grateful to have been a part of the @UTSASLab!
This Post is from an account that no longer exists. Learn more
I gave a guest lecture in Rachee’s class on Intro to Networking at Cornell today! I talked about some of the work we have been doing at RiSE in @MSFTResearch on optimizing collective communication in large-scale ML systems. Thanks for inviting me @rachee_singh !
Aashaka Shah retweeted
Chandrayaan-3 Mission:
'India🇮🇳,
I reached my destination
and you too!'
: Chandrayaan-3
Chandrayaan-3 has successfully
soft-landed on the moon 🌖!.
Congratulations, India🇮🇳!
#Chandrayaan_3
#Ch3
Thank you Rachee. It was a pleasure working with everyone! The black slide says it all - collective algorithms (used in distributed ML communication) need to be topology-aware and data-size-aware.
An excellent talk by Aashaka (@aashaka_) at NSDI 2023. Aashaka presented TACCL (usenix.org/conference/nsdi23…), a system that synthesizes algorithms for collective communication for distributed ML training. Joint work with @madanMus @vj_chidambaram @cowanmeg27 and others not on Twitter
We build TACCL, which uses a novel abstraction of communication sketches to specify intuitive parts of the communication, around which we then synthesize data transfer algorithms using an interesting stage-wise synthesis technique.
1
1
382
So happy that I got to give a talk at @DeepMind today!! This is the stuff that dreams are made of :)
I talked about my work on memory optimization for deep learning training (MONeT).
(Arxiv: arxiv.org/pdf/2010.14501.pdf,
Github: github.com/utsaslab/MONeT)
Here are the main insights:
Aashaka Shah retweeted
On Sat, Oct 30, 10am US/Eastern, to dispel the vast misinformation out there, I will answer questions about applying to US computer science PhD programs. Especially hoping to reach HUGs, students w/out access to research, etc. Please help spread the word!
docs.google.com/forms/d/e/1F…
Aashaka Shah retweeted
Undergrads who are getting started with research often get stuck because they are unfamiliar with various tools. I now point them to this very useful course: missing.csail.mit.edu/















