
Associate Professor @gatech_scs @gtcomputing | PostDoc @Berkeley_EECS @ucbrise | ML/LLM Systems: cloud & edge.
Atlanta, GA
Joined December 2012
- Tweets255
- Following309
- Followers578
- Likes896
Alexey Tumanov retweeted
It is good to finally see the open-source community finally catching up ~1.5 years after we originally published the recipe for long context serving. Still missing several critical components - good for microbenchmarking not for end-to-end deployments. arxiv.org/pdf/2409.17264
Alexey Tumanov retweeted
Five months ago, we started Avartha with the belief that better systems can unlock the next generation of real-time applications. Today, we’re announcing Ava Zero, our voice agent built entirely on open models. Ava ranks #1 on @SierraPlatform voice agent benchmark, ahead of OpenAI.
Incredibly proud of our team. We have some of the best systems builders in the world, and we’re just getting started.
Introducing Avartha.
We're building the infrastructure to bring frontier-grade intelligence to real-time applications. Our first release, Ava Zero, is a voice agent entirely running open models using our inference infrastructure and ranks #1 on Sierra's tau-voice benchmark with highest task completion rate and generation quality and just costs $2.5 per hour.
Open models are now at the frontier of voice AI.
Alexey Tumanov retweeted
LLM inference is quickly becoming a global infrastructure problem. Leveraging AI agents to optimize these systems is the natural way to accelerate their development. But AI driven optimization requires a fast, cheap, and accurate evaluation mechanism. 🧵
Systems research is changing: AI is enabling white-box algorithm discovery, not just black-box tuning.
Our new ADRS paper evaluates three frameworks and shows consistent gains over human designs—up to 13× speedup in load balancing for MoE experts and 35% lower cloud costs across different cloud regions.
Alexey Tumanov retweeted
After hitting evaluation puzzles like this in our own work, we analyzed patterns across LLM inference papers and identified 8 systematic evaluation issues that can make performance comparisons misleading. We have compiled a practical evaluation checklist to help avoid these pitfalls.
📄 arxiv.org/abs/2507.09019
We're also releasing Veeksha, our comprehensive LLM inference evaluation framework, later this month to help the community design more robust benchmarks! 🛠️
What evaluation issues have you discovered in your systems work? Let's learn from each other's mistakes!
@nitinkedi @jayashree2912 @kwatra @thisissouvikk @ramaramjee @alsched @gtcomputing @MSFTResearch @intel
Alexey Tumanov retweeted
Interesting work on long context inference from @nvidia, where they scale KV parallelism on gb200-nvl72 systems! To learn more about accelerating long context inference and trade-offs between different parallelism dimensions checkout out our paper, Medha: arxiv.org/abs/2409.17264
What if you could ask a chatbot a question the size of an entire encyclopedia—and get an answer in real time?
Multi-million token queries with 32x more users are now possible with Helix Parallelism, an innovation by #NVIDIAResearch that drives inference at huge scale.
🔗 nvda.ws/4eCXxqh
Alexey Tumanov retweeted
Congratulations 👏 to our faculty who were recognized on the Spring 2025 CIOS Honor Roll for their outstanding teaching and educational impact:
Assoc. Prof. Alexey Tumanov and Asst. Prof. Jan Van Den Brand!
Alexey Tumanov retweeted
Full code 🔓 github.com/sachitkuhar/PLUM
Collaboration with @jinga_lala1 and @alsched. (6/6)
#EfficientAI #EdgeAI #Quantization #TMLR #AI #GaTech #GeorgiaTech
Alexey Tumanov retweeted
Super excited to share another incredible systems that we have built over the past two years! Training giant foundation models (like Llama-3 405B) costs a FORTUNE 💰 (millions of dollars)! Optimizing the training "recipe" (parallelism, memory tricks, etc.) is critical but incredibly complex. The wrong choices can waste millions. How do we find the best setup without burning GPUs?
This is the problem we tackle with Maya, GPU cluster emulation tool -- have a 1000 GPU job, want to know how it would will perform, all you need is a one cpu and Maya virtual GPU runtime ✨
Arxiv: arxiv.org/abs/2503.20191
Code: Coming soon! 🧵
Alexey Tumanov retweeted
Super long-context models with context window spanning millions of tokens are becoming commonplace (@GoogleDeepMind Gemini, @xai Grok 3, @Alibaba_Qwen Qwen2.5). But efficiently serving these models is tough, especially alongside short requests. Head-of-Line (HOL) blocking becomes a major issue, hurting latency for everyone.
We present Medha, a system designed to handle this mix efficiently. Achieving 30x lower latency, and 5x higher throughput compared to the state-of-the-art. Full paper: arxiv.org/pdf/2409.17264. 🧵
Alexey Tumanov retweeted
Maya offers a transparent, accurate, and efficient way to model and optimize large-scale DL training without needing expensive hardware clusters for exploration. A crucial step towards sustainable AI!
Read the paper: arxiv.org/abs/2503.20191
Work done with @Y_Srihas , @1ntEgr8 , Hakesh Darapaneni, Mitali Meratwal, Shivam Mittal, Pranavi Bajjuri, Srinivas Sridharan, @alsched at @GeorgiaTech @NVIDIA @NVIDIAAI
Alexey Tumanov retweeted
Sequence pipeline parallelism being rapidly adopted for extreme long context inference in the industry! Checkout our paper on system design for long context inference for more details arxiv.org/abs/2409.17264
We're leveling up the game with our latest open-source models, Qwen2.5-1M ! 💥 Now supporting a 1 MILLION TOKEN CONTEXT LENGTH 🔥
Here's what’s new:
1️⃣ Open Models: Meet Qwen2.5-7B-Instruct-1M & Qwen2.5-14B-Instruct-1M —our first-ever models handling 1M-token contexts! 🤯
2️⃣ Lightning-Fast Inference Framework: We’ve fully open-sourced our inference framework based on vLLM , integrated with sparse attention methods. Experience 3x to 7x faster processing for 1M-token inputs! ⚡⚡
3️⃣ Tech Deep Dive: Check out our detailed Technical Report for all the juicy details behind the Qwen2.5-1M series! 📊
📖 Technical Report: qianwen-res.oss-cn-beijing.a…
📄 Blog: qwenlm.github.io/blog/qwen2.…
Experience Qwen2.5-1M live:
👉 Play with Qwen2.5-Turbo supporting 1M tokens in Qwen Chat (chat.qwenlm.ai)
👉 Try it on Huggingface (huggingface.co/collections/Q…)
👉 Or head over to Modelscope (modelscope.cn/collections/Qw…)
Alexey Tumanov retweeted
At SoCC’24, Anastasia Ailamaki from EPFL will give a keynote on how disaggregated memory resources are becoming the norm and how this “new memory wall” affects database system design. This talk will be amazing, make sure to be there!!
acmsocc.org/2024/keynotes.ht…
Alexey Tumanov retweeted
⚡ Speed Meets Accuracy:
Unlike approximation-based methods, Mnemosyne achieves exact inference—ensuring that the generated output remains precise, even when processing 10 million tokens by effectively combining all these parallelization techniques to scale up to hundred of GPUS
Alexey Tumanov retweeted
Replying to @Google
@Google has silently but surely developed an edge over @OpenAI. Long context processing seems to be the key to Google's AI strategy. NotebookLM is a prime example of what long context processing can unlock. In our latest paper, we talk about how systems can be built to support multi-million context length matching Google's capabilities. In case you missed the paper, here is the NotebookLM generated podcast!
Podcast: notebooklm.google.com/notebo…
Arxiv: arxiv.org/abs/2409.17264
Alexey Tumanov retweeted
🔗 Curious to learn more? Dive into our paper to explore the technical details behind Mnemosyne: arxiv.org/abs/2409.17264v1. Join work between @gtcomputing, @Microsoft and @UCSDJacobs with amazing Esha Choukse, @alsched, @ramaramjee, @Junda_Chen_, Íñigo Goiri & Chaojie Zhang!
Alexey Tumanov retweeted
Three SCS faculty members were recognized by their students for outstanding teaching and educational impact. Congratulations to @AshutoshDhekne, @alsched, and Umakishore Ramachandran 👏👏👏
blog.ctl.gatech.edu/2024/05/…
Alexey Tumanov retweeted
Replying to @agrawalamey12 @MLSysConf
Learn more about our lab: gatech-sysml.github.io, microsoft.com/en-us/research…
Alexey Tumanov retweeted
11/ Ready to dive deeper? Access our code here: 🔗 github.com/microsoft/vidur. For a detailed exploration of our methodology and findings, read our paper: 🔗arxiv.org/abs/2405.05465.
Alexey Tumanov retweeted
Great to see chunked prefills from Sarathi getting adopted in DeepSpeed as Dynamic SplitFuse. We provide more insights about the efficacy of this mechanism under different workload scenarios in our paper arxiv.org/abs/2308.16369.
Introducing DeepSpeed-FastGen 🚀
Serve LLMs and generative AI models with
- 2.3x higher throughput
- 2x lower average latency
- 4x lower tail latency
w. Dynamic SplitFuse batching
Auto TP, load balancing w. perfect linear scaling, plus easy-to-use API
github.com/microsoft/DeepSpe…



















