Ram Ramjee (@ramaramjee) profile picture

Tweets on (computer) science, philosophy, finance, weight lifting, ...

Bengaluru, India
Joined November 2008
Photo by Ram Ramjee (@ramaramjee)Photo by Ram Ramjee (@ramaramjee)Photo by Ram Ramjee (@ramaramjee)Photo by Ram Ramjee (@ramaramjee)Photo by Ram Ramjee (@ramaramjee)Photo by Ram Ramjee (@ramaramjee)Photo by Ram Ramjee (@ramaramjee)Photo by Ram Ramjee (@ramaramjee)
Ram Ramjee retweeted
I am presenting this work, "No Request Left Behind: Tackling Heterogeneity in Long-Context LLM Inference with Medha" at @ACMSoCC today, 14:00 local time. It has already been adopted by OSS from our earlier preprint. Key contributions: * chunked prefill unrolls the prefill computation in time abandoning the SOTA-standard spatial context parallelism (CP) * demonstrating that Sarathi-style chunked prefill read amplification cost can be fully hidden, because arithmetic intensity depends only on chunk size (the paper proves this) * contributing a 3D parallelism mechanism (TP+SPP+KVP) that fixes both TTFT and TBT (TPOT) latency challenges for long context (> 1M tokens) * LARS scheduler : prioritizing least remaining slack per unit total work -- a novel scheduling technique that outperforms much stronger baselines than the typical FCFS scapegoat. paper: dl.acm.org/doi/10.1145/38157… arxiv: arxiv.org/abs/2409.17264v4 This work is pure, approximation-free play. It's becoming increasingly pertinent in contexts extending beyond LLM inference serving alone. Perhaps the most foundational aspect of this work for me, from scheduling perspective, is that it achieves liquidity in scheduler decision space. It doesn't get more fundamental than this. That enables mixed batching, preemption, dropping the convoy effect, and LARS.
Thank you for pointing out the inadequacy of current SOTA implementations of long context support. Our team has demonstrated the superiority of chunked prefill mechanism for enabling preemptive scheduling in support of long context requests (10M tokens): arxiv.org/abs/2409.17264
1
3
9
446
Five months ago, we started Avartha with the belief that better systems can unlock the next generation of real-time applications. Today, we’re announcing Ava Zero, our voice agent built entirely on open models. Ava ranks #1 on @SierraPlatform voice agent benchmark, ahead of OpenAI. Incredibly proud of our team. We have some of the best systems builders in the world, and we’re just getting started.
Introducing Avartha. We're building the infrastructure to bring frontier-grade intelligence to real-time applications. Our first release, Ava Zero, is a voice agent entirely running open models using our inference infrastructure and ranks #1 on Sierra's tau-voice benchmark with highest task completion rate and generation quality and just costs $2.5 per hour. Open models are now at the frontier of voice AI.
5
6
50
47,490
Ram Ramjee retweeted
Our paper, LLM-42, will appear at SOSP 2026, happening at the end of September in Prague, Czechia 🇨🇿. Check out the paper to learn more about the work! Paper: arxiv.org/abs/2601.17768 Code: github.com/microsoft/llm-42 Conference: sigops.org/s/conferences/sos…
2
9
1,264
Interesting work on long context inference from @nvidia, where they scale KV parallelism on gb200-nvl72 systems! To learn more about accelerating long context inference and trade-offs between different parallelism dimensions checkout out our paper, Medha: arxiv.org/abs/2409.17264
What if you could ask a chatbot a question the size of an entire encyclopedia—and get an answer in real time? Multi-million token queries with 32x more users are now possible with Helix Parallelism, an innovation by #NVIDIAResearch that drives inference at huge scale. 🔗 nvda.ws/4eCXxqh
5
14
1,081
How well do foundation models like GPT-4o, o4-mini, and Gemini handle classic computer vision tasks? 🧐 We put them to the test on tasks like semantic segmentation and dense depth prediction. 🔗Interactive visualizations + details: fm-vision-evals.epfl.ch/
We benchmarked leading multimodal foundation models (GPT-4o, Claude 3.5 Sonnet, Gemini, Llama, etc.) on standard computer vision tasks—from segmentation to surface normal estimation—using standard datasets like COCO and ImageNet. These models have made remarkable progress; however, it is unclear exactly where they stand in terms of understanding vision in detail. Especially when it comes to tasks beyond question-answering. How well do they understand an object's segments or geometry? Our analyses yield an assessment that is quantitatively and qualitatively detailed and is compatible with evaluations developed in the field of computer vision over the past decades. Observed trends: 🔹 The foundation models consistently underperform task-specific SOTA models across all tasks. However, they are respectable generalists, which is remarkable as they are presumably trained primarily on image-text-based tasks. 🔹 They perform semantic tasks notably better than geometric ones. 🔹 GPT-4o performs the best among non-reasoning models, getting the top position in 4 out of 6 tasks. 🔹 Reasoning models, e.g., o3, show improvements in geometric tasks. 🔹 The 'image generation' models, e.g., GPT-40 Image Generation, which have been natively trained multimodally, exhibit quirks. E.g., hallucinated objects, misalignment between the input and output, etc. 🔹 While the prompting techniques affect performance, better models exhibit less sensitivity to variations in prompts. We control for the variance introduced by the prompting methods in our experiments. 🌐 Detailed analyses, visualizations: fm-vision-evals.epfl.ch ⌨️ code: github.com/EPFL-VILAB/fm-vis… 🧵 1/n
1
6
699
RT @agrawalamey12: Super cool paper, deepseek style communication-computation overlap on steroids! Deepseek creates separate microbatches t…
This quoted post is unavailable.
1
28
Super long-context models with context window spanning millions of tokens are becoming commonplace (@GoogleDeepMind Gemini, @xai Grok 3, @Alibaba_Qwen Qwen2.5). But efficiently serving these models is tough, especially alongside short requests. Head-of-Line (HOL) blocking becomes a major issue, hurting latency for everyone. We present Medha, a system designed to handle this mix efficiently. Achieving 30x lower latency, and 5x higher throughput compared to the state-of-the-art. Full paper: arxiv.org/pdf/2409.17264. 🧵
1
13
2
31
3,721
Happy to share that this has now been accepted in Neurips 2024! Check it out if you are curious to see how compressed LLMs should be evaluated! #NeurIPS2024 teapaw.org/abhinavdutta555/status…
🚨 Are LLM compression methods (𝘲𝘶𝘢𝘯𝘵𝘪𝘻𝘢𝘵𝘪𝘰𝘯, 𝘱𝘳𝘶𝘯𝘪𝘯𝘨, 𝘦𝘢𝘳𝘭𝘺 𝘦𝘹𝘪𝘵) too good to be true and are existing eval metrics sufficient? We've looked into it in our latest research at @MSFTResearch 🧵 (1/n) arxiv.org/abs/2407.09141
1
7
801
🚨 Are LLM compression methods (𝘲𝘶𝘢𝘯𝘵𝘪𝘻𝘢𝘵𝘪𝘰𝘯, 𝘱𝘳𝘶𝘯𝘪𝘯𝘨, 𝘦𝘢𝘳𝘭𝘺 𝘦𝘹𝘪𝘵) too good to be true and are existing eval metrics sufficient? We've looked into it in our latest research at @MSFTResearch 🧵 (1/n) arxiv.org/abs/2407.09141
2
7
4
19
5,414
🚀 Introducing Metron: Redefining LLM Serving Benchmarks! 📊 Tired of misleading metrics for LLM performance? Our new paper introduces a holistic framework that captures what really matters - the user experience! 🧠💬 github.com/project-metron/me… #LLM #AI #Benchmark
2
15
5
33
6,710
Did you ever feel that @chatgpt is done generating your response and then suddenly a burst of tokens show up? This happens when the serving system is prioritizing someone else’s request before generating your response. But why? well to reduce cost. 🧵
1
5
3
26
7,456
[proud advisor moment] Happy to see @agrawalamey12 work in close collaboration with MSR-India already enjoying adoption and impact at @anyscalecompute! Please refer to Sarathi-Serve (arxiv.org/abs/2403.02310) for the better reference and coverage of the chunked prefill mechanism.
One of @vllm_project's strengths is that it exposes the ability to trade off latency and throughput. However, higher qps regimes cause significant latency degradation. The underlying reason has to do with inference taking place in two stages: prefilling (processing the input context) and decoding (generating output tokens). Normally in vLLM, these stages don't happen at the same time and so an incoming request will trigger prefill computations that interrupt ongoing decoding (leading to a spike in latency). Chunked prefill breaks prefill computations into multiple "chunks" and batches them along with ongoing decoding computations to avoid interruptions. In a low qps regime, this makes no difference, but in the high qps regime with constant interrupts, this is a big deal. More details in this paper arxiv.org/pdf/2308.16369 by @agrawalamey12. Also in this RFC github.com/vllm-project/vllm….
2
21
1,414
We at @MSFTResearch and @GeorgiaTech believe that running LLM's shouldn't be so expensive 💵 So, we built a tool 🛠️ that will enable you to run it cheaper, make it cheaper. Introducing Vidur👳🏽, the first LLM Inference System simulator. #mlsys #vllm #llm #llama #gpt
1
3
15
552
1/ LLM inference systems are like high-performance engines ⚙️—complex, powerful, and full of intricate settings. Efficiently deploying them to maximize GPU performance is a challenge typically tackled by experts at orgs like @OpenAI and @AIatMeta 🚀. 🧵
1
13
39
3,247
Ever wondered why @OpenAI charges 2x price for output tokens compared to input? Turns out that an output token can be up to 200x more compute time than an input token. Why? We explored this phenomenon during my internship at @MSFTResearch. 🧵
7
46
2
373
78,213
The Doctor said he needed to cut the finger to remove the ring. The Goldsmith said he needed to cut the ring to remove it. What did the Engineer do?
2,762
131,288
8,490
322,580
Tip: Find YOUR Perfect Squat, Not Someone Else's t-nation.com/training/tip-fi…
1
4
#NIPS2017 #MachineLearning – at Long Beach, CA
3
29
4
70
Ram Ramjee retweeted
Be interesting. Avoid boring people.
4
254
30
647
Infatuation vs. Love
421
102,759
8,537
199,315