Jayashree Mohan (@jayashree2912) profile picture

Researcher | Microsoft Research India

Bengaluru, India
Joined November 2015
Photo by Jayashree Mohan (@jayashree2912)Photo by Jayashree Mohan (@jayashree2912)Photo by Jayashree Mohan (@jayashree2912)Photo by Jayashree Mohan (@jayashree2912)Photo by Jayashree Mohan (@jayashree2912)Photo by Jayashree Mohan (@jayashree2912)Photo by Jayashree Mohan (@jayashree2912)Photo by Jayashree Mohan (@jayashree2912)Photo by Jayashree Mohan (@jayashree2912)Photo by Jayashree Mohan (@jayashree2912)
Jayashree Mohan retweeted
TokenWeave – Efficient Compute-Communication Overlap for Distributed LLM Inference. Why? Even with highspeed NVLink on H100 DGX, communication overhead for distributed LLM inference can be > 20 %! Can we recover this overhead? (1/10)
1
5
1
17
1,472
Check out this exciting recent work from our group that shows why legacy eval metrics like accuracy don't completely capture the compressed LLM model behavior and the metrics you should actually look out for! Read details here...
🚨 Are LLM compression methods (𝘲𝘶𝘢𝘯𝘵𝘪𝘻𝘢𝘵𝘪𝘰𝘯, 𝘱𝘳𝘶𝘯𝘪𝘯𝘨, 𝘦𝘢𝘳𝘭𝘺 𝘦𝘹𝘪𝘵) too good to be true and are existing eval metrics sufficient? We've looked into it in our latest research at @MSFTResearch 🧵 (1/n) arxiv.org/abs/2407.09141
10
2,198
Conventional latency and throughput metrics are insufficient in capturing user-facing performance for interactive LLM applications. Introducing our effort : Metron, a benchmark with fluidity metric and fluidity token generation rate that captures user experience!
🚀 Introducing Metron: Redefining LLM Serving Benchmarks! 📊 Tired of misleading metrics for LLM performance? Our new paper introduces a holistic framework that captures what really matters - the user experience! 🧠💬 github.com/project-metron/me… #LLM #AI #Benchmark
1
11
2,181
Jayashree Mohan retweeted
[proud advisor moment] Happy to see @agrawalamey12 work in close collaboration with MSR-India already enjoying adoption and impact at @anyscalecompute! Please refer to Sarathi-Serve (arxiv.org/abs/2403.02310) for the better reference and coverage of the chunked prefill mechanism.
One of @vllm_project's strengths is that it exposes the ability to trade off latency and throughput. However, higher qps regimes cause significant latency degradation. The underlying reason has to do with inference taking place in two stages: prefilling (processing the input context) and decoding (generating output tokens). Normally in vLLM, these stages don't happen at the same time and so an incoming request will trigger prefill computations that interrupt ongoing decoding (leading to a spike in latency). Chunked prefill breaks prefill computations into multiple "chunks" and batches them along with ongoing decoding computations to avoid interruptions. In a low qps regime, this makes no difference, but in the high qps regime with constant interrupts, this is a big deal. More details in this paper arxiv.org/pdf/2308.16369 by @agrawalamey12. Also in this RFC github.com/vllm-project/vllm….
2
21
1,414
Talk to @agrawalamey12 or @nitinkedi at #osdi24 to learn more about our recent work Sarathi-Serve, a scheduler that addresses the latency-throughput tradeoff in LLM serving.
Did you ever feel that @chatgpt is done generating your response and then suddenly a burst of tokens show up? This happens when the serving system is prioritizing someone else’s request before generating your response. But why? well to reduce cost. 🧵
11
2,059
Checkout our recent work on speeding up token generation for LLM inference. See the tweet deck below to learn more! Full paper is now up on arXiv - arxiv.org/pdf/2308.16369.pdf
Ever wondered why @OpenAI charges 2x price for output tokens compared to input? Turns out that an output token can be up to 200x more compute time than an input token. Why? We explored this phenomenon during my internship at @MSFTResearch. 🧵
18
2,572
NEW! Spring Deadline : Eurosys 2023 Consider submitting your work to the Spring deadline (May 11 - abstract, May 18- full paper)
#CFP @EuroSys_conf 2023 will take place in Rome, #Italy. Eurosys adopts a new dual-deadline format. The first of the two deadlines (Spring & Fall) is 11-May-2022 (for abstract). CFP and other details can be found at 2023.eurosys.org/cfp.html#cf…
3
17
The @UTSASLab family 🤗 Successfully defended my PhD thesis today. Mixed feelings! Going to miss the enthusiasm and energy of this lab more than anything! Can’t believe time flew by in the blink of an eye! We missed you @SuprShastri !
5
92
Wohoo. Back after a long break on Twitter! You now know what I was upto 😅 The past 5 years have been a beautiful experience and it’s hard to bid goodbye. So fortunate to have a caring and supportive advisor @vj_chidambaram and group @UTSASLab – at Round Rock, TX
This Post is from an account that no longer exists. Learn more
13
2
133
I am excited to (virtually) present our work on fine-grained DNN checkpointing - CheckFreq at #FAST21 tomorrow! This is work done in collaboration with @MSFTResearch. Tune in to the Transactions & Deduplication session on Wed 24th, 8:30am PST to learn more about our work.. – at Bengaluru, India
1
1
1
43
Jayashree Mohan retweeted
The Student Editorial Board for @jsysresearch is coming together! Featuring @jayashree2912, @iangneal, @SolalPirelli, @Yizhou_Shan, @chinmay, @kkaffes, and more! escholarship.org/uc/jsys/stu…
8
2
25
We were and are very excited by this work. This was long overdue! If you are interested in systems for ML, do give this a read :) – at Udupi, India
This Post is from an account that no longer exists. Learn more
1
45
Wrapping up the year on a good note. Stay tuned for the full version of the paper and presentation! – at Udupi, India
This Post is from an account that no longer exists. Learn more
9
Jayashree Mohan retweeted
I am on the tenure-track job market! I work at the intersection of systems and disciplines such as economics (PhD) and law (post-doc). I would love to join a dept that values interdisciplinary work! Application material: cs.utexas.edu/~shastri/shast…
2
9
33
I need a job that provides sponsored travel to exotic locations. My PhD life seems to be achieving this target really well. – at Moscow, Russia
2
2
55
This was a great experience! Such a well organized conference #heisenbug @HeisenbugConf – at Moscow, Russia
1
2
1
25
Replying to @JFKairport
@JFKairport The staff weren’t helping the situation. In fact, some of them asked me to get the air train when I was just sent back from the air train station. No one has any clue what’s happening there. Please prep the staff to handle the situation better. – at Queens, NY
Replying to @DAMitsDevon
Hi Devon, the Airtrain is temporarily experiencing a service disruption. Please follow the directions from our Airport Customer Experience Specialist (ACES) in the red coats for alternative accomendations and shuttle services.*JB
1
1
I could barely make it to my international flight, which initially denied me boarding. Thanks to the inclement weather, some nice people at the checkin counter escorted me to my delayed flight. – at Queens, NY
2
As I update my CV, I realize that I've spoken about CrashMonkey at 6 different venues so far, including #osdi18 - and there's still no video recording of my talk. I hope at least my next venue @HeisenbugConf does it.
2
12