eng @modal, eecs&business @ucberkeleymet
sf
Joined August 2018
- Tweets71
- Following1.1K
- Followers888
- Likes2.4K
Pinned Tweet
I have spent the last year in the weeds with teams running open LLM inference on Modal.
Modal was already the best place to do this: truly elastic GPUs, fast cold starts, low-overhead routing, and deep observability.
Very excited to raise the bar even higher today.
Auto Endpoints package the Modal inference stack into one command:
modal endpoint create --name agent --model zai-org/GLM-5.2-FP8
One of my favorite things about systems is how often old ideas find new life in completely different concepts (for example, virtual memory inspiring PagedAttention). There’s a lot to learn from decades of system research.
Really cool to see Shreya building a community around this in SF. Highly recommend attending SFSRG!
the future of inference feels increasingly bespoke: continuously optimized for each model*hardware*workload. agents are what make this level of specialization economically viable
really impressive work from @DecagonAI! and cool to see @modal powering the experimentation here
every inference endpoint presents a different optimization problem, shaped by its model architecture, traffic patterns, hardware, and latency constraints.
we built an autoresearch loop to tackle these problems and carry lessons from one investigation into the next, with the broader goal of building systems that get better at research as they accumulate evidence.
are you getting it???
🤖 Made with AI
Today we're announcing that Modal has crossed $1B in annualized run rate, as well as raised a $750M Series D at a $15.75B valuation, led by @Accel.
Since our Series C, usage has tripled and we’ve grown more than 15x year-over-year.
We are deeply grateful to the developers and companies building on Modal. There's still so much more to build, and we can't think of a better time to be building software.
Very proud to have worked on this ambitious and total overhaul of how we route requests for Modal Functions. Check out this great read from @dshaar_ on how we shaved off ~80ms from our p50 function call latencies!
Many of us at Modal have pointed our harnesses to Kimi K3 (running on Modal!) and are _really_ happy with the results. Especially with the blazing fast performance thanks to our custom speculator.
Give it a try!
it's true, 460 tok/s!
Deven Navani retweeted
Modal trained a DFlash speculator that's much faster than MTP, making it a great boost for inference speeds!
Inkling by @thinkymachines is now available on Modal, backed by a custom DFlash speculator for 67% higher throughput and interactivity.
Running on Modal Auto Endpoints with SGLang today.
Inkling is seriously impressive, try it out in Auto Endpoints! (with incredible performance enabled by our custom DFlash speculator)
A very big day for American open weights 🇺🇸 and we @modal are excited to support @thinkymachines with the launch
Inkling by @thinkymachines is now available on Modal, backed by a custom DFlash speculator for 67% higher throughput and interactivity.
Running on Modal Auto Endpoints with SGLang today.
