
Research @togethercompute, EE/CS @Stanford
Joined February 2023
- Tweets226
- Following883
- Followers1.5K
- Likes2.8K
December Unwrapping TPUs Finale 🎄
Surreal to see all the incredible support over the last 24 hours!!
To read more, TPU Demo write-up here:
chewingonchips.substack.com/…
Code linked on GitHub as usual!!
Demo: github.com/Alanma23/tinytiny…
Archive: github.com/Alanma23/tinytiny…
Stay tuned for our Diffusion router hack + 20-ISA update :)
See you soon,
@abiralxshakya and @Alan_Ma_
TPUs Unwrapped!!🎄
- real-time FPGA inference
- custom ISA + open-source tooling
- PyTorch + uint8 quantization
- gesture recognition inference demo
been a crazy month shipping, working night and day, but we finally got something working!! special shoutout to @abiralxshakya for being the greatest hack partner and all the crazy support we've had along the way...
HAPPY NEW YEAR!
as always, more to ship and more details to share ✨
2026 🔜
Alan Ma retweeted
Over the past week, I’ve been overwhelmed by the love and support of friends, industry colleagues, and people I've never met. Thank you to everyone who has shown kindness to my family and me. It means the world to me.
I came to the United States 53 years ago, and becoming an American citizen is one of the greatest gifts and proudest moments of my life. NVIDIA has been my life’s work. Building it into the company it is today alongside my colleagues has been a privilege and an honor. Nowhere but in America could a company like NVIDIA be possible. I love our country, and I love NVIDIA.
I did not live up to the standards expected of everyone at NVIDIA, and I have made serious mistakes in my personal life. I created problems for the Board that I deeply regret. I own my mistakes completely, and I am committed to cooperating with the Board’s independent investigation.
Since my departure, I’ve seen an outpouring of calls from NVIDIA’s employees, investors, business partners, and customers for me to return as CEO. Many continue to reach out to me directly as the company navigates complex technology, market, competitive, and geopolitical dynamics. NVIDIA is a once-in-a-generation company. If the Board of Directors decides that my return is in the best interest of our shareholders, our employees, the company, and America, I would be honored to answer these calls and return as CEO.
Thank you again to everyone who has shared their good wishes. I am profoundly humbled by your support. To my family: I love you, and I will spend the rest of my life being worthy of yours.
Jensen
Alan Ma retweeted
Had a great time working on this! The future of systems is in agents that can continuously optimize and target the workloads that they serve.
Introducing the Together Compiler: continual compilation for AI inference! I’m super proud to finally share this work from our research team! ✨
Today’s inference is “Polymorphic” or shape-shifting, it doesn’t sit still. On OpenRouter, 80% of traffic goes to models released in the last three months and the top ten most popular open models each use a different architecture. Our customers constantly change their usage behaviors. Together Compiler leverages agents to continually optimize our deployments immediately as these changes occur.
Together Compiler research highlights include:
1. 252x better throughput: TKCC lets agents try 252x more optimization experiments per hour compared to prior agent inference optimization systems.
2. State-of-the-art TPS across InferenceX workloads we tried: TKCC outperforms the best InferenceX engine across all tested settings in just 24 hours & 2 nodes per compile. Including 1111 TPS on MiniMax M3.1 B200 and 1205 TPS on DeepSeek v4.1 Flash GB300 at batch size 1. NVIDIA, AMD, Trainium2.
3. Continuous recompilation: TKCC currently serves many trillions of production tokens per day and adapts to traffic shifts within just a few hours of recompilation.
4. Bitter lessons: TKCC’s design builds on 2+ years of building frameworks to accelerate AI Inference, starting from ThunderKittens and Megakernels. Compilers have always used abstractions and IRs, but we show that agentic compilers now reach higher performance ceilings without abstractions.
From an incredible team @reyna_abhyankar @dylan__lim @realDanFu @togethercompute!!
Alan Ma retweeted
We turned Qwen3.8-27B into a multimodal decision model.
It beat Pokémon FireRed’s elite four and champion with sub-100 ms decisions from live game state.
With SGLang’s native /v1/decisions, you can now turn LLMs and VLMs into classification and scoring models.
We also added /v1/systemone so Jev-like open models can work with the TypeSafe SDK.
Alan Ma retweeted
a bit of deep learning history/lore: batch normalization appeared in 2015, took the deep learning world by storm, and subsequently influenced the design of early TPUs (v2)!
i have a feeling we'll see an explosion of architecture research over the next few years (if not months?).
(goat @ChrSzegedy)
Alan Ma retweeted
Spent some time creating a lightweight, performant async RL stack in pure JAX with @AdityaMakkar0! We share a work log covering inference, RDMA weight transfer, memory optimizations, sharding and a few more JAX engineering insights we've seen along the way! It’s a bit of a long read, but I hope it sheds some light on the systems side of scaling RL!
Alan Ma retweeted
made some updates to temp chats:
toggleable personalization,
no more sidebar clutter,
and you can now save when it becomes useful!
go try it out :)
We heard your feedback that sometimes a Temporary chat becomes something worth saving.
Now you can choose to personalize a Temporary chat with your existing memories, custom instructions, and plugins—and save it to your history if you want to keep it.
Temporary Chats don't create new memories, and stay out of your sidebar unless you choose to save them.
Alan Ma retweeted
update: made TPUs go brrr this summer @sailresearchco
also built HTDYM, a tool for figuring out analytically which chips (and shardings) can serve a model the cheapest
hint: it's not the same for every model / workload
check it out! sailresearch.com/blog/htdym
Alan Ma retweeted
"We are frugal with machines and generous with humans." @neilmovva went deep on how we're building the world's most efficient token factory from data centers all the way up into DSV4F tokens with @patrick_oshag . If chasing the speed of light on frontier hardware sounds like your idea of fun, reach out!
Neil Movva (@neilmovva) started his career at Nvidia, working on GPUs and kernels, and has an unusually deep understanding of inference, from software to chips to power.
We spend a lot of time on each of those layers, how they connect, and where the important tradeoffs are.
What makes this conversation special is how detailed it is (like a 401-level class), yet Neil makes it remarkably clear and easy to follow.
Today he runs Sail Research, a company building infrastructure for agents to make tokens as cheap as possible.
We discuss:
- Latency versus throughput
- Why there are no bad chips, only bad pricing
- The end of kernel engineering
- Buying chips and power no one else wants
- New chip architectures
- Nvidia lore + his contrarian view of the company
- Open source and the frontier labs
I learned a ton. Enjoy!
TIMESTAMPS
0:00 Intro
0:38 Building a “Token Factory”
4:21 The Future of Background Agents
13:09 Nvidia and the GPU Stack
23:27 Chips, Memory, and Transformers
36:14 The Future of AI Training Data
44:32 Chip Scarcity and Compute Arbitrage
52:44 Reinventing the AI Data Center
59:01 Power and the “Scavenger Strategy”
1:10:10 Open vs. Closed AI
Simile trains a simulation model. Also, we train another model that predicts the error of the simulation model (the “confidence model”). Tucked away in a secret dungeon below Mission Bay, we also have a “confidence confidence model” that predicts the error of the confidence model, but this one is saddled with a certain sinister statistical spirit that must be contained at all costs! To contain San Francisco’s crisis of confidence, Members of Technical Staff across the valley must come together and read Simile’s confidence blog post today!
Anyone can simulate the future. But the simulation only matters if it’s trustworthy.
At Simile, we train two types of models: simulation models and confidence models. Our first research blog post explores the origin of our proprietary confidence model, which predicts the accuracy of our population simulations and, in turn, makes them actionable.
simile.com/blog/confidence?v…
Alan Ma retweeted
Jalapeno beats VR200 in A0 silicon with a vibe-ported, non-specdec implementation of DeepSeek 🌶️ 🌶️
And with much faster program execution than VR200, and with a B0 update landing imminently
Congrats to the Jalapeno team!!!
Speed on the track starts with innovation behind the scenes. 🏎️
Great to connect with the @ArrowMcLaren team as we celebrate a partnership built on innovation, precision, and technology.
As McLaren Racing's Official Compute Partner, Intel powers the advanced computing behind race-day performance.
Alan Ma retweeted
It feels like new AI accelerators are released daily, but the real bottleneck has always been the software needed to make them truly usable.
Hawkeye designs a minimal set of operations that provide coding agents with the concise and highly valuable information needed to bootstrap hardware aware optimizations for achieving performant kernels.
As a result, we are able to beat expert baselines over a variety of workloads!
We see in recent trends that models are increasingly able to even one-shot SOTA NVIDIA kernels, this work is exciting to see how we can harness these model's capabilities for new and unfamiliar accelerators.
Cool work with amazing collaborators!
We’ve seen an explosion of new ML chips with unique architectural features, but software support remains the critical bottleneck
Achieving peak performance increasingly relies on hardware-specific optimizations in the kernels, but we observe that coding agents are particularly weak at this
Introducing Hawkeye, a framework that brings hardware-awareness to coding agents by grounding them in a minimal and comprehensive taxonomy of optimization strategies
For new GPU or ML accelerator architectures, you only need to write 10 unit tests and solution kernels (one per optimization strategy), and we show that coding agents can effectively scale test-time compute with this minimal supervision to write hardware-aware kernels
Hawkeye can port kernels across architectures (Ampere, Hopper, Blackwell), vendors (NVIDIA, AMD), and precisions (FP8, NVFP4, MXFP4) while consistently leveraging hardware features and approaching expert kernel performance
Work co-led with @keramakr and done in collaboration with Alexander Ingare @simonguozirui @18jeffreyma @ZishenW @simran_s_arora @Azaliamirh @profvjreddi
Alan Ma retweeted
Excited to share some of our research at Bridgewater AIA Labs in collaboration with @thinkymachines!
Grateful to work with @SarahhhSuuu, @XiaoEmily41333, @rohanalur, and @ddkang
Sorting which financial docs are worth an analyst's time is surprisingly hard for frontier LLMs. With an expert-labeled dataset and on-policy distillation, Bridgewater fine-tuned a model to do it reliably and cheaply.
thinkingmachines.ai/news/lea…
Alan Ma retweeted
I had a great time this week at AI Systems DevLabs in the Bay Area. @Google definitely hosts the best conferences hands down. I went to this conference a couple times in 2024, and this one was definitely the best I have ever seen. Keynote speakers were also very interesting. I got to hang out with some fun people - @MainzOnX @A_K_Nain
Alan Ma retweeted
Samir Menon @blintzbase and I are thrilled to announce Sail @sailresearchco ! We build infrastructure for long-horizon agents: inference served at unbeatable prices-per-token for open models, plus sandboxes designed to run for days, weeks, or longer.
We've raised $80M, w/ our seed led by @Sequoia
and series A led by @KleinerPerkins. We're using this capital to build the most efficient infrastructure for long-horizon agents.
What makes agents so different? Unlike a human waiting at a keyboard (top priority: speed), agents need scale, reliability, and sustainable cost. Sail finds this efficiency everywhere in the stack: we carefully choose our chips, write custom inference engines, and run a global controller that fully utilizes every computer in our fleet. Tight integration from silicon to API lets Sail open up the cost / latency frontier to our customers - the most patient agents can now access 10x more intelligence per dollar.
We're excited to be working with great companies like
@parallelweb, @detaildotdev,@Jackandjillai, and @quadrillion_ai to deploy long-horizon agents with trillions of tokens.
Our team is thoughtful in our engineering craft and relentlessly ambitious in our pursuit of peak performance. We previously trained at companies like NVIDIA, OpenAI, Google, and so many trading firms. Now we're ready to do the work that will define our careers, in the most compute intensive market of all time.
Welcome to the era of abundant intelligence. We can't wait to build with you!
We’ve designed and built our first AI chip: Jalapeño.
Designed from the ground up by OpenAI and brought to production with @Broadcom, Jalapeño is purpose-built for the LLM workloads powering ChatGPT, Codex, the API, and future agentic products.
Chips are foundational to the AI economy. Building our own expands our full-stack platform from products to models to infrastructure, and will help us scale intelligence, serve more people, and expand access to AI.











































