EXO Labs

The memory unit economics of Apple Silicon are insane. When you realise LLM inference is massively memory-bound (even at high concurrency!), you find that $ per memory bandwidth ($/GB/s) is the most important metric for inference. M5 Ultra: $5499, 96GB @ 1.2TB/s, $4.5/GB/s DGX Spark: $6950, 128GB @ 273GB/s, $25.5/GB/s DGX Station: ~$150K, 252GB @ 7.1TB/s (HBM3e), $21.1/GB/s M5 Ultra has 5.6x better $/GB/s than DGX Spark and 4.7x better $/GB/s than DGX Station. I am surprised Apple didn't increase their prices more (maybe they will so might be a good time to buy...) I expect 512GB M5 Ultra Mac Studio to start at $15899 (following the usual +$25/GB incremental memory price). That's a crazy good price for that much high speed memory. With RDMA over Thunderbolt 5 you get single digit microsecond latency and you can connect up to 5 Thunderbolt 5 cables between Macs for 400Gb/s network bandwidth (same as ConnectX-7). Thunderbolt 5 cables are ~$25 each. The main problem is the software on Apple Silicon is not great. There are about 30 competing inference engines, each supporting a small set of models, and quite unstable. The ecosystem around MLX is immature and limited (e.g. no Dynamo, LMCache, llm-d, ...). In our experience, more serious customers like Governments and Enterprises are put off by that and want something stable with an ecosystem of tools and software to run inference in production.

My GTM lead asked me to share this. For serious business people pls. tally.so/r/lbx4l6

Apple's AI strategy is Apple Silicon. The iPhone, iPad, MacBook, Mac Mini and Mac Studio all use the same hardware architecture. Apple Silicon is energy efficient, quiet, and the memory unit economics are incredible. Apple has leaned hard into Local AI, now serving every segment of Local AI, from SLMs running on an iPhone/iPad to 2T models running on a cluster of 4 x 512GB M5 Ultra with RDMA using @exolabs (the cluster has 2TB memory @ 4.8TB/s). Most inference will run locally, and Apple wants it to run on their silicon. The main issue I see right now is the software. We need a stable, canonical inference engine rather than 10 unstable, incomplete ones. Who is solving this?

hiring a founding designer for @exolabs in-person in SF $20k referral fee jobs.ashbyhq.com/exo/2d9292b…

Interesting article in @theinformation. Macs are Apple's fastest growing business right now, driven by massive AI demand. Apple's AI strategy IS the Mac. Trillion dollar opportunity imho. theinformation.com/articles/…

Someone posted this on reddit and now it’s #1 on r/LocalLLM! Answering questions about exo and local AI in the thread.

exo is currently top of r/LocalLLM (in response to this tweet). answering questions there in detail on my alt: Longjumping_Crow_597. feel free to ask any questions, exo or local AI related. link: reddit.com/r/LocalLLM/commen…

2 years ago, we achieved the first big milestone with @exolabs. We clustered 2 MacBooks to run Llama 405B. It felt like magic. The consensus was running this model was only possible in a data center. We ran it on consumer hardware, on 2 M3 Max MacBook Pros. Most people thought it was a gimmick. It only ran at 2 tok/sec! But, we believed that improvements to the software, hardware, and models would all compound. So that maybe in a few years, we thought, this would improve 10x in software, 10x in hardware, 10x models = 1000x. That was the vision. We imagined a world where you would have frontier intelligence running quietly on your desk. Today is the day that vision became reality. The M5 Ultra is a 10x step-change improvement vs the M3 Max we originally clustered. That, compounded with software improvements like RDMA over Thunderbolt, MTP and better kernels, and high intelligence density models like Qwen 3.8 27B, means we now have 1,000x better Local AI than when we started. I am so grateful to the small group of people at Apple (including @doogie69 @awnihannun @angeloskath @DiganiJagrit @doogie69) who believed in this vision and had the foresight as well as the courage to take a swing at this early on. I'm confident they're just getting warmed up (looking forward to 4-bit / 8-bit compute units in M7 Ultra🤞). With the M5 Ultra Mac Studio, we are going to have unmetered tokens running at API speeds at effectively zero marginal cost, running on your desk, so quietly and consuming so little power you won't even notice it. Local AI is good now.