DeepSeek just open-sourced DeepInfer, the inference framework behind their large-scale production services. There is A LOT in there: kernels, massive expert parallelism, KV cache tricks… But the scheduling is probably the part interesting that got me really interested 👀 DeepInfer's scheduler increasingly decides *where compute is actually worth spending* rather than treating every request/token the same. For speculative decoding, a confidence head predicts which draft tokens are likely to be accepted, and the scheduler uses that + measured hardware throughput to decide how many tokens to verify for each request. That means the duration of a batch becomes more predictable: no longer will one replica take longer than the others to finish and slow the whole group down. More regular batches -> more throughput. They apply the same philosophy elsewhere: - prefill chunks are dynamically sized from a compute budget + hardware cost model - decode routing balances both request count and KV cache length - CPU scheduling is overlapped with GPU and tightly integrated with speculative decoding - cached prefixes are decoupled from compute placement, so routing can optimize for available compute instead of cache locality This is exactly the kind of inference work I'd love to see become reusable beyond one model or one serving stack: Transformers already has CB, paged KV caches and async batching; @remi_or_ and the team are currently working on branches to see which of DeepInfer's ideas we can generalize so that they can benefit many more architectures. Preliminary rsults in the image below! Thanks DeepSeek for open-sourcing once again ❤️

PyTorch 2.15 is a pretty big release if you run models on a Mac. Over the past months, the @PyTorch team at @huggingface has been working upstream on MPS, replacing MPSGraph paths with Metal kernels. Some of the results are kind of wild: - MoE decoding is now 5-6.5x faster than in PyTorch 2.14 - Qwen3.6-35B-A3B: 9.4 → 61.3 tok/s on an M3 Ultra - LFM2.5-8B-A1B: 23.6 → 123.6 tok/s - softmax over a 128k vocab is ~8x faster - FP8 is now supported on MPS - Wan2.2 VAE decoding goes from ~270 GiB peak driver memory to ~8 GiB This isn’t a specific runtime, or a separate inference engine, it’s just PyTorch getting faster on Apple silicon! Impressive work by @Is36E! 2.16 is already looking even faster 👀 huggingface.co/blog/mps-pyto…