Turns out you can make LLM inference fully deterministic across devices, with no loss to quality or speed. This weekend at the @spacexai hackathon I got Qwen3-0.6B to produce identical hashed logits from a 512-token generation across 2 GPUs and 3 CPUs: an A100, an H100, an Apple M5 Max, an AMD EPYC, and an Intel Xeon. The main reason why inference isn't deterministic is because floating-point addition is not associative. (a + b) + c ≠ a + (b + c), since every add rounds. Accumulation order changes with the hardware used and kernel selected, so the same prompt can give you different outputs even at temperature 0. Integers ARE associative. So why doesn't integer quantization already fix this? Because, while weights and activations get quantized, the non-linear ops (softmax, normalization, SiLU) dequantize back to float and requantize afterwards. Each of those steps hands you back to floating point rounding. True integer-only inference does exist, but it's historically been motivated by edge hardware without FPUs, which doesn't make much sense for LLMs. One 2024 paper (I-LLM) did it on LLaMA from that angle and didn't get much attention. Nobody seems to have looked at it from the determinism side. I wrote my own implementation, simplifying the approach from the paper, so that every operation between the input ids and the int32 logits is exact integer arithmetic. To test it I chain-hashed the logits at every step and ran that across the devices and configurations below. Every integer run gave the same hash: 64430dd985f8. Every fp16 run gave a different one, all diverging on the very first token. WikiText2 perplexity came out to 20.72 vs 20.95 for fp16 (slightly better than the float baseline), and CUDA-graphed integer decode hits 106 tok/s at batch 1 on an A100, 3.6x the fp16 eager baseline. Github repo is listed in the comments. Plan to do a writeup over this eventually!

LLMs can be near-perfect lossless compressors via arithmetic coding, yet nobody uses them for compression. Found out why: ran an 0.8B model on 1MB of text. It compressed 2x better than gzip, but ran ~70,000x slower (1 hour vs 0.05s)

New post! A drawback of diffusion LLMs is that bidirectional attention causes keys and values to drift across steps, breaking KV caching. But generation quality is robust to slight KV drift, and much work has gone into maximizing stale KV reuse without causing quality drops (1/3)

I noticed that Apple Notes has a similar UI as the AI chat apps, so I turned it into a Claude/ChatGPT frontend. Use any LLM API to interact or chat with in Apple Notes