h0ngcha0 (ℵ)

So Anthropic has a paper on SNARKs for proving the correctness of LLM inference. The idea of snarks for LLMs is that you can check a response came from the claimed model without the operator exposing the model’s weights. It's Cool work. TL;DR: it’s about 70-95x slower than their chosen baseline, which is very impressive. I’m still reading, but my biggest question is whether that overhead is really ~100x. if I spent a few million tokens’ worth of engineering effort, would the baseline improve more than their SNARK scheme? Of course, at some point this doesn’t matter. The point is cost for the capability. If you want verifiable LLM results, on a non-local model, and can’t use a TEE, is 100x overhead acceptable? Note: they do this with a mixture-of-experts model and prove after the fact, in batches, so the prover only does work for the experts that actually ran. That doesn’t seem to translate to 100 tokens/second for other models. But like I said, I’m still reading. anthropic.com/research/prova…