Every LLM quant, diffed against the original. Measured KLD, accuracy with 95% CIs, tool calls and speed for 8–16GB GPUs and Macs. Open data.
- Tweets15
- Following22
- Followers4
- Likes0
ALT Scatter chart of 6 LFM2.5 24B A2B GGUF quants on an Apple M2 Pro 32GB: generation speed in tok/s across, accuracy on 500 questions (GSM8K+MMLU+ARC-C) up, with 95% CI bars. LiquidAI QAD-Q4_0 (13.6 GB) sits far right at 88.9 tok/s and 72.2% (68.1–75.9). Others: Q4_K_M 14.5 GB 71.4% 78.5 tok/s; UD-Q2_K_XL 8.0 GB 70.2% 73.8 tok/s; UD-Q4_K_XL 14.9 GB 70.8% 69.1 tok/s; UD-Q3_K_XL 10.9 GB 70.0% 68.8 tok/s; UD-Q5_K_XL 17.9 GB 72.8% 62.8 tok/s. All bars overlap the best file's CI band (68.7–76.5%), so all are tied on accuracy.
ALT Two bar charts for Sage-A1.5 GGUF quants. Left: accuracy on 500 GSM8K+MMLU+ARC-C questions with 95% CI: i1-Q2_K 12.3 GB 86.8%, i1-IQ3_M 14.7 GB 88.0%, IQ4_XS-exps 18.0 GB 90.2%, Q4_K_M 20.2 GB 89.8%. Right: generation speed on Apple M2 Pro 32GB (llama-bench tg128): 49.2, 50.2, 50.1, 50.9 tok/s. No KLD: BF16 reference too large for our 32GB test Mac.
ALT QuantDiff chart: Flint, 3 unsloth GGUF quants. Accuracy UD-IQ1_M 66.8%, UD-Q2_K_XL 75.8%, UD-Q3_K_XL 78.2% (500 questions, 95% CI). Speed on Apple M2 Pro: 55.6, 54.5, 49.9 tok/s.
ALT Two charts for LFM2.5 24B A2B GGUF quants: accuracy 70.0 to 72.8 percent with 95% CIs, and M2 Pro generation speed 62.8 to 88.9 tok/s
ALT QuantDiff chart for Spark-X2.6 4B, 10 GGUF quants. Left: accuracy on 500 items (GSM8K+MMLU+ARC-C) with 95% intervals: IQ2_M 49.0%, IQ3_XXS 67.2%, Q3_K_M 72.4%, IQ4_XS 75.8%, mradermacher i1-Q4_K_M 76.8%, bartowski Q4_K_M 78.8%, Q5_K_M 78.2%, XHToken Q4_K_M 78.6%, Q6_K 77.8%, Q8_0 78.8%. Right: mean KL divergence vs BF16 by file size, from 1.00 (IQ2_M) down to 0.0027 (Q8_0).
ALT QuantDiff chart for Nemotron 3.5 Bolt E4B, 8 unsloth GGUF quants. Left: accuracy on 500 items (GSM8K+MMLU+ARC-C) with 95% intervals: UD-IQ2_M 78.0%, UD-IQ3_XXS 79.4%, UD-Q2_K_XL 79.2%, UD-Q3_K_XL 80.0%, UD-Q4_K_XL 80.4%, UD-Q5_K_XL 81.4%, UD-Q6_K_XL 81.0%, UD-Q8_K_XL 81.4%. Right: mean KL divergence vs BF16 by file size, measured on chat-format text, from 0.20 (UD-IQ2_M) down to 0.0009 (UD-Q8_K_XL).
ALT QuantDiff quant picker for an NVIDIA RTX 5070 12GB at 8k context, general chat. Top pick: Qwen3.8 27B UD-Q2_K_XL by unsloth, 9.15 GB, general chat score 87.7% (84-91), 48 tok/s measured on RTX 5070. Smaller and faster, statistically tied: Qwen3.5 9B Q8_0 by unsloth, 85.7%, 66 tok/s.
ALT Chart: Qwen3.8 27B, 4 unsloth GGUF quants. Left, accuracy on 500 items (GSM8K + MMLU + ARC-C) with 95% CI: UD-Q2_K_XL 9.2 GB 90.4%, UD-Q3_K_XL 12.2 GB 91.4%, UD-Q4_K_M 15.3 GB 91.8%, UD-Q5_K_M 18.4 GB 91.8%. Right, llama-bench generation speed on an Apple M2 Pro 32GB: 9.7, 9.4, 8.8 and 7.6 tok/s; UD-Q2_K_XL on an RTX 5070 12GB: 48.1 tok/s. No KLD: the BF16 reference is too large for our 32GB test Mac.
ALT Qwen3.5 9B GGUF quant accuracy chart with 95% confidence intervals.
ALT Gemma 4 12B GGUF quants on RTX 5070: accuracy UD-Q2_K_XL 67.2%, Q3_K_M 85.0%, Q4_K_M 87.4%, Q5_K_M 87.0%, Q6_K 88.2%, Q8_0 89.2%; speed 98.4, 77.7, 71.9, 62.9, 56.9, 13.7 tok/s.
ALT Ornith 1.5 9B GGUF quants: accuracy with 95% CIs (Q4_K_M 87.2%, Q5_K_M 87.4%, Q6_K 87.2%, Q8_0 86.0%) and RTX 5070 generation speed (99.4, 89.7, 83.0, 65.2 tok/s).