idk what I’m doing half the time. but sometimes I design systems for constrained compute | local & edge ai current curiosity: designing a model
Earth
Joined September 2023
- Tweets2.7K
- Following246
- Followers3.7K
- Likes4.8K
Pinned Tweet
I got a 1T (trillion) parameter model running on my MacBook Pro.
Kimi-K2. 1.029T params.
~1 TB raw weights.
524 GB converted.
~1.7 tok/s.
Yesterday it was 671B. Today it's 1T.
Same laptop. Same M4 Max. No cloud.
When I say we: I mean Claude and me.
i fed a 1M-token prompt to Qwen3.8-27B on one DGX Spark. 70 minutes to read it, then it answered at 4.3 tok/s
512 tokens: first token in 0.2 s, then 11.4 tok/s
256k: 5.6 min, then 7.9 tok/s
1M: 70 min, then 4.3 tok/s at 8.8 J per token, GPU-only
the reading is what hurts: 248 tok/s at 1M vs 2,274 at 512. the answer itself only slows 2.6x
past the model's 262k native window this is speed and energy only. i did not check what it answered
Qwen3.8-27B · NVFP4 · vLLM 0.28 · fp8 KV · no speculation · one DGX Spark · 2-5 runs per point
the fastest setting i tried for a code edit on my DGX Spark was 7.3x faster and broke the file. one stray line, every run. the setting that was 3.4x faster got it exactly right
same task as YASAKANI's 5090 post: rename one name in textwrap.py, return the whole ~4,700-token file. Qwen3.8-27B, greedy, thinking off, same speed math
no speculation, both 4-bit engines hit the same wall:
llama.cpp Q4_K_XL: 11.3 tok/s
vLLM NVFP4: 11.2
speculation on the edit, vLLM NVFP4:
MTP2: 26.3, exact
MTP4: 38.0, exact, byte-identical
n-gram: 82.0, wrong. Python won't parse the file
the n-gram file has one line the original never had: "_WHITESPACE_CHARS." on line 15, right above the real one. same spot in all 3 runs
n-gram speculation drafts 8 tokens at a time by copying earlier text. on a file that is mostly copy, 89.7% of drafts land, so it flies
greedy speculation is supposed to be lossless. here it wasn't. MTP on the same vLLM build was exact. only checking the output caught it
the more the output copies its input, the more drafts land. MTP4 accepts 57% on a summary, 85% on fresh code, 99.8% on this edit: 2.25x -> 3.0x -> 3.39x. it tops out because it only drafts 4 tokens
YASAKANI's post checked that all three engines returned the same 4,711 tokens. that check is the whole game. speed means nothing until the file parses
next on this box: llama.cpp with its own speculation on the same edit, then more engines
Qwen3.8-27B · vLLM 0.28 NVFP4 · llama.cpp 4801e3c56 · one DGX Spark · 3 runs per setting · llama.cpp plain decode from an earlier run on another prompt · GPU-only energy 2.98 -> 0.89 J/token with MTP4
apple limiting their own hardware
Linux now beats macOS on Apple silicon GPU memory bandwidth.
M1 Max, Omarchy:
macOS: 361 GB/s
Linux before: 241 GB/s
Linux now: 368 GB/s
The kernel never told the power controller the GPU was busy. One bit fixed it.
github.com/omacom/linux-auro…
"An AI (gpt-6 astra) edited 0.016% of a 0.8B model's weights and its math score jumped 6x."
reproduced it with claude and measured the energy: 21.7 kJ → 2.6 kJ per correct answer.
asked a 0.8B model: 441 + 2(21)(19) + 361 = x
it got 1600, talked itself out of it, then wrote "Wait, 802 + 798 = 1700." 1,428 times, until it hit the 32k token cap
edit 130k of its weights, no training, and it answers 1600 in 4k tokens
Qwen3.5-0.8B · MATH-500 · my DGX Spark
stock: 7.2%
edited: 44.0%
energy per correct answer: 8.4x less
21.7 kJ -> 2.6 kJ
the edit is from a paper out yesterday, Weight-Space Programming.
energy per correct answer is the number that moved most
per kWh at the GPU, the stock model gets 166 answers right. the edited one gets 1,390
energy per token barely moved: 0.051 -> 0.048 J. the edit does not make tokens cheaper. it stops the model wasting them. 429k tokens per correct answer -> 54k
that loop is not one bad problem. 91% of the stock model's answers looped until the cap cut them off. edited: 28%
this is why i count joules per correct answer, not per token. per token, these two models look the same
how the edit works: a frontier model in the loop looked for the direction two MLP layers (15 and 19) write when the model starts repeating itself, then subtracted it six times over from what 64 neurons in each layer write. it needed a 50 forward-pass budget. with 10 or 25 passes its edits only reached 12-14%
they released the diff. i applied it, matched both edited tensors' hashes to their manifest, and ran stock and edited back to back: same 125 problems, same seed, same sampling
theirs: 7.9% -> 48.1%. mine: 7.2% -> 44.0%. both of theirs sit inside my 95% intervals. 47 problems flipped to solved, 1 went the other way
caveats: one sample per problem, on every 4th MATH-500 problem. energy is the Spark's GPU module, not the wall
Qwen3.5-0.8B BF16 · vLLM 0.28 · T0.6 top_p 0.95 top_k 20 presence 1.5 · 32,768 token cap · graded with math-verify 0.9.0 · DGX Spark GB10
256k-token prompt, then a 1,000-token answer:
DGX Spark: 12.7 min, 16.6 Wh
MacBook Pro M4 Max: 61.5 min, 39.1 Wh
4k-token prompt, same answer:
DGX Spark: 94 s, 1.3 Wh
MacBook Pro M4 Max: 72 s, 0.9 Wh
same Qwen3.8-27B file on both, same llama.cpp build, one stream, raw chip watts
so i split every request into its two stages and put time and energy on one chart for each
reading the prompt: the Spark lands lower and further left at every depth. at 256k it gets to the first token 5.9x sooner on 2.5x less energy
writing the answer: the Mac lands lower and further left up to 128k, 1.2-1.9x sooner on 1.5-2.0x less energy. at 256k the Spark writes 1.1x faster and the Mac spends 7% less per token
a short prompt goes to the Mac because the answer is most of the work. a long prompt goes to the Spark because the reading is
and the shared GGUF is not the Spark's best format
256k prompt on the Spark, time to first token: NVFP4 in vLLM 5.6 min, FP8 6.2, BF16 7.1 (an August run, no power meter on it), the GGUF 9.9
decoding there, NVFP4 with 3-token speculation does 19.8 tok/s at 1.9 J per token, measured this week. the GGUF does 5.8 tok/s at 6.8 J
the Mac's fastest answer at 256k is a different model: the 35B-A3B MoE, 3B active, 17.3 tok/s at 1.8 J. a 3B-active token and a 27B token are not the same work, so it sits outside the head to head
the rule i ended up with, per request:
long prompt, short answer -> Spark
short prompt, long answer -> Mac
256k prompt -> Spark, at any answer length
where it flips, the answer length at which the two tie, joules | time:
512 context: 48 | 41 tokens
4k: 224 | 426
16k: 987 | 1,637
32k: 1,849 | 3,586
64k: 3,906 | 9,531
128k: 14.9k | 37.0k
256k: 182k | never. no 256k context has room for that answer
i drew a first version of this line yesterday. this one has the Spark on the same llama.cpp build as the Mac, so the numbers shifted
the caveats. these are raw chip watts with idle included: the Spark GPU module idles near 4.4 W, the Mac SoC near 0.1 W, and the SoC number leaves out the Mac's memory and fans. subtract idle and the Spark is 5% cheaper per decoded token at 256k. no wall plug on the Spark yet
Qwen3.8-27B UD-Q4_K_XL · llama-bench 4801e3c56 on both · f16 KV · DGX Spark GB10 vs M4 Max 128GB
223 decode runs: 0.6B to 35B on both boxes, across llama.cpp, MLX, vLLM and SGLang
raw chip power: 14 to 66 W
decode speed: 4.3 to 448 tok/s
energy per token: 0.066 to 9.4 J
power moves 5x. speed moves 100x. energy per token moves 142x, and speed alone explains 97% of that spread
which is close to arithmetic: joules per token is watts divided by tokens per second. the chip draws about the same whatever you run on it, so the token rate does the rest. joules per token is mostly a speedometer
reading the prompt is the same story: 222 prefill runs, 31 to 96 W at the chip, and speed explains 94% of the energy per prompt token
the cheapest token on my bench: Llama 3.2 1B on MLX at 512 context, 0.066 J
the most expensive: Qwen3.8-27B in BF16 on the Spark at 32k, 9.4 J
142x apart, and nobody swaps a 27B for a 1B to save power. the tokens are not the same work
that is the catch with joules per token. it prices the token, not the answer. it is also why a 35B MoE with 3B active looks 3.3-4.5x cheaper per token than the dense 27B on the same Mac
the number i actually want is joules per correct answer
the first one i have: Qwen3.6-35B-A3B on all 164 HumanEval problems, greedy, on the Mac. 154 solved, 41,294 tokens, 53.9 kJ. that is 350 J per solved problem, measured at the whole machine, not at the chip like the charts above
Logics-SWE, a fine-tune of the same model in a 9% smaller file, solved 152 (a 2-problem gap is within noise) and came out at 336 J per solve, 4% less. it also wrote 5% faster
that is the point of the unit: speed and accuracy end up in one number
next: the same 164 problems from 0.6B up to 27B, so the cheap tokens have to show what they buy
scatters: raw chip watts (Spark GPU module, Mac SoC) · Mac llama.cpp 4801e3c56 and MLX · Spark llama.cpp 4801e3c56 and dde879bc4, vLLM plus a DFlash2 fork, SGLang · HumanEval: whole machine · M4 Max 128GB · DGX Spark GB10
DGX Spark vs Mac
performance as a function of time & energy
64k of context, 2k-token answer:
DGX Spark: 5 min, 17 kJ
MacBook Pro M4 Max: 10 min, 23 kJ
4k prompt, 8k-token answer:
DGX Spark: 12 min, 33 kJ
MacBook Pro M4 Max: 7 min, 19 kJ
same Qwen3.8-27B file, llama.cpp on both, one stream. each box wins one request outright, on the clock and on joules
so i found the line. answer length where the two tie, joules | time:
512 context: 55 | 38 tokens
4k: 248 | 457
16k: 1,071 | 1,809
32k: 1,971 | 4,071
64k: 4,640 | 11.5k
128k: 17.5k | 58.8k
256k: never | never. the Spark wins every request
the Spark reads 2.2-2.8x cheaper and 2.7-6.5x faster at every depth
the Mac writes 1.6-1.9x cheaper up to 64k and ~1.7x faster up to 32k. by 256k the Spark writes faster too
at 64k the Spark starts answering in 88 s. the Mac takes 7.5 min
64k context with an 8k answer is the odd one: the Spark finishes first, the Mac spends fewer joules
chip watts only, Spark GPU module vs Mac SoC. no wall plug yet. and the Spark ran llama-server while the Mac ran llama-bench, matched rerun queued
Qwen3.8-27B UD-Q4_K_XL · llama.cpp · f16 KV · flash attn · DGX Spark GB10 vs M4 Max 128GB
fine tuning on the DGX Spark -
opssat anomaly: 26 -> 96 (27B: 97)
clinc intents: 59 -> 99 (98)
financial phrasebank: 51 -> 91 (93)
typed decisions: 59 -> 82 (82)
jigsaw toxicity: 80 -> 97 (97)
chaosnli: 60 -> 77 (83)
boolq: 82 -> 90 (93)
arc: 92 -> 96 (97)
decision accuracy, one Qwen3.5-4B. left number is zero-shot with the chat template, right is after a LoRA fine-tune (r16, 3 epochs, 6k labelled decisions). in brackets: the same fine-tune on Qwen3.8-27B.
fine-tuning moved the 4B by 3 to 70 points.
swapping it for a model six times bigger then moved it by -0.5 to +5.8.
that swap costs 4.4x the training time and 59 GB instead of 12.
chaosnli and boolq are where the big model still earns its keep: ambiguous labels and true/false judgments. everywhere else the 4B is done.
one seed per dataset except typed decisions and opssat (3 each). seed spread at 4B is 0.4 points.
the labels did the work. the parameters watched.
spark-train · DGX Spark GB10 · Qwen3.5-4B / Qwen3.8-27B · LoRA r16
follow up to the energy per token post. same 27B GGUF, same llama.cpp, DGX Spark vs my M4 Max, now run out to the model's full 256k context
to be clear - this is not playing to each machine's strengths and ofc this is the M4 max. chip power only on both sides: GPU module on the Spark, SoC on the Mac. not a perfect comparison..
to 64k the Mac SoC is 1.6 to 1.9x cheaper per decoded token. 2.3 to 2.5 J vs 3.9 to 4.7 J on the Spark GPU
then the curves meet. at 256k it is 6.38 J on the Mac, 6.59 J on the Spark. same energy per token
what changed is the balance, not the winner. the Spark reads the 256k prompt in 9 minutes, the Mac takes 58. the Spark decodes 6.4 tok/s there, the Mac 5.3. the Mac gets to parity on energy by drawing 34 W to the Spark's 42, not by being faster
it doesn't seem quite clear if 2 sparks are better or worse than an mac m5 ultra with 256gb..
sure if you're just doing some quick questions the mac wins
when you pass the point of 128k tok its not as clear. the sparks prefill makes up for its slower decode.
the mac beings to slow down and the energy seems to almost match
im currently wondering if i should get another spark or get the 256gb m5 ultra.
i dont think the its as simple to say the mac is better because it has 1.2tb/s of bandwidth on mem.
in practical use cases (subjective) you'll see a substantial amount of context.
what is great though is the macs RDMA which for the price of 'only' 80k you could have 2tb of high bandwidth mem. which is objectively a good price in this market for those that can afford it.