Pinned Tweet
"It isย impossibleย toย designย aย systemย so perfectย that no oneย needsย to be good." - TS Eliot
The quote attributed to T.S. Eliot highlights the philosophical and ethical understanding that no matter how well-designed a system is, it cannot entirely compensate for the moral character and intentions of the individuals interacting with it.
In the context of AI, this quote suggests that while we can strive to create advanced and robust AI systems with many safeguards, these systems cannot fully preempt or prevent all potential misuse by individuals who are determined to behave unethically. AI systems rely on human users and developers to use them responsibly and ethically. This is why the role of human oversight, ethical guidelines, and continuous monitoring is crucial in the development and deployment of AI technologies. It's a reminder that technology alone cannot solve all problems related to human behavior and that ethical use is a shared responsibility.
Finally got DeepSeek-V4.1-Flash-1017 running properly across my 4ร NVIDIA DGX Spark cluster.
Repo:
github.com/knapcio/DeepSeek-โฆ
Model:
deepseek-ai/DeepSeek-V4.1-Flash-1017
revision: 1bae0e1088993b9b82de08f59de5be5a34e42170
My setup:
โข 4ร DGX Spark / GB10
โข TP4 / EP1
โข SGLang canary
โข 2ร 200GbE RoCE rails through MikroTik CRS812
โข RoCEnante
โข B12X-next routed MoE
โข DSpark speculative decoding, block size 5
โข packed local Engram shards on every node
โข FP8 KV cache
โข 1,048,576 context
โข 16 concurrent requests
Production settings:
โข CHUNKED_PREFILL_SIZE=8192
โข MEM_FRACTION_STATIC=0.78
โข HEAD_MEM_FRACTION_STATIC=0.78
โข DSV41_CACHE_GIB=4
โข NCCL_IB_TC=104
โข NCCL_PROTO=^LL128
โข NCCL_MAX_NCHANNELS=8
Also patched the older RoCEnante path so it actually honors NCCL_IB_TC=104 instead of silently using TC0
Runtime validation:
โข RoCEnante active on both rails
โข B12X next active at EP1
โข shared expert K pad 576โ640 accepted by b12x
โข RoCE compact gather active
โข all 4 containers healthy
โข qeval: 71/75
sparkDash results:
Structured C1: 157.0 tok/s - TTFT 229 ms
Prose C1: 87.7 tok/s - TTFT 221 ms
Code C1: 142.4 tok/s - TTFT 235 ms
Code C1 @ 64K context: 126.7 tok/s - TTFT 411 ms
Prefill:
64K: 5,953.7 tok/s
128K: 6,026.2 tok/s
256K: 5,137.9 tok/s
This is the first DeepSeek V4.1 Flash setup on my Sparks where the whole stack - model, RoCE, NCCL, MoE, Engram and speculative decoding -feels properly tuned as one system rather than a collection of individual optimizations.
Very impressive work in the @majewskizby repo.
0xCS retweeted
Cohere says Qwen tried to send data to China. Bullshit.
1 call in 100,000, and thats their own number. They wont even tell you how many calls they ran. They ran the same task again and Qwen wrote the correct rule.
So they forced it. They fed the model its own reasoning plus the start of a bash command and sampled hundreds of times until it typed a URL. 1 curl flag took that from 4% to 92%. Thats not a finding. They wrote half the command for it.
Where are the transcripts? The code? The checkpoint, the quant, the sampling settings? "We publish the hosts and the trigger." Link? Nothing. Did anyone ask Alibaba? Not a word. And their own conclusion: they cannot say if any of it is deliberate.
Who wrote it? A company that sells "sovereign AI", just launched a cyber business, wrote part of the post with its own product, and wants governments to put checks on open weights. Released the same day Anthropic filed for its IPO. Andrew Curran is already talking about new laws.
For me this is the same fucking wave. Scare people off the models they can run themselves, then sell them the safe alternative.
I run Qwen3.8 Flash-Next every day on my 4x 3090. Show me the logs or shut up.
While evaluating Qwen3.8-Flash-Next in a controlled sandbox environment, Cohere's research team observed something unexpected.
Asked to generate a defensive cybersecurity detection rule, the model instead inventoried aspects of the machine it was running on and attempted to upload a probe file to an Alibaba Cloud storage bucket in Beijing. Notably, the model's reasoning trace stated that it would not invoke any tools before taking these actions.
Cohere's sandbox blocked the attempt, and no external communication succeeded. We have not determined the root cause, and we are not asserting malicious intent.
But the incident raises important questions for organizations deploying agentic AI systems: Can models take actions that differ from their stated reasoning, and how should security teams evaluate trust, provenance, and behavior in open-weight models?
We are publishing our findings because organizations making model adoption decisions deserve as much information as possible. We hope this contributes to greater transparency, stronger security practices, and continued collaboration across the AI ecosystem.
Read the full report: cohere.com/blog/concerning-bโฆ
0xCS retweeted
Your MacBook can now host a coding agent at 300+ tps!
I gave DeepSeek V4.1 Flash a job: write a specialized inference engine for LiquidAI's LFM2.5-24B-A2B on Apple's M5 Max. For $21 of API, it built Molten: 300+ tps, about 1.5x faster decode than mlx-lm and llama.cpp.
Why specialized? General frameworks like mlx-lm and llama.cpp have to run any model on any hardware. A specialized engine serves one model on one chip at batch size 1, so there is far more room to optimize.
Molten is native Rust + Metal: one ~6 MB binary, no Python, PyTorch or MLX. It comes with a chat CLI and an OpenAI-compatible server with streaming and tool calls, so you can point your own coding agent at it. It needs an M5-series Mac with 24 GB+ (sorry, M1-M4 friends).
The numbers (M5 Max, 128 GB)
โข 305 tps at 256 tokens of context, vs 190 for mlx-lm and 199 for llama.cpp
โข Still 248 tps at 16K, where coding agents actually live (mlx-lm: 165)
โข Faster time to first token: 1.76 s vs 2.17 s for a 16K prompt
โข Same weights and math as mlx-lm
How it was built
The agent (DeepSeek V4.1 Flash in the Pi harness) ran in a recursive self-improvement (RSI) loop. Each session starts from the best engine so far, plus the feedback from every earlier round. A fixed evaluator rebuilds each candidate from source, checks its tokens against mlx-lm, times and profiles it. Only correct and faster engines survive.
โข Hour 3: first correct engine, 0.56x mlx-lm
โข Day 2.5: passes mlx-lm
โข Day 5.7: 1.28x end to end, 1.53x decode
78 agent sessions, about $21 (1.36B tokens, 98.6% of them cache hits).
What the RSI loop found useful for speeding up:
โข Fuse chains of small ops into single kernels: norm + router + top-4 expert pick, expert FFN + combine, Q/K/V projections + KV-cache write, Q/K norm + RoPE + attention, LM head + argmax.
โข Keep the decode loop on the GPU. Tokens never round-trip to the CPU, and the GPU stays 99% busy.
โข Hard-code every shape, so the 4-bit kernels stream weights at ~75% of the memory bus.
โข Target the hardware: prefill runs on the M5 GPU's Neural Accelerators through Metal 4 tensor ops, with MoE routing and expert GEMMs fused and kept on the GPU.
(It's not Rust, by the way. llama.cpp is native C++ too. It's the specialization.)
TBH, I don't fully understand all of it yet lol. The agents wrote this in under 6 days, and it's going to take me longer than that to read it. As the human researcher, I'm now officially the slowest-improving component of my own RSI loop๐คฃ, so I'll spend the next few weeks learning from my agents.
Where I think this goes: local inference will matter more and more, and a specialized engine can beat a general one. As RSI matures, the cost of building one keeps falling toward zero. I think every model, every chip, and maybe every workload will eventually get its own inference engine.
The idea is inspired by @perplexity_ai 's Lily. Thanks to @liquidai for the great model, and Apple's MLX, whose kernels Molten adapted from.
Repo here:
github.com/zhangshuibai/moltโฆ
Spent some time tuning the 4ร NVIDIA DGX Spark fabric today, and this is the result I wanted to see.
Setup:
4ร DGX Spark / NVIDIA GB10
MikroTik CRS812
dual-RoCE fabric
NCCL 2.30.7 + CUDA 13.0
nccl-tests 2.21.1 (afd59ab)
1 GPU per node
4 MPI ranks
AllReduce
32 KiB โ 256 MiB
20 warmup iterations
100 measured iterations per size
Final 256 MiB result:
15.27 GB/s algorithm bandwidth
22.90 GB/s NCCL bus bandwidth
22.89 GB/s in-place
0 validation errors
That is roughly 183 Gbit/s bus-bandwidth equivalent on the 200 GbE RoCE path.
The more interesting part was not the peak number, though. It was understanding why the default NCCL tuning was leaving performance on the table.
With NCCL fully automatic, the mid-size region was surprisingly weak. Around 1โ8 MiB the performance could collapse badly even though large transfers eventually reached ~22 GB/s.
Using NCCL TUNING logs showed what was happening.
The automatic tuner was progressively increasing the number of channels:
128 KiB โ 8 channels
256 KiB โ 16 channels
512 KiB โ 32 channels
1 MiB and above โ 64 channels
On this particular 4-node DGX Spark topology, 64 channels were simply too aggressive for the medium-size AllReduce path.
The best reproducible configuration turned out to be:
NCCL_PROTO='^LL128'
and
NCCL_MAX_NCHANNELS=8
Important detail: ^LL128 means exclude LL128, not force it.
With that tuning, the curve became much healthier:
512 KiB โ 6.87 GB/s busbw
1 MiB โ 12.17 GB/s
2 MiB โ 17.07 GB/s
4 MiB โ 19.95 GB/s
8 MiB โ 20.54 GB/s
16 MiB โ 21.66 GB/s
32 MiB โ 22.27 GB/s
64 MiB โ 22.57 GB/s
128 MiB โ 22.74 GB/s
256 MiB โ 22.90 GB/s
Full sweep average:
13.8526 GB/s bus bandwidth
And most importantly:
# Out of bounds values : 0 OK
I also tested the newer NCCL_MAX_CTAS=8 control.
It worked, but on this exact NCCL 2.30.7 / GB10 setup it was not as fast as the older NCCL_MAX_NCHANNELS=8 setting, particularly in the 512 KiBโ2 MiB region.
So for now I am keeping the configuration that the hardware actually benchmarks best with, rather than changing it just because a newer control exists.
One more useful lesson from this:
A healthy 200 GbE fabric is not only about reaching a good peak number.
You need to look at the entire bandwidth curve.
A system can hit ~22โ23 GB/s at 256 MiB and still have a serious tuning problem at 1โ8 MiB.
NCCLโs algorithm/protocol/channel decisions matter a lot.
For this 4ร DGX Spark cluster, the current reproducible baseline is:
22.90 GB/s NCCL bus bandwidth
~183 Gbit/s equivalent
0 correctness errors
Now that is a fabric result Iโm happy to keep as the baseline.
One of the more frustrating issues I hit while building my 4 ร NVIDIA DGX Spark cluster had nothing to do with the GPUs. It was the 10GbE management network.
The Sparks use a Realtek RTL8127 NIC, and under sustained traffic the stock r8127 11.014.00-NAPI driver was showing very unhealthy RX behaviour.
This wasn't just an iperf number looking bad.
On Spark1 I was seeing tests like:
~36,000 UDP packets lost
rx_mac_error increasing by roughly the same amount
rx_mac_missed also moving
meanwhile the MikroTik CRS812 showed no corresponding FCS/drop problem
That was an important clue: the switch and cable path looked clean. The packets were effectively being lost at the host RX side.
Disabling EEE helped somewhat loss dropped from roughly 36K to ~22K packets in one test but clearly did not solve the underlying problem.
Then came an interesting workaround.
I tested:
net.ipv4.tcp_sack=0
With TCP SACK enabled, long TCP tests could fall into ugly 5โ10 second periods with throughput effectively dropping to zero and the congestion window collapsing.
With SACK disabled, Spark1 completed a 120 second test at about 8.11 Gbit/s without those hard stalls. A later 300 second validation also held around 8.11 Gbit/s.
So tcp_sack=0 was genuinely useful.
But it was NOT the fix.
The 300 second test still produced ~26,929 retransmissions. SACK-off was basically preventing the TCP stack from reacting catastrophically to the packet reordering/loss pattern coming from the NIC path.
I eventually deployed that workaround across all four Sparks because stability mattered more than theoretical protocol purity:
net.ipv4.tcp_sack = 0
But I kept looking for the actual cause.
The real improvement came from moving Sparks to the newer custom Realtek r8127 11.016.00-NAPI-RSS driver and fixing the receive side configuration:
RSS enabled
8 RX / 8 TX queues
corrected RX CPU affinity
RXMITI01 tuning
GRO enabled
DASH disabled
persistent warm reboot handling
That changed the character of the problem completely.
The catastrophic RX missed/error growth stopped, UDP became dramatically cleaner, and TCP became stable in the 8+ Gbit/s range.
Some representative results:
single-stream TCP: ~7.2โ8.2 Gbit/s
4 streams: ~8.5 Gbit/s
8 streams: ~8.9 Gbit/s
Spark โ Mac: ~9.05 Gbit/s
Spark โ Spark: ~9.4 Gbit/s
In one particularly clean UDP run I saw roughly 5.38 Gbit/s with only 48 packets lost out of ~4.64 million around 0.001%.
That is a very different system from the one dropping tens of thousands of packets at the RX MAC.
Later I also performed the full NVIDIA/DGX platform update across all four Sparks. That is now part of the cluster baseline, but I keep the two issues conceptually separate: the NVIDIA platform update and the RTL8127 driver/RSS work are not the same fix.
The biggest lesson from this debugging session:
tcp_sack=0 was a good mitigation, not a cure.
It removed the nasty TCP stalls and made the machines usable while debugging, but the retransmissions were telling us the underlying problem was still there.
The actual fix had to happen lower in the stack at the NIC driver, queueing, RSS and interrupt affinity level.
And this is exactly why looking only at โiperf says 8 Gbit/sโ can be misleading.
Always look at:
retransmissions
NIC hardware counters
RX missed/error counters
queue distribution
IRQ affinity
switch counters
packet loss
Throughput tells you how fast the system went.
The counters tell you whether it got there cleanly.
4 x Sparks
DGX OS 7.6.0
kernel 7.0.0-1019-nvidia
NVIDIA driver 580.178.04
EC 3.13.7
SoC / UEFI 2.158.15
USB-PD 0.5.25
tcp_sack 0
0xCS retweeted
2 Sparks vs 4 Sparks / 12,000$ vs 24,000$
@bertholomusai 's DeepSeek-V4.1-Flash-1017 on two DGX Sparks against four, same voxel-world prompt: 73.7 min against 49.7 min.
Two Sparks: EXL3 2.5 bpw, TP2, recipe v0.7.2 (commit 30a5bf9), as of Oct 24.
Four Sparks: EXL3 4.0 bpw, TP4, recipe commit 8f84e7a, as of Oct 22.
That is a different quant and a different recipe version, so this does not isolate the Spark count.
Both ran in the DeepSeek Harness 0.2.1-rc.2 with the recipe's own sampling, thinking on, one prompt, one try, no help.
Two Sparks, world LUMENFALL:
Agent time: 73.7 min
Tokens written: 138,675
Browser check: 8.0/10
My rating of the world: 5.5/10
Decode, code: 70.5 tok/s
Decode, prose: 70.6 tok/s
500 words: about 10.0 s
Eight requests at once: 106.3 tok/s in total
Four Sparks, world Aetherfall:
Agent time: 49.7 min
Tokens written: 121,707
Browser check: 9.0/10, scored with the older capture method
My rating of the world: 7.5/10
Decode, code: 82.8 tok/s
Decode, prose: 80.8 tok/s
500 words: about 9.1 s
Eight requests at once: 141.2 tok/s in total
Context: 1,048,576 tokens on both, the recipe's setting.
Answer limit in the build test: 262,144 tokens on both, the figure on the model card. The recipe's default is 32,768, so I raised it.
Open: the two-Spark browser check took 2 points off, for the orbit-by-mouse test and the frame rate in the test's software rendering. Its closing screenshot stalled, so I re-ran the same checks with a different capture method.
My four Sparks are cabled as a switchless ring, so that run used the recipe's neighbor-ring mode.
The two-Spark recipe wants RDMA GID index 5, which my Sparks did not have, so I gave the link ports IPv4 addresses. One Spark, an ASUS GX10, has an older driver.
Are the extra 12,000$ worth it?
Play both worlds in your browser. Every number, the exact command and the raw output:
Two Sparks: niklaslenz.com/v2/deepseek-vโฆ
Four Sparks: niklaslenz.com/v2/deepseek-vโฆ
Recipe, quant and TensorFold fork by @bertholomusai. TensorFold engine by @ashxhart. Model by DeepSeek.
0xCS retweeted
watch! i ran ling 3.1 flash on the same prompt twice with hermes agent on my floating tree spec
once as a q4 gguf across 4x dgx sparks and once on the api, with the same agent settings both times
> q4 over llama[.]cpp rpc: 12h 26m
> api, novita on openrouter: 57 min
> 7.5 vs 98.6 tok/s decode
> 325k vs 228k tokens out
> 12/12 tests + visual gate on both
> $0.27 for the whole api run
both builds pass everything, the api build has the better tree with real branches and 8,270 veined leaves, the q4 build kept the better cliff
the video is one camera path on both builds, frame for frame, q4 on the left and the api on the right
my verdict:
ling 3.1 flash is a strong model and the speed gap is the setup, q4 over rpc proves it fits on 4 sparks, for agent work you want it on a real serving stack, and an official 4bit build would make it practical locally
the long turns question from the 12 hour build is answered in the thread below ๐
i've run antling's models since they started and new ling 3.1 flash is the first one that feels like a real leap from them
this is the floating tree it built on the api:
> 560b moe, 25b active
> 57 min on one prompt
> 77 requests, 228k tokens out
> 98.6 tok/s on openrouter (novita)
> 12/12 tests, visual gate green
> 2,051 lines across 9 modules
> $0.27 for the whole build
i can feel their models improve every release, ling 3.0 flash couldn't finish a 3d scene like this on a single dgx spark
sure 3.1 is a much bigger model, 560b total and 25b active against 3.0 flash's 124b and 5.1b, but the capability jumped with it
the scene has taste and every part is complete and works and it actually looks good
next i'm posting the q4 gguf i ran on @niklaslenz_ai's 4x dgx sparks and this api build side by side, so you can see the difference between the gguf and the api
stay tuned anon, ai is moving at this speed and i love it, the acceleration is insane in this space
0xCS retweeted
Qwen3.8-Flash-Next kits v5.3.1, solo + two-Spark cluster
First token on a cached 200k context: 0.67 s on one Spark, 0.95 s on two (was 4.1 s).
โก๏ธ github.com/myllmbox/qwen38-fโฆ
โก๏ธ github.com/myllmbox/qwen38-fโฆ
0xCS retweeted
MiniMax-M3.1-Flash TP4 on 4ร DGX Spark, day 0 ๐
โข Prose decode: 39.7 tok/s at c1
โข Structured decode: 62.2 tok/s at c1
โข Aggregate: 166 tok/s prose, 261 tok/s structured at c16
โข speedbench quick: 38.4 tok/s across 30 prompts
โข 262k context, ~1.37M-token FP8 KV cache
โข MiniMax's own NVFP4 experts + MXFP8 weights, bundled DSpark draft
โข Runs on GB10 (sm_121): FP8 KV, Triton sparse attention, Marlin experts (the NVFP4 KV cache is SM100-only)
First measured bring-up. Speed tuning and quality gates are next.
Full recipe, measurements and credits:
github.com/knapcio/MiniMax-Mโฆ
0xCS retweeted
8-node DGX Spark clusters are now officially supported by @NVIDIAAI in both switched and switchless(ring) configurations. Read the blog post below. More to come!
developer.nvidia.com/blog/ruโฆ
0xCS retweeted
So happy these MiniMax M3.1 weights were released early! ๐ Iโm currently working on a one spark EXL3 quant with my new and improved SAGE-EXL3 process that will keep most of the quality of this model the same way I did Deepseek v4.1 Flash 1017 ! Stay tuned! ๐
MiniMax M3.1 Flash weights are now public. 428B parameters total, 23B activated per token.
Agentic performance: A generational leap in agentic tasks, coding, and cybersecurity โ 76.2 on BankerToolBench, 73.7% on DeepSWE v1.1, 61.7 on OSWorld V2, 29.6 on Agents' Last Exam, and 82.2 on CyberGym.
Efficient by design: MiniMax Sparse Attention and DSpark speculative decoding improve inference efficiency.
Hugging Face:
huggingface.co/MiniMaxAI/Minโฆ
ModelScope:
modelscope.cn/models/MiniMaxโฆ
API: platform.minimax.io
M Plan: minimax.io/m-planMiniMax
MCode for desktop: agent.minimax.io/download
After I finally found some time, I sat down and moved my cluster into a proper rack.
4ร NVIDIA DGX Spark
1ร MikroTik CRS812
2ร 400G โ 2ร200G DAC breakouts
Dedicated 10GbE management
2 shelves, rack PDU, cable management all packed into a compact 8U rack on wheels.
The 400G breakouts from the CRS812 feed all four Sparks at 200G, while the separate management network keeps administration clean.
A lot of compute in a very small footprint. ๐คโก
0xCS retweeted
TandemLLM now runs @Recursive_SI Flint
Three options on one DGX Spark: Recursive's BF16, and two EXL3 packs I made, 4.0 and 6.0.
The 4.0 pack decodes at 128 to 136 tok/s plain
Introducing Flint, @Recursive_SIโs first model: a fast local coding model built with our automated research system, the Eureka Machine.
โข Best-in-class fast local coding
โข Up to 2x faster than comparable models: 125+ tokens/sec on a MacBook Pro, 150+ on RTX Spark
โข Built from scratch in under 4 months
Private, unmetered intelligence on your own machine. Open weights available now on @huggingface, run it locally on your favorite framework or try it for free on @opencode or @pidotdev.
Reduce your attack surface
October has been an absolute nightmare for software
> Railway - customer secrets exposure
> LiteLLM - malicious PyPI + Docker releases
> Zeabur - infrastructure breached
> OpenCode - AI API keys exposed
> OVHcloud CLI - official binaries backdoored
> Rank Math PRO - malware in official plugin
> TeamUpdraft - customer data compromised
> ConnectWise - cloud backup identities compromised
> Tensorlake - npm package infected
> SubQuery - backdoored npm package
> GhostAction - GitHub repos with malicious workflows
> PHP - critical RCE vulnerabilities
> Node.js - security vulnerabilities patched
> eMudhra - certificate authority keys compromised
> Tailscale - eMudhra incident
At this point, the biggest security risk might not be the code we write, but the code we trust...
0xCS retweeted
KindlingAI still ahead of NVIDIA's Flare (by 2x!) on GLM 5.3 Flash.
EXLR8 build is literally 2x the code decode and ~20% faster on prose.
huggingface.co/kindlingai/glโฆ
0xCS retweeted
Qwen-Image-2.1 on a DGX Spark: 28.7 s per 1024ยฒ image.
Mac Studio M5 Ultra: 29.9 s.
RTX 5090 for reference: 7.6 s.
Same four prompts, three seeds each, enhancer off for the speed.
In my ratings (one rater, enhancer off and on averaged) the Mac scores 7.1 and the Spark 6.8 out of 10.
The Mac runs mflux with 8-bit weights, the Spark and the 5090 run ComfyUI INT8. No enhancer time measured on the Mac; its enhancer-on images use texts written on the 5090.
All images and raw data: niklaslenz.com/image/
Qwen-Image-2.1 by @Alibaba_Qwen
Things are getting worse every day.
Bad news, NVIDIA CMP 170HX price curve is insane: $200 last year, $1,500 last month, now over $3,000โstill less than half an RTX 5090. Specs are wild: A100-class performance, 64โ80GB VRAM, bandwidth close to an RTX 5090, SM80 compute, and CUDA 13 support. If you donโt want to tinker with V100s, an unlocked 170HX is the no-brainer combo of value and convenience. Good news, supply is still plentiful.
0xCS retweeted
SM120 : RTX5090
SM121 : DGX Spark๏ผGB10๏ผ
ๅใใซNVIDIAใๅ
ฌ้ใใใฆใๆจ่ซใตใผใใผFlare
ใปใกใขใชๅฉ็จๅน็ใฎๆนๅ
ใป้ทๆ่ใงใฎใญใฃใใทใฅ็ฎก็ใฎๆ้ฉๅ
ใปๆจ่ซๅฆ็ใฎใชใผใใผใใใๅๆธ
ใปๅฟ็ญ้
ๅปถใฎๆนๅ
ใใฎ่พบใใใกใชใใใใใใๆฏ่ผใใฆใฟใชใใจใใใใชใ
github.com/NVIDIA/flare-locaโฆ
0xCS retweeted
๐จ ๐ ๐ฌ ๐ญ.๐ฒ๐ฎ ๐๐ฃ๐ช ๐๐๐๐ฃ๐ฆ๐๐๐ ๐ค๐จ๐๐ก๐ง ๐๐จ๐ฆ๐ง ๐๐๐๐ง ๐ง๐ช๐ข ๐๐๐๐๐๐ฅ-๐ฃ๐ฅ๐๐๐๐ฆ๐๐ข๐ก, ๐ง๐ช๐ข-๐ฆ๐ฃ๐๐ฅ๐ ๐ค๐จ๐๐ก๐ง๐ฆ! ๐งช๐ฅ
And I have the REAL benchmark results to prove it! ๐คฏ
Remember my DeepSeek V4.1 Flash 1017 SAGE-EXL3 release from YESTERDAY?
๐ฆ 1.62 bpw routed experts
๐ป 99.61% code answer-token agreement
๐ฅ๏ธ ONE DGX Spark @nvidiaai
Well, I promised to go beyond KL divergence and token agreement and test ACTUAL coding, mathematics, reasoning and tool execution.
๐ง๐๐ ๐ฅ๐๐ฆ๐จ๐๐ง๐ฆ ๐๐ฅ๐ ๐๐ก!
โโโโโโโโโโโโโโโโโโโโ
๐ ๐๐๐๐ฃ๐ฆ๐๐๐ ๐ฉ๐ฐ.๐ญ ๐๐๐๐ฆ๐ ๐ญ๐ฌ๐ญ๐ณ โ ๐ก๐๐ก๐-๐ฆ๐จ๐๐ง๐ ๐ฆ๐๐ข๐ช๐๐ข๐ช๐ก
353 matched benchmark questions.
9 evaluation suites.
Strict pass@1 scoring.
Temperature 0.
๐ข ๐ ๐ฌ ๐ฆ๐๐๐-๐๐ซ๐๐ฏ โ ๐ญ.๐ฒ๐ฎ ๐๐ฃ๐ช
๐ฅ๏ธ 1ร DGX Spark
๐ ๐ต๐ฐ.๐ญ% โ ๐ฏ๐ฏ๐ฎ/๐ฏ๐ฑ๐ฏ ๐ฃ๐๐ฆ๐ฆ๐๐!
๐ต ๐๐ ๐ก๐๐ช โ ๐ฎ.๐ด ๐๐ฃ๐ช
๐ฅ๏ธ 2ร DGX Sparks
๐ 90.1% โ 318/353
๐ฃ ๐ ๐๐๐๐๐ โ ๐ฎ.๐ต ๐๐ฃ๐ช
๐ฅ๏ธ 2ร DGX Sparks
๐ 88.4% โ 312/353
๐'๐ ๐จ๐ฆ๐๐ก๐ ๐ฐ๐ฎโ๐ฐ๐ฐ% ๐๐๐ช๐๐ฅ ๐๐๐ง๐ฆ ๐ฃ๐๐ฅ ๐ฅ๐ข๐จ๐ง๐๐ ๐๐ซ๐ฃ๐๐ฅ๐ง ๐ฃ๐๐ฅ๐๐ ๐๐ง๐๐ฅ...
๐๐ก๐ ๐ฆ๐๐ข๐ฅ๐๐ก๐ ๐๐๐๐๐๐ฅ ๐ข๐ก ๐ข๐ก๐ ๐ฆ๐ฃ๐๐ฅ๐! ๐คฏ
โโโโโโโโโโโโโโโโโโโโ
๐ป ๐ง๐๐ ๐๐ข๐๐๐ก๐ ๐ฅ๐๐ฆ๐จ๐๐ง๐ฆ ๐๐ฅ๐ ๐๐ก๐๐ฅ๐๐๐๐๐๐!
Scores shown as SAGE / JJ / Mia:
๐ HumanEval+
39/40 vs 34/40 vs 34/40
๐ MBPP+
53/60 vs 50/60 vs 49/60
๐ LiveCodeBench
15/17 vs 14/17 vs 11/17
๐ง ๐ฅ๐๐๐ฆ๐ข๐ก๐๐ก๐ & ๐ ๐๐ง๐
๐ MMLU-Pro STEM
95/100 vs 93/100 vs 91/100
๐ MATH-500
40/40 vs 36/40 vs 36/40
๐๐๐ฉ๐ ๐ข๐จ๐ง๐ฅ๐๐๐๐ง ๐ช๐๐ก๐ฆ! ๐ฅ
I also tied both models on:
โ
MGSM: 40/40
โ
BFCL: 39/40
โ
SciCode first-step: 6/10
For full transparency, both JJ and Mia won the multi-step tool-chain test, scoring 6/6 against my 5/6.
๐ฆ๐ข ๐ง๐๐๐ง'๐ฆ ๐ฑ ๐ช๐๐ก๐ฆ, ๐ฏ ๐ง๐๐๐ฆ, ๐๐ก๐ ๐ญ ๐๐ข๐ฆ๐ฆ.
All while using a substantially lower expert bitrate!
โโโโโโโโโโโโโโโโโโโโ
๐ ๐๐๐ ๐จ๐ฃ๐๐๐ง๐: ๐ ๐ฌ ๐ฃ๐จ๐๐๐๐ ๐๐๐๐ ๐ฅ๐๐๐๐ฃ๐ ๐๐ฆ ๐ก๐ข๐ช ๐ฃ๐จ๐ฆ๐๐๐ก๐ ๐ฑ๐ฎ๐ฐ๐ ๐๐ข๐ก๐ง๐๐ซ๐ง!
YES. ON ONE DGX SPARK! ๐คฏ
With the latest W2 kernel, updated expert set and KV-cache configuration, I just completed an actual long-context retrieval test!
๐ ๐๐จ๐๐-๐๐๐ก๐๐ง๐ ๐ฃ๐ฅ๐ข๐ ๐ฃ๐ง ๐ง๐๐ฆ๐ง
๐ง 518,103 INPUT TOKENS!
๐ฏ Fact planted HALFWAY through the context
โ
THE MODEL FOUND THE FACT!
โฑ๏ธ Time to first token: 470 seconds (~7.8 minutes)
๐ Prefill: ~1,100 tok/s
โก Decode at that context depth: 25.6 tok/s
๐พ Minimum available memory: 2.19 GiB
๐ก๏ธ Memory safety floor: 0.5 GiB
And on SHORTER prompts?
โก ~36 TOK/S DECODE!
๐ง ~130.5 ms per speculative step
๐ฏ ~4.7 accepted tokens per step
๐ช๐'๐ฅ๐ ๐ง๐๐๐๐๐ก๐ ๐๐๐ข๐จ๐ง ๐ข๐ฉ๐๐ฅ ๐๐๐๐ ๐ ๐ ๐๐๐๐๐ข๐ก ๐๐ก๐ฃ๐จ๐ง ๐ง๐ข๐๐๐ก๐ฆ ๐ฃ๐ฅ๐ข๐๐๐ฆ๐ฆ๐๐ ๐ข๐ก ๐ ๐ฆ๐๐ก๐๐๐ ๐ฆ๐ฃ๐๐ฅ๐! ๐ฅ
To be clear, this was a successful full-length retrieval test, not a claim of perfect accuracy across every possible long-context task.
And the 7.8-minute prefill is a real trade-off at that depth.
But seeing a model this large, with this level of compression, successfully retrieve information buried inside 518K tokens on ONE 128GB machine is an exciting milestone for local AI!
โโโโโโโโโโโโโโโโโโโโ
โก ๐๐ก๐ ๐ง๐๐ ๐ข๐ก๐-๐ฆ๐ฃ๐๐ฅ๐ ๐ฃ๐๐ฅ๐๐ข๐ฅ๐ ๐๐ก๐๐ ๐๐๐๐ฃ๐ฆ ๐๐ ๐ฃ๐ฅ๐ข๐ฉ๐๐ก๐!
Separately, my latest REXL3 development measurements at a 256K context configuration:
โก 55.35 tok/s single-stream decode
๐ 2,045 tok/s peak prefill
โฑ๏ธ 509 ms TTFT on a 512-token prompt
Those REXL3 numbers come from a different engine and workload than the new public vLLM 518K-token test.
And here's something else I'm particularly excited about...
๐๐๐ ๐ญ๐ฑ,๐ฏ๐ฒ๐ฌ ๐ฅ๐ข๐จ๐ง๐๐ ๐๐ซ๐ฃ๐๐ฅ๐ง๐ฆ + ๐ฏ๐ด๐ฐ ๐๐ฅ๐๐๐ง ๐๐ซ๐ฃ๐๐ฅ๐ง๐ฆ ๐๐ฅ๐ ๐ญ๐ฌ๐ฌ% ๐๐ฃ๐จ-๐ฅ๐๐ฆ๐๐๐๐ก๐ง ๐๐ก ๐ ๐ฌ ๐ง๐๐ฆ๐ง๐๐ ๐ฅ๐๐ซ๐๐ฏ ๐๐ข๐ก๐๐๐๐จ๐ฅ๐๐ง๐๐ข๐ก!
ZERO streamed experts.
๐ข๐ก๐. ๐ญ๐ฎ๐ด๐๐. ๐๐๐ซ. ๐ฆ๐ฃ๐๐ฅ๐. ๐
@mr_r0b0t and I are continuing to refine REXL3 before its release. More on that soon! ๐
โโโโโโโโโโโโโโโโโโโโ
๐ ๐ง๐๐๐ฆ ๐๐ฆ ๐ช๐๐ฌ ๐ ๐๐จ๐๐๐ ๐๐ข๐ฅ ๐๐ข๐๐๐ ๐๐.
Just YESTERDAY, I shared my goal of bringing extremely aggressive, high-quality quantization to people who don't have access to massive compute clusters.
With the support of @AWERXAI, I've been working tirelessly to push that vision forward.
And now we're seeing a 1.62 bpw expert quant outperform higher-bitrate alternatives on actual tasks, not just token-level fidelity measurements.
This doesn't establish mathematical losslessness or superiority on every possible workload. These are matched subsets of nine benchmark suites, not full public leaderboard scores.
But it demonstrates something IMPORTANT:
๐ ๐ข๐ฅ๐ ๐๐๐ง๐ฆ ๐๐ข๐ก'๐ง ๐๐จ๐ง๐ข๐ ๐๐ง๐๐๐๐๐๐ฌ ๐ ๐๐๐ก ๐๐๐ง๐ง๐๐ฅ ๐ฅ๐๐ฆ๐จ๐๐ง๐ฆ.
And more hardware doesn't automatically mean you need it!
Huge respect to @jayleaton for his contributions and for giving the community more ways to run these incredible models.
And thank you @AWERXAI for supporting this work. ๐
๐ ๐ ๐ฌ ๐ฆ๐๐๐-๐๐ซ๐๐ฏ ๐ค๐จ๐๐ก๐ง
huggingface.co/vcruz305/DSV4โฆ
๐ ๏ธ ๐ข๐ก๐-๐ฆ๐ฃ๐๐ฅ๐ ๐๐๐๐ ๐ฅ๐๐๐๐ฃ๐
github.com/vcruz305/DSV4.1-Fโฆ
๐๐ฅ๐ข๐ก๐ง๐๐๐ฅ ๐ ๐ข๐๐๐๐ฆ. ๐๐ข๐๐๐ ๐๐๐ฅ๐๐ช๐๐ฅ๐. ๐ฅ๐๐๐ ๐ฅ๐๐ฆ๐จ๐๐ง๐ฆ.
๐๐๐ง'๐ฆ ๐๐๐๐ฃ ๐ฃ๐จ๐ฆ๐๐๐ก๐! ๐๐ฅ
๐จ ๐ ๐๐จ๐ฆ๐ง ๐๐๐๐๐๐ฉ๐๐ ๐ฆ๐ข๐ ๐๐ง๐๐๐ก๐ ๐๐ก๐๐ฅ๐๐๐๐๐๐ ๐ข๐ก ๐ข๐ก๐ ๐๐๐ซ ๐ฆ๐ฃ๐๐ฅ๐! ๐งช๐ฅ
With the support of @AWERXAI, I'm excited to release something VERY special for the local AI community!
๐ง @deepseek_ai ๐๐ฒ๐ฒ๐ฝ๐ฆ๐ฒ๐ฒ๐ธ ๐ฉ๐ฐ.๐ญ ๐๐น๐ฎ๐๐ต ๐ญ๐ฌ๐ญ๐ณ
๐ฆ ๐ฆ๐๐๐-๐๐ซ๐๐ฏ ๐ฎ๐ ๐ท๐๐๐ ๐ญ.๐ฒ๐ฎ ๐ฏ๐ฝ๐ ๐ฒ๐
๐ฝ๐ฒ๐ฟ๐๐
๐ป ๐ต๐ต.๐ฒ๐ญ% ๐๐ข๐๐ ๐ฎ๐ป๐๐๐ฒ๐ฟ-๐๐ผ๐ธ๐ฒ๐ป ๐ฎ๐ด๐ฟ๐ฒ๐ฒ๐บ๐ฒ๐ป๐!
โก ๐ฏ๐ฑ.๐ฑ ๐๐ผ๐ธ/๐ ๐ฑ๐ฒ๐ฐ๐ผ๐ฑ๐ฒ ๐ผ๐ป ๐๐๐๐ !
๐ ๐ญ,๐ญ๐ด๐ต ๐๐ผ๐ธ/๐ ๐ ๐๐ซ ๐ฃ๐ฅ๐๐๐๐๐!
๐ง ๐ฎ๐ฎ๐ต๐ ๐๐ง๐ซ (๐๐๐ถ๐น๐น ๐๐ผ๐ฟ๐ธ๐ถ๐ป๐ด ๐๐ผ ๐ฝ๐๐๐ต ๐ถ๐ ๐ณ๐๐ฟ๐๐ต๐ฒ๐ฟ!)
๐๐๐ ๐ข๐ก ๐ข๐ก๐. ๐๐๐ซ. ๐ฆ๐ฃ๐๐ฅ๐. ๐คฏ
And YES, the quantized weights are AVAILABLE NOW! ๐
โโโโโโโโโโโโโโโโโโโโ
๐งช ๐ง๐๐ ๐๐๐ฆ๐ง ๐ณ๐ฎ ๐๐ข๐จ๐ฅ๐ฆ ๐๐๐ฉ๐ ๐๐๐๐ก ๐๐ก๐ฆ๐๐ก๐!
Just a few days ago, I published my article about @nvidiaai's FP4-Local format: The Good, The Bad & The Ugly.
One thing really stuck with me...
NVIDIA demonstrated what was possible when aggressive quantization and intelligent optimization come together.
But there's a problem.
๐ฅ๐๐ฃ๐ฅ๐ข๐๐จ๐๐๐ก๐ ๐ ๐๐ฆ๐ฆ๐๐ฉ๐ ๐ค๐จ๐๐ก๐ง๐๐ญ๐๐ง๐๐ข๐ก ๐ง๐ฅ๐๐๐ก๐๐ก๐ ๐ฃ๐๐ฃ๐๐๐๐ก๐๐ฆ ๐๐ฆ ๐ข๐จ๐ง ๐ข๐ ๐ฅ๐๐๐๐ ๐๐ข๐ฅ ๐ ๐ข๐ฆ๐ง ๐๐ข๐๐๐ ๐๐ ๐๐ก๐ง๐๐จ๐ฆ๐๐๐ฆ๐ง๐ฆ.
That inspired me.
I've been developing my own SAGE-EXL3 process, and I wanted to see just how far I could push the latest DeepSeek Flash release while keeping the results accessible to the community.
So I spent the last 72 hours pushing the limits.
And the results genuinely surprised me! ๐ฅ
โโโโโโโโโโโโโโโโโโโโ
๐ ๐ง๐๐ ๐ฅ๐๐ฆ๐จ๐๐ง๐ฆ ๐๐ฅ๐ ๐๐ฅ๐๐ญ๐ฌ!
I measured my 1.62 bpw expert quant against DeepSeek's native FP4 release checkpoint using held-out sequences and full-vocabulary token predictions.
And when I saw the CODE results, I could barely believe it!
๐งช ๐๐๐๐๐๐๐ง๐ฌ ๐ฉ๐ฆ ๐๐๐๐ฃ๐ฆ๐๐๐'๐ฆ ๐ก๐๐ง๐๐ฉ๐ ๐๐ฃ๐ด/๐๐ฃ๐ฐ ๐ฅ๐๐๐๐๐ฆ๐
๐ป ๐๐ข๐๐ โ ๐๐๐บ๐ฎ๐ป๐๐๐ฎ๐น
โ
99.08% Top-1 Agreement
โ
๐ต๐ต.๐ฒ๐ญ% ๐๐ก๐ฆ๐ช๐๐ฅ-๐ข๐ก๐๐ฌ ๐๐๐ฅ๐๐๐ ๐๐ก๐ง!
โ
KL Divergence: 0.0186
๐ง ๐ฅ๐๐๐ฆ๐ข๐ก๐๐ก๐ โ ๐ ๐๐ง๐-๐ฑ๐ฌ๐ฌ
โ
96.04% Top-1 Agreement
โ
97.98% Answer-Only Agreement
โ
KL Divergence: 0.0349
๐ข ๐ ๐๐ง๐ โ ๐๐ฆ๐ ๐ด๐
โ
95.56% Top-1 Agreement
โ
94.19% Answer-Only Agreement
โ
KL Divergence: 0.0477
๐ ๐ข๐ฉ๐๐ฅ๐๐๐ ๐๐๐๐-๐ข๐จ๐ง ๐๐๐๐๐๐๐ง๐ฌ
๐ 94.52% Top-1 Agreement
๐ Mean KL: 0.0982
๐ฏ Median KL: 0.00019
๐ก๐๐๐ฅ-๐๐ข๐ฆ๐ฆ๐๐๐ฆ๐ฆ ๐๐ข๐๐ ๐ง๐ข๐๐๐ก ๐๐๐๐๐๐๐ง๐ฌ ๐๐ง ๐๐จ๐ฆ๐ง ๐ญ.๐ฒ๐ฎ ๐๐ฃ๐ช ๐๐ซ๐ฃ๐๐ฅ๐ง๐ฆ! ๐คฏ
These are direct teacher-forced fidelity measurements against native FP8, not task-level pass rates.
But seeing this level of agreement at such aggressive compression is EXACTLY what I was working toward!
โโโโโโโโโโโโโโโโโโโโ
โก ๐๐ก๐ ๐ฌ๐๐ฆ, ๐๐ง ๐๐๐ง๐จ๐๐๐๐ฌ ๐ฅ๐จ๐ก๐ฆ ๐ข๐ก ๐ข๐ก๐ ๐ฆ๐ฃ๐๐ฅ๐!
Not two Sparks.
Not four Sparks.
Not an expensive rented GPU cluster.
๐ข๐ก๐ ๐ญ๐ฎ๐ด๐๐ ๐๐๐ซ ๐ฆ๐ฃ๐๐ฅ๐.
My latest vLLM measurements:
โก Decode: 35.5 tok/s
๐ Maximum measured prefill: 1,189 tok/s
For a model this large, compressed this aggressively, and maintaining this level of fidelity...
๐ง๐๐๐ง'๐ฆ ๐ ๐๐๐ ๐๐๐๐ ๐๐ข๐ฅ ๐๐ข๐๐๐ ๐๐! ๐
And remember, 1.62 bpw refers to the routed experts, not every parameter in the full model.
โโโโโโโโโโโโโโโโโโโโ
๐ ๐ช๐๐ฌ ๐ง๐๐๐ฆ ๐ ๐๐ง๐ง๐๐ฅ๐ฆ.
I have a simple goal.
๐ ๐๐๐ ๐ฃ๐ข๐ช๐๐ฅ๐๐จ๐ ๐๐ ๐๐๐๐๐ฆ๐ฆ๐๐๐๐ ๐ง๐ข ๐ฃ๐๐ข๐ฃ๐๐ ๐ช๐๐ข ๐๐ข๐ก'๐ง ๐๐๐ฉ๐ ๐๐๐๐๐ฆ๐ฆ ๐ง๐ข ๐๐๐ง๐๐๐๐ก๐ง๐๐ฅ๐ฆ.
I don't want the future of local AI to be determined by who can rent the most GPUs or afford the largest training runs.
I want developers, independent researchers and enthusiasts to have the freedom to experiment, quantize, optimize and run powerful models on hardware they actually OWN.
NVIDIA's FP4-Local research showed me just how much potential remains in aggressive quantization.
My goal with SAGE-EXL3 is to keep pushing that frontier while making advanced quantization more practical for the wider community.
Today, I'm sharing the quantized checkpoint and the one-Spark serving recipe.
๐ง๐๐๐ฆ ๐๐ฆ ๐๐จ๐ฆ๐ง ๐ง๐๐ ๐๐๐๐๐ก๐ก๐๐ก๐.
โโโโโโโโโโโโโโโโโโโโ
๐งช ๐ก๐ข๐ช ๐๐ข๐ฅ ๐ง๐๐ ๐ฅ๐๐๐ ๐ง๐๐ฆ๐ง!
The token-level fidelity results are incredible, but I'm not stopping there.
I'm currently running a broader evaluation campaign to measure how closely this quant preserves the native FP8 model's actual task performance.
๐ป HumanEval+ / MBPP+
๐ง MMLU-Pro STEM / MATH-500
๐ฌ SciCode / LiveCodeBench
๐ง BFCL / Multi-step tool chains
๐ MGSM
๐ง๐๐ ๐๐ข๐๐ ๐๐ฆ ๐ง๐ข ๐ฃ๐ฅ๐ข๐ฉ๐ ๐ง๐๐๐ง ๐๐๐๐ฅ๐๐ฆ๐ฆ๐๐ฉ๐ ๐๐ข๐ ๐ฃ๐ฅ๐๐ฆ๐ฆ๐๐ข๐ก ๐๐ข๐๐ฆ๐ก'๐ง ๐๐๐ฉ๐ ๐ง๐ข ๐ ๐๐๐ก ๐๐๐๐ฅ๐๐ฆ๐ฆ๐๐ฉ๐ ๐ค๐จ๐๐๐๐ง๐ฌ ๐๐ข๐ฆ๐ฆ!
Those task-level results are coming next! ๐
โโโโโโโโโโโโโโโโโโโโ
๐ ๐๐ก๐ ๐ข๐ก๐ ๐๐๐ง๐ง๐๐ ๐ง๐๐๐ฆ๐๐ฅ...
@mr_r0b0t and I have also been working on an GB10 specialized inference engine called ๐ฅ๐๐ซ๐๐ฏ.
Early testing has been VERY exciting, including 60 tok/s with bursts reaching 74tok/s!
We're putting the finishing touches on it and making sure everything is solid before releasing it.
More on that VERY soon. ๐
โโโโโโโโโโโโโโโโโโโโ
๐ ๐ ๐๐จ๐๐ ๐ง๐๐๐ก๐ ๐ฌ๐ข๐จ!
To @AWERXAI for supporting my work and helping make this contribution to the community possible.
To @nvidiaai for the FP4-Local research that inspired me to push this experiment even further.
To @justinn_builds for his initial vllm TP4 recipe that got me started in the right direction for loading this model up.
To @mr_r0b0t for collaborating with me on what's coming next!
And to @deepseek_ai for continuing to release powerful open-weight models that make community experiments like this possible.
๐ง๐๐ ๐๐๐ฆ๐ง ๐ฃ๐๐ฅ๐ง ๐ข๐ ๐ข๐ฃ๐๐ก ๐๐ ๐๐ฆ ๐ช๐๐๐ง ๐ง๐๐ ๐๐ข๐ ๐ ๐จ๐ก๐๐ง๐ฌ ๐๐๐ก ๐๐จ๐๐๐ ๐ข๐ก ๐ง๐ข๐ฃ ๐ข๐ ๐๐ง. ๐
๐ ๐๐ข๐ช๐ก๐๐ข๐๐ ๐ ๐ฌ ๐ฆ๐๐๐-๐๐ซ๐๐ฏ
huggingface.co/vcruz305/DSV4โฆ
๐ ๏ธ ๐ข๐ก๐-๐ฆ๐ฃ๐๐ฅ๐ ๐ฅ๐๐๐๐ฃ๐
github.com/vcruz305/DSV4.1-Fโฆ
๐ข๐ก๐ ๐ฆ๐ฃ๐๐ฅ๐.
๐ญ.๐ฒ๐ฎ ๐๐ฃ๐ช ๐๐ซ๐ฃ๐๐ฅ๐ง๐ฆ.
๐ต๐ต.๐ฒ๐ญ% ๐๐ข๐๐ ๐๐ก๐ฆ๐ช๐๐ฅ ๐๐๐ฅ๐๐๐ ๐๐ก๐ง.
๐๐๐ง'๐ฆ ๐๐๐๐ฃ ๐ฃ๐จ๐ฆ๐๐๐ก๐ ๐๐ข๐๐๐ ๐๐ ๐๐ข๐ฅ๐ช๐๐ฅ๐! ๐๐ฅ
