Spent some time tuning the 4× NVIDIA DGX Spark fabric today, and this is the result I wanted to see. Setup: 4× DGX Spark / NVIDIA GB10 MikroTik CRS812 dual-RoCE fabric NCCL 2.30.7 + CUDA 13.0 nccl-tests 2.21.1 (afd59ab) 1 GPU per node 4 MPI ranks AllReduce 32 KiB → 256 MiB 20 warmup iterations 100 measured iterations per size Final 256 MiB result: 15.27 GB/s algorithm bandwidth 22.90 GB/s NCCL bus bandwidth 22.89 GB/s in-place 0 validation errors That is roughly 183 Gbit/s bus-bandwidth equivalent on the 200 GbE RoCE path. The more interesting part was not the peak number, though. It was understanding why the default NCCL tuning was leaving performance on the table. With NCCL fully automatic, the mid-size region was surprisingly weak. Around 1–8 MiB the performance could collapse badly even though large transfers eventually reached ~22 GB/s. Using NCCL TUNING logs showed what was happening. The automatic tuner was progressively increasing the number of channels: 128 KiB → 8 channels 256 KiB → 16 channels 512 KiB → 32 channels 1 MiB and above → 64 channels On this particular 4-node DGX Spark topology, 64 channels were simply too aggressive for the medium-size AllReduce path. The best reproducible configuration turned out to be: NCCL_PROTO='^LL128' and NCCL_MAX_NCHANNELS=8 Important detail: ^LL128 means exclude LL128, not force it. With that tuning, the curve became much healthier: 512 KiB → 6.87 GB/s busbw 1 MiB → 12.17 GB/s 2 MiB → 17.07 GB/s 4 MiB → 19.95 GB/s 8 MiB → 20.54 GB/s 16 MiB → 21.66 GB/s 32 MiB → 22.27 GB/s 64 MiB → 22.57 GB/s 128 MiB → 22.74 GB/s 256 MiB → 22.90 GB/s Full sweep average: 13.8526 GB/s bus bandwidth And most importantly: # Out of bounds values : 0 OK I also tested the newer NCCL_MAX_CTAS=8 control. It worked, but on this exact NCCL 2.30.7 / GB10 setup it was not as fast as the older NCCL_MAX_NCHANNELS=8 setting, particularly in the 512 KiB–2 MiB region. So for now I am keeping the configuration that the hardware actually benchmarks best with, rather than changing it just because a newer control exists. One more useful lesson from this: A healthy 200 GbE fabric is not only about reaching a good peak number. You need to look at the entire bandwidth curve. A system can hit ~22–23 GB/s at 256 MiB and still have a serious tuning problem at 1–8 MiB. NCCL’s algorithm/protocol/channel decisions matter a lot. For this 4× DGX Spark cluster, the current reproducible baseline is: 22.90 GB/s NCCL bus bandwidth ~183 Gbit/s equivalent 0 correctness errors Now that is a fabric result I’m happy to keep as the baseline.