Ivan Rocha (@irr) header
Ivan Rocha (@irr) profile picture
Lisbon
Joined March 2009
Photo by Ivan Rocha (@irr)Photo by Ivan Rocha (@irr)Photo by Ivan Rocha (@irr)Photo by Ivan Rocha (@irr)Photo by Ivan Rocha (@irr)Photo by Ivan Rocha (@irr)
Ivan Rocha retweeted
if you have a dgx spark, give qwen3.8 27b at 6.0 bpw a try. it's the closest thing to the full bf16 model that still fits on one spark. the one to pick when quality matters more than speed. try it and tell me how it runs on your box. link in the first comment.
10
6
2
59
6,756
Ivan Rocha retweeted
A few more improvements to our LLM inference engine on AMD Strix Halo! Qwen3.8-Flash-Next now reaches 1,885 tok/s in exact prefill at 8K on a single node, or 2,008 tok/s with our FAST profile. DeepSeek V4.1 Flash is moving forward too. On two Strix Halo nodes over USB4, we're getting 1,165 tok/s exact and up to 1,425 tok/s FAST with longer prompts. We've also been working on reducing GPU cycles per token, improving scheduling and making multi-request serving more efficient. Some changes are already in main, while others are still being tested. It's a lot of small improvements rather than one big breakthrough, but they keep adding up.
16
5
1
68
3,610
Ivan Rocha retweeted
DGX Spark, Qwen3.8-27B: median first token for 16 agents falls from 142 s to 25 s, answers token-identical 1 Spark, 128 GB: 16 agents, one chat of about 17,000 tokens each, all follow-ups at once TensorFold 1.2.5 First token, median: 142.3 s Speed of all replies: 27.4 tok/s Chats found in cache: 0 of 16 TensorFold 1.2.7 First token, median: 25.4 s Speed of all replies: 99.0 tok/s Chats found in cache: 16 of 16 Ash's numbers on a GB10, from the 1.2.6 release notes: median first token 105 s to 25.8 s, speed 33.6 to 100.8 tok/s Setup: official TensorFold 1.2.7 package, Qwen3.8-27B MLX 4-bit with DFlash2, default flags Method: median over the 16 agents, 1.2.7 median of 3 runs (27.6 / 23.3 / 25.4 s), 1.2.5 one run Method: temperature 0, thinking off, replies up to 512 tokens (our choice) Video: replay of one recorded run at 12x with real timestamps, 1.2.7 median there 28.5 s Ash's test tool is not published, so we rebuilt the load from his PR description. Our 142.3 s for 1.2.5 is our own value, not his 105 s. TensorFold and the 1.2.6 cache work by Ash @ashxhart, DFlash2 draft by Inco AI @inco_ai, base model by Qwen @Alibaba_Qwen.
3
5
52
10,425
Ivan Rocha retweeted
Introducing Edge0-145B and Edge0-120B, @Edge0_AI’s first device-native 100B-class foundation models with open weights. Cloud-level intelligence on your laptop. Free, offline, and fast enough for real vibe coding. - Smart: beats Claude Opus 4.6 on Terminal-Bench 2.1 (86.5% vs. 78.2%) - Light: needs just 29G memory, 1/3 of a standard setup (100G+) - Fast: decodes 75+ toks/s, prefills 1050+ toks/s on RTX Spark and Mac. - Steady: handles 128K–256K tokens of context without slowing down, even on long tasks Frontier-class agentic coding. No cloud API. No token bill. Your code and data never leave your machine. Open weights are live on @huggingface. Run it locally or plug it into your favorite coding agent.
169
226
127
1,610
733,823
Ivan Rocha retweeted
As engineers, we usually want exact answers from our data. However, sometimes it's worth giving up a little accuracy to save a lot of memory. That's where probabilistic data structures come in: — Bloom filter checks if an item might be in a set. — Count-min sketch estimates how many times each item has occurred. — HyperLogLog estimates the number of distinct items. — T-digest estimates quantiles of a stream of numbers. — Top-k sketch identifies the most frequent items in a stream. I implemented these data structures in Go: antonz.org/probabilistic-dat…
3
25
200
5,218
Ivan Rocha retweeted
GLM-5.3-Flash on a single DGX Spark 3.5 bpw, not 2-bit. TideLM + @MiaAI_lab's EXL3 3.5 bpw achieves 26 tok/s on a US history prompt Pagoda voxel garden demo and DSH video in reply github.com/raspy135/tideLM
15
2
1
20
1,480
Ivan Rocha retweeted
New Pi extension for improving your prompts: pi-better. Type /better in front of your prompt and a forked subagent reads your code in the background, then rewrites the prompt with details you didn't think to ask for. Your agent can use it too when prompting its own subagents via pi-subagents. Built on @felixrieseberg's prompt from felixrieseberg.com/how-to-be… (prob my fav prompt of the year) pi install npm:pi-better github.com/nicobailon/pi-bet…
43
43
11
937
39,836
Big new open-weights model! This one could be a top model. Check out Code Arena scores. Beats DeepSeek V4.1 and Kimi K3 in coding Baidu released a 280B MoE model that activates only 14B parameters per token. And it's already challenging DeepSeek and Kimi in coding. 🇨🇳 This is ERNIE 5.5 Flash Preview. Model Stats 🧠 280B total / 14B active 📚 1 MILLION token context ⚡ Baidu reports 200+ tokens/sec on its hosted platform 🔧 Coding, tool calling and agentic workflows 🔓 Apache 2.0 weights coming soon Now look at the early Code Arena WebDev scores: 🔥 ERNIE 5.5 Flash Preview → 1668 🟢 DeepSeek V4.1 Flash (Max) → 1664 🔵 Kimi K3 (Max) → 1658 These are early results, with ERNIE's score based on Arena's automated evaluation rather than human votes. Baidu is showing ERNIE working across entire agent workflows like these examples ... 🎮 Built a third-person Unreal Engine 5.8 game in an 11-hour coding session. 🕹️ Built nearly 100 mobile mini-games from design documents. ✈️ Two agents collaborated to build an airport and Boeing 747 in Blender. ⚙️ It created an editable W16 engine in FreeCAD. 📊 Produced research reports, spreadsheets and presentations. How much hardware will we need? Raw weight math for 280B parameters: 💾 BF16 → 560GB 💾 8-bit → 280GB 💾 4-bit → 140GB 💾 3-bit → 105GB 💾 2-bit → 70GB Before quantization metadata, KV cache and runtime overhead. ⚠️ API available now. The post-trained Preview weights are NOT downloadable yet. Baidu says Apache 2.0 weights are coming. Link in ALT
22
7
1
133
11,867
Ivan Rocha retweeted
TideLM is an LLM engine for TWO DGX Spark which supports GLM-5.3 Full (Not Flash!) and Deepseek v4.1 1017 native MXFP4. I changed the checkpoint to EXLR8 after seeing @bertholomusai 's TP4 recipe. Thank you for sharing the information. More weight resident = more accurate. In the video, DSH reported 26 tok/s. github.com/raspy135/tideLM
6
3
2
28
4,031
AWS Emulator is a local emulator of the AWS APIs. You use it to test your software without calling AWS, on your own machine, on a development server, or in continuous integration (CI). github.com/aws/aws-emulator
68
Ivan Rocha retweeted
Blackfold is my Uncensored DeepSeek v4.1 1017 TP2 recipe and daily driver. It’s not just your standard DS It was built for me. I wanted reliable, I wanted fast, I wanted capable, I wanted uncensored. But I also didn’t want to waste $40,000 worth of hardware when you can get similar performance out of two Sparks. So I put it in a neat little box packaged and gift wrapped for you. Enjoy. github.com/Blackwellboy/Deep…
8
11
2
94
5,139
Ivan Rocha retweeted
I promised a DeepSeek recipe update. How about a 4x efficiency boost on 2x DGX Sparks! We are already pushing the limits of the hardware, so I thought about how else we could make improvements without more hardware. The model writes the code or math problems, and the engine actually runs it on CPU lanes instead of guessing the result with even more tokens. No tool-call round trip to the harness, and because it can perform actions in its thinking, it's not only faster but also takes fewer turns to find the right solution. On generated tasks, both setups got 60/60. With Compute Lanes: 6.3s vs 26.9s per reply, and 78% fewer tokens on average. On the original suite of tests, it even solved 5 more problems than without Compute Lanes on. It's not always a net positive though, I think that depends on the model training. On GSM8K, MATH-500 and HumanEval, the model never called the tool, and scores barely moved. This helps when a task needs real computation. PR here will hopefully be merged within a few hours to my main recipe. github.com/jayleaton/deepsee…
13
5
1
152
8,164
Serve DeepSeek-V4.1-Flash-1017 from two NVIDIA DGX Sparks through an OpenAI-compatible API, with the full 1M-token window, image input, tool calling, thinking on or off, and DSpark speculative decoding github.com/MiaAI-Lab/DeepSee…
1
2
100
Ivan Rocha retweeted
Thanks to @alexintosh and @AgentSparko for some bug reports in our GLM 5.3 flash single-DGX-Spark recipe! github.com/kindlingai/glm-5.… What's fixed: - The model wasn't processing reasoning content sent back by certain harnesses and GLM is very sensitive to this. - There was a bug in the attention on TP=1 that wouldn't properly attend to all the tokens in some cases - We bumped the quant up to bit201 - a 2-bit + toolcall calibrated quant that slightly improves KL on agentic workflows - We added more pmcgx layers to the GLM 5.3 flash model on a few that came out to be more sensitive.
5
7
1
35
2,188
Save this amazing collection of C++ articles - structured nicely. modernescpp.com/index.php/ta…
4
44
372
11,002
Ivan Rocha retweeted
Halogen 0.20.0 is out for Strix Halo • OvisOCR3 document parsing on the @AIatAMD NPU (your own weights) • OCRBench 85.7 vs 85.3 upstream • ROCm 10.1: ~9–15% less energy, same speed • 2–6% faster prompt reading on Qwen3.8-Flash-Next github.com/peonist-ai/haloge…
4
9
2
67
4,190
Ivan Rocha retweeted
This week we were blessed by: 1. @Nouamanetazi dropped MoE training handbook: huggingface.co/spaces/huggin… 2. @yb2698 started training harness adapters: huggingface.co/blog/harness-… 3. @joelniklaus released the harness leaderboard: huggingface.co/spaces/joelni… Have a lovely weekend y'all. 🤗
11
77
678
35,863
Ivan Rocha retweeted
For all llama.cpp Qwen3.8-27B users ! With dspine release from @TencentHunyuan, 27b now runs even faster. Especially for maths and code 🚀 Speed run result of Qwen3.8-27B-UD-Q4_K_M on my RTX 3090 with dflash2 and dspine! dspine is the fastest at every context depth i tested🔥 dspine vs dflash2 vs no spec - 8k: 35.6 vs 26.6 vs 28.1 - 32k: 31.1 vs 28.0 vs 23.8 - 64k: 35.5 vs 23.0 vs 19.5 - 128k: only no spec fits, 14.6 Also llama-benchy uses a novel as the text, which is the worst case for drafting. on code and math it's way better, ~125 tok/s, 4.5x over no spec - code/math: ~125 tok/s (4.5x) - mixed chat: 90 tok/s (3.2x) Trade off is context, on my 3090, dspine fits 64k for kv q8_0, with n-max of 7 max context increases to 98k. gguf: huggingface.co/aj9o9/Qwen3.8… You need my llama.cpp branch for now: github.com/Ajay9o9/llama.cpp…
THIS IS HUGE !! Now you can run Qwen3.8-27B even faster !! @TencentHunyuan just released DSpine for Qwen3.8-27B and it beats DFlash2 really well. I ported this to llama.cpp and have created ggufs for it, find all my drafters in my hf link below. Since it's not supported in upstream llama.cpp, you can clone my updated fork and give it a spin. - Baseline: 28.2 tok/s - DFlash2: 50.6 tok/s. - DSpine: 94–132 tok/s DSpine's lead over DFlash2 is about 1.75x either way. DSpine predicts more tokens per verify step on math and code (7.3–7.6 vs 5.7–5.8), which is where its big wins come from. github.com/Ajay9o9/llama.cpp… All information is provided on model card. huggingface.co/aj9o9/Qwen3.8…
28
9
4
165
14,783