Machine Learning Engineer | Researcher | Dota2 veteran
Sri Lanka
Joined May 2009
- Tweets2.9K
- Following1.7K
- Followers553
- Likes2.1K
Samitha Jayaweera retweeted
We are announcing three new discoveries in mathematics, computer science, and physics about harmonic analysis, complexity theory, and quantum field theory.
openai.com/index/building-sc…
Samitha Jayaweera retweeted
The good news: you can mid-train yourself to the frontier!
At a fraction of the cost! Without distillation!
What we did: We took Qwen3.5 14B and mid- and post-trained it to beat every other open model and perform as well as the very best frontier models.
This is no simple toy task or format fixing, we are training the model to understand and reason over DNA sequences. Bio is the next frontier for AI and universal access will be crucial to make progress.
The fun part: because we train the model on both natural language and DNA sequences you can actually 'chat to your DNA': Ask it questions, ponder the impact of mutations, or ask it to generate a small sequence.
We'll soon see the frontier labs move away from math and cyber towards bio. It already started, and we believe researchers should have access to capable models.
We are releasing the full recipe, so you can do it, too!
Model: huggingface.co/HuggingFaceBi…
Blog: huggingface.co/spaces/Huggin…
Samitha Jayaweera retweeted
Life is good now. 🎉 Tell your agent to:
- create a new volume in you mac
- disable fseventsd (and spotlight) on it
- and move your projects folder to it
- remount it
For extra points move you pnpm cache folder to it.
(Start the conversation outside of your projects folder, Also you might need to kill a ton of processes)
Samitha Jayaweera retweeted
We were able to fine tune an open weight model (Qwen3.5-9B) on proprietary internal data at @coinbase, and it out performed frontier models, at much lower cost and speed, for a specialized task (fraud prevention in this case).
I think we'll see more of this pattern inside companies. The frontier models are amazing generalists. But increasingly you'll see specialist models out perform on more narrow tasks, either working on teams, or by themselves.
We're making it easy to fine tune models internally for our employees. We even have a skill for it in our internal harness that makes it easy to spin these up and start training them:
"/fine-tune-and-serve-llm I want to fine-tune Qwen for payments risk review. help me get started"
I imagine we'll have some backend services that could be replaced entirely by this new architecture.
Samitha Jayaweera retweeted
Try Underdog: free, capable, private AI running locally on your device
New build and Saluki 27B are out. Under 8GB, beats Qwen 3.8 27B at tool calling
Like, repost, and comment one task you wouldn’t give a cloud agent to get an invite🦴
Today we're releasing Underdog Saluki 27B
Based on Qwen 3.8 27B, Saluki is almost 7x smaller than its full-precision counterpart while it keeps 96% of its benchmark performance and beats Qwen at tool calling
It's the best model <8GB that runs in Underdog
Saluki is available today under Apache 2.0. Try today in Underdog underdog.ai/saluki
Samitha Jayaweera retweeted
realising MLX-LM-LoRA version v5.8.6 🚀
"pip install -U mlx-lm-lora"
github.com/Goekdeniz-Guelmez…
New training algorithms added:
- KLPO: KL-Regularized Policy Optimization for Critic-Free Agentic Reinforcement Learning by @yifanzhang_
- DSLA: Directional and Similarity-aware Latent Alignment by me
Updated:
- Online training backend version 2: extreme speed increase in rollout generation and general. GRPO for example have 40% to 60% speed increase and 10% to 25% less RAM usage thanks to MLX's (by tpegolotti) new VJPs.
- Offline training: general increases in training speed and RAM usage too.
More in the comments!
Give it a try and let me know what you think 😁
Samitha Jayaweera retweeted
In light of the latest Anthropic's latest not-particularly-oblique attempt to build a case against open weights, I thought I would write down how I currently think about open weight AI models.
(1) Open weights have been deeply, irreplaceably useful for AI safety.
Samitha Jayaweera retweeted
1/ Introducing Cua Perception: Cua Driver can now see.
Nothing to click? It reads the pixels. Canvases, games, remote desktops and custom-drawn apps become labelled regions an agent can act on, parsed locally on the CPU.
github.com/trycua/cua
Samitha Jayaweera retweeted
1/ Fast Computer Use is now solved with @typesafeai Jev + Cua Driver.
Available in development preview for macOS, Windows, and Linux. We call it jev-use.
Draft #3943: github.com/trycua/cua
Samitha Jayaweera retweeted
After co-inventing ChatGPT, I kept asking myself: why have superhuman chat models not led to AGI?
I’ve spent the last 2 years in stealth building a new way to train models (RLCD), and a new type of frontier AI model that we are releasing today: Jev
• 20-200x faster
• 40-400x cheaper (w/ output tokens free)
• Frontier composable intelligence optimized for decisions
AFAICT the shortest path to AI-based economic revolution
Samitha Jayaweera retweeted
new course just dropped: CS 312: Deep Learning Alchemy w/ @stanfordnlp !
A very interesting new class concept from @tatsu_hashimoto :
- Assignments are worth 0% of the grade
- 85% of your course grade is 40-minute quizzes where you get a code diff for an experiment you've never run and ask you to predict validation loss differences
- The premise? that prediction == understanding. If we don't have any intuition as to the outcomes of an experiment, then it's often safe to say we don't understand it (or at least, it means we don't have beyond surface level knowledge).
And also, everything is public (in the style of cs336!)
(Answer to the question below in, and links 🧵)
Samitha Jayaweera retweeted
The VRAM barrier is officially dead.
I just ran Qwen 3.8 Flash Next (MoE) 125B A6B with a 250,000 context window on a single 24GB RTX 4090.
21 tokens/sec decode. 364 t/s prefill.
no mtp. no dflash. no kv cache quantization!
We are running datacenter models on consumer hardware.
Tested on Ubuntu 22 | CUDA 13.0 | PCIe 4.0 x16 | 110 GB DDR4 System RAM with a continuous 28k prompt across all runs.
### The Benchmarks & Scaling
# 1. Hybrid Offload (-ncmoe 40 @ 80k Context)
Offloaded 40 expert layers to the GPU, pushing VRAM to the ceiling.
./build/bin/llama-server -m Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf -c 80000 --port 8080 -v --fit off -b 4096 -ub 4096 -ncmoe 40
Prefill: 383.85 t/s | Decode: 22.52 t/s
Footprint: 23.85 GB VRAM | 97 GB RAM
# 2. Full CPU MoE Offload (-cmoe @ 80k Context)
Pinned all 512 expert layers to DDR4 RAM (-cmoe), keeping attention on the 4090.
llama.cpp flags: (Same as above, replace -ncmoe 40 with -cmoe)
Prefill: 355.72 t/s | Decode: 20.84 t/s
Footprint: 11.66 GB VRAM (12GB+ VRAM freed up!) | 110 GB RAM
# 3. The 180,000 Context Run
Prefill: 357.75 t/s | Decode: 20.98 t/s | VRAM: 15.6 GB | RAM: 110 GB
# 4. The 250,000 Context Absolute Ceiling
./build/bin/llama-server -m Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf -c 250000 --port 8080 -v --fit off -b 4096 -ub 4096 -cmoe
Prefill: 364.29 t/s | Decode: 20.97 t/s
Footprint: 18.3 GB VRAM (Still ~5.7 GB of VRAM headroom!) | 110 GB RAM
### Key Insights:
-b 4096 -ub 4096: doubles the prompt ingestion from ~150 to 364+ t/s.
-cmoe Free Lunch: Shifting expert layers to DDR4 RAM slashes VRAM from 24GB to 11.6GB with virtually zero decode penalty (22.5 -> 20.9 t/s), enabling the 250k context ceiling.
Qwen 3.8 Flash-Next (UD-Q4_K_XL) is a massive 111.4 GB model split across 4 shards. To run this architecture, you must build from the experimental PR branch (#27742) by @danielhanchen:
git clone && cd llama.cpp
git fetch origin pull/27742/head:qwen-next && git checkout qwen-next
cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=native -DBUILD_SHARED_LIBS=OFF
cmake --build build --config Release -j $(nproc) --target llama-server
A single 4090 paired with 100 GB of cheap DDR4 RAM will comfortably serve production grade 125B inference.
While Qwen 3.8 27B (dense) still holds the crown for single 3090/4090 rigs, Flash Next proves 125B hybrid models are officially viable on consumer hardware.
Hugging Face GGUF link and complete performance telemetry graphs are dropped in the replies below.
GLM 5.3 Flash VS Qwen 3.8 Flash Next, which one takes the open weights crown this week?
Qwen3.8-Flash can now be run locally! 🔥
The 125B MoE model outperforms Claude-Opus-4.6 (Max).
Run on 75GB RAM via Unsloth GGUFs.
Qwen3.8-Flash-Next enables CPU RAM / unified mem setups to deliver near VRAM speeds.
Guide: unsloth.ai/docs/models/qwen3…
GGUF: huggingface.co/unsloth/Qwen3…
Samitha Jayaweera retweeted
Claude has become a language, so I built a translator.
English <-> Claudish
Samitha Jayaweera retweeted
Inference Engineering is free for everyone to read:
baseten.co/inference-enginee…
Samitha Jayaweera retweeted
When I worked at Apple, I remember seeing code that was 15 years old. Every file had a comment at the top with the date and the author’s name. You’d open a file and read through it, sometimes touching a feature no one had touched in a decade.
Everything felt very permanent, like a house made of bricks. Everyone who made a brick put their name on it before slotting it into the house. Even in a massive company, you were proud of your bricks. Each one fit so satisfyingly into the bricks around it.
When someone was renovating a room you had worked on years ago, they’d reach out and say, “I found this brick with your name on it. Can you tell me what it was doing here?”
Sometimes old bricks would get taken out during a renovation, but they were usually kept in a pile in the corner to reuse in the future.
Now you can will whole houses and neighborhoods into existence. You don’t even see the bricks go in. They just tumble in from the sky. You walk into the new house and wave your hands to push walls back, throw up a chimney, or drop in a kitchen.
It's so fucking cool. But you forget what a brick feels like.
And it’s hard to know how sad to be about that. The houses are better, they go up faster, and building them increasingly feels like magic.
Still, every once in a while you miss holding a brick.
Good moment today to remind everyone to stay frosty with AI. I was working on making a server stoppable via the CLI, and Codex decided the best path forward was to implement an unauthenticated network API to every server to shut itself down. Excellent work.
I responded with "I don't know about that." (I usually give more direct feedback but this was so absurd) and it thought for like 20 seconds and responded "Server stopping should not be part of an unauthenticated network API." Yes. lol.
By the way, the result I was looking for was that we automatically start a locally discovery Unix-socket bound server so I wanted it to discover that (using prior mechanisms already made) and gracefully stop that.
The driver is absolutely accountable here (me). My lazy prompting led AI to think I wanted to be able to stop ANY configured server I was talking to. Shame on me for thinking it was obvious here. But, I also reviewed it and found this idiocy, so, that's where good AI drivers come in. Also me.
Samitha Jayaweera retweeted
DFlash 2 is here! Qwen3.8-27B at 70 tok/s on an M5 Max MacBook Pro.
⚡ Up to 4.6× the speed of autoregressive decoding, with the same output.
This is the next generation of DFlash, seeded at Z Lab and upgraded at Inco AI. Get one more accepted token on every pass, for free!
inco.ai/blog/dflash2/
Samitha Jayaweera retweeted
A lot of people are asking whether Qwen 3.8 27B dense runs on a Mac. I have tested it, and I can tell you it does. Here is how to run it fast.
Base M4, 32GB: 5 to 6 tok/s. Entry tier. It runs.
M1 Max, 64GB, 2021: 15 tok.s Reading speed, from a five year old laptop. (crazy wtf)
M3 Max, 48GB: 19 tok/s
M4 Pro, 48 to 64GB: 25 to 27 tok/s. Mine is the 64GB.
M4 Max, 4-bit: 47 on prose, 56 on code, with the 239MB split build. Author did not state RAM. Every Max config clears the model.
M5 Max: 73 peak, current ceiling.
If you have 16GB, as far as I can tell, the 17GB model won't load on it
My receipt. Paired test on my M4 Pro 64GB, same prompt, same answer: 12.0 tok/s decode baseline, 27.0 with the head active. 2.24x. Youssofal's published multiplier from the previous generation on an M5 Max is also 2.24x. Same trick, one generation and two chip tiers apart, same answer. The speedup survives the silicon.
Two words first, because they make every Mac benchmark legible.
Prefill is the model reading your prompt. Decode is the model writing the answer. Different jobs, different limits. Decode is memory bandwidth, and Apple Silicon is rich in bandwidth. Prefill is raw compute, and Apple Silicon is modest in compute. A Mac writes quickly and reads slowly.
The optimized setup, as of this week.
The trap first. LM Studio on Mac offers an MTP toggle. Do not flip it. Through llama.cpp's Metal backend, every speculative setting measured 11 to 24 percent slower than plain decoding (llama.cpp issue #23752, compounded in LM Studio's own tracker #1969). The guesser pays a kernel dispatch per draft on Metal, and the toll exceeds the winnings.
This model ships with a second small brain trained inside it, the MTP head. It drafts a few tokens ahead, the big model checks all guesses in one pass, every accepted guess is a free token. Same words out the other end, roughly double the speed.
To speed up, you need MTP:
The correct vehicle. MTPLX, built by Youssofal, runs the model's own MTP head through MLX with custom Metal kernels, where the dispatch toll does not exist.
He had Qwen 3.8 builds staged as placeholders before the weights went public and filled them the hour they landed. Three tiers: Bare Speed for burst chat, Optimized Speed for coding, Optimized Quality at 8-bit for long sessions.
The overnight upgrade. mlx-community split the model so the MTP head ships as a 239MB addition instead of a full re-download.
WescheNex1q had measured 29.5 tok/s on M4 Max and concluded no Mac runtime could drive the head. He re-ran with the split build and posted 46.8 on prose, 56.1 on code.
The new trick. ARahim3's mlx-dspark, released this week, drafts in blocks with a small diffusion model instead of the MTP head. Measured on my chip class: 4-bit goes 14.5 to 25.3 tok/s, and 8-bit hits 2.45x with 3.00x on math. Different guessing strategy, same verify-and-keep contract.
We should be honest about what M chips actually are.
The honest tax is prefill, and you should know its shape. My 8K test thought for about 90 seconds before writing a word. The reading ladder, tokens per second of prefill:
M1 Max: about 70.
M4 Pro: about 100. Mine.
M4 Max: about 250.
Ultra: about 400.
DGX Spark: about 2,000.
Paste giant documents all day and that gap is your whole decision. Chat and code sessions rarely feel it.
The quiet advantage of big memory, 32GB and up. Only 16 of this model's 64 layers keep a running conversation memory; the rest carry a fixed-size state. Translation: 128K of context costs about 8GB. The 4090 owner is rationing tokens at that depth. You are not.
These numbers moved twice this week. They will move again. And for scale: a 4090 with the same trick does 71.7 at short context and still 50.3 at 90K, rationing every token of cache to get there. Different machines, different wins.
None of this came from a spec sheet. It came from a tool author shipping day-zero builds, a community re-splitting weights overnight, and three people posting receipts within hours of each other.
This research is only possible by people doing work who are much smarter than me.
How to run it on your Mac and sources in reply 👇
Samitha Jayaweera retweeted
🧩 DeepSeek Harness v0.1 is now available in Developer Preview!
🔹 We’re opening it up to developers building agent harnesses worldwide and open-sourcing the codebase in MIT license.
🔹 Powered by the Cordis meta-framework, DeepSeek Harness is an agent harness built around one core idea: Everything is a plugin. Models, tools, skills, sessions, sandboxes, filesystems, loops, orchestration, and UI are ALL implemented as plugins, and can be mixed, matched, replaced, and extended.
Try it now!
github.com/deepseek-ai/deeps…
Samitha Jayaweera retweeted
Oh my god.
It's hard to express how big a positive impact this tip can have on your life if you talk to Claude a lot.
> Tell Claude to write in ASD-STE100, or Simplified Technical English
As a bonus, tell Claude to follow Zinsser's four principles of quality writing:
1. Simplicity
2. Brevity
3. Clarity
4. Humanity
h/t @DanielLockyer
