Ant Group and the vLLM community have open-sourced Humming, a kernel library for quantized inference. It runs on NVIDIA GPUs from Turing onward, and on T-Head's Zhenwu 810E. At small and medium batch sizes, its W4A8 path reaches approximately 4.2× the DeepGEMM W8A8 baseline on H20 and 4.5× on H200. One W4A16 kernel compiled in 1.82 seconds with NVCC, against 23.35 seconds for CUTLASS with NVCC. The team also ran DeepSeek-V4.1-Flash on eight H20 GPUs with TP=8. Switching MoE execution from Marlin W4A16 to Humming W4A8 improved throughput by approximately 14%–43% and cut mean TTFT by approximately 25%–32%. At concurrency 64, output throughput rose from 442.37 to 633.94 tokens/s. My read: compare two rows. Humming at concurrency 64 waits 3882.94 ms for the first token. Marlin at concurrency 32 waits 3546.74 ms. So Humming serves twice the requests for about a tenth longer wait, at 633.94 tokens/s against 398.83. That is one model on one box, so I'd want to see other hardware. vllm.ai/blog/2026-10-21-humm…

Opposable is an Android app from Reval Labs that taps and types in other apps. It can now run a 2-billion-parameter screen model on the phone, with no server. The model is GUI-Owl 1.5 2B in 4 bits on llama.cpp. On a Mac M2 CPU, one step takes 7.3 s, down from 10.6 s, with the same 27 of 36 test screens right. Cropping the screen and dropping flat background tokens were part of that. Only the vision encoder went to the NPU. Compiled for the Galaxy S25 in fp16, it ran in 236 ms against 3.9 s on the M2. Only Snapdragon has an NPU build so far. On eight held-out tasks, feeding the model one subgoal at a time and checking each on screen passed 6 of 8 in both rounds. The whole task at once passed 4 and 3. My read: on the Mac, a step is 3.6 s of encoding plus 2.5 s of prefill and 1.1 s of decode. The NPU takes the encoder to about a quarter second. If the phone's prefill and decode run anywhere near those Mac numbers, they are now the slow part. getopposable.com/blog/comput…

Simon Edwardsson fine-tuned a 360M-parameter model to act as a terminal, and it runs in a browser tab. There is no shell behind it. Each keypress goes in, and the model predicts how the screen changes. On a 106-command shell test it got 80 right. Qwen2.5-Coder-0.5B and Qwen3-0.6B, given three worked examples each, got 52 and 60, on a slightly different task. The biggest gap was memory. It got 9 of 13 file readbacks right, against 2 and 3 for the prompted models. Decoding screen rows in parallel cut p95 latency on large redraws from 6.7 s to 0.87 s on an RTX 4090. To fit a browser, the vocabulary got cut. The base model's 151,936-token embedding table was 136M parameters, and the terminal only needed 333 tokens. That took it from 494M to about 360M. Int4 then brought it to 208 MB. My read: 136M out of 494M means over a quarter of the model was vocabulary this job never used. For a narrow task, trimming the vocab looks like a cheap first cut before you even get to quantizing. simedw.com/2026/10/21/world-…

IBM researchers are open sourcing a vLLM plugin that lets you watch and steer a model while it serves, with little drag on speed. Most interpretability tools run Python code inside each model layer. In vLLM that means turning off graph capture and replay, the feature behind its fast throughput. IBM's vLLM-Hook MIA captures and steers model internals before the graph is recorded, so replay stays on. IBM's technical report says it was up to 20 times faster than the earlier vLLM-Hook. It was also up to 14 times faster than a similar method that inserts code into the model. On an 8B Llama and a 1.5B Qwen, steering delayed inference by just 2% in some cases. On Llama 70B the overhead rose to 6%. My read: the 70B model paid more, but 6% is still small. IBM pitches it for last-minute safety checks, and at that price a check could plausibly stay on in production. research.ibm.com/blog/fast-i…

Anthropic has opened a research preview of Claude Duet, an app where Claude works as a proactive agent. It sends a short brief each morning and messages you when something needs your attention. It has its own virtual machine and browser, so your laptop or phone can stay off. Anthropic says it can use the over 6,000 connectors and plugins in the Claude Marketplace. Duet is built around Claude Opus 5.5. Access is by waitlist, starting with Pro and Max users in the US. It shares your plan's usage limits, and its background checks count toward them. Anthropic says it checks your accounts every few hours because checking nonstop would burn through limits much faster. Before it sends a message, makes a payment, signs a document or deletes anything, it shows you the action and waits for approval. My read: the line I'd point investors to is the hard limit. Duet can prepare and run payroll in Gusto. But trading securities and moving money between accounts are off-limits even when you approve them. claude.com/resources/article…

DeepSeek has open-sourced DeepInfer, the inference engine it says powers its production services. This first release is narrow. It serves one model, DeepSeek-V4.1-Flash, and only offline inference. It runs on NVIDIA SM100 GPUs and Huawei Ascend 950 chips. To speed up decoding, it drafts tokens ahead and uses confidence estimates to decide when to verify them. It can also force output to match a JSON schema or a grammar. A full service needs more than this. The README lists a KV cache system and load balancing across replicas as parts that are not included. Its benchmarks ran on 4 GPUs for prefill and 64 for decode. My read: deterministic output is the feature I'd check first. To get it, the README has you switch the draft-token mode off, and it is marked NVIDIA only. So exact repeatability costs you the decoding speedup, and this release doesn't offer it on Ascend. github.com/deepseek-ai/DeepI…

Microsoft Research says a 31-year-old math problem is now solved, with AI agents doing a lot of the search. The problem is the KLS conjecture from 1995. For a wide class of high-dimensional distributions, it asks whether the best flat cut is nearly as good as the best cut of any shape. The new proof says the two stay within a fixed bound, however many dimensions there are. The two researchers picked the direction and pruned dead ends. Agents tried variations of an idea in parallel. Roughly one hundred trajectories ran before the key step showed up. Unguided runs on several leading models, with about the same token budget, fell short. On the same day, a separate group of experts posted a full proof built on the first Microsoft version. A Lean 4 formalization came later. My read: the October 1 version still let the dimension creep in through an iterated logarithm. The October 4 version removed it. So the last step took three days, and the unguided runs never got there on the same tokens. That makes me think the human pruning did a lot of the work. microsoft.com/en-us/research…

Recursive has released Flint, an open-weight coding model fast enough to use on a MacBook Pro. Flint is a mixture-of-experts model with 40 billion total parameters. It activates 2.5 billion of them per token. The weights are under Apache 2.0. Recursive says Flint leads MoE models with fewer than 50 billion total parameters on agentic coding benchmarks. It also says Flint rivals dense 25 to 40 billion parameter models on coding while running more than five times faster. At short contexts, Recursive reports more than 125 tokens per second on an M4 Pro MacBook Pro. It gives more than 150 on a 64GB NVIDIA RTX Spark. The team quantized the output layer to 4 bits. Recursive says the final NVFP4 model scores within 0.7 points of the 16-bit model across its benchmarks. My read: Recursive recommends staying within 256k tokens. Its speed chart averages a 128k-token session, half that ceiling. The 125 and 150 figures are short-context speeds. So for long agent runs on a laptop, the back half of the context is the part I'd measure myself. recursive.com/updates/introd…

Google Research tested an AI system that reads a simple ultrasound sweep to estimate how far along a pregnancy is. In live clinics it matched the expert method. The scans use a standard sweep pattern. Health workers without sonography training can do it with a low-cost handheld probe. It was trained on Sonosite, GE and Butterfly probes, then paired with a Clarius C3 for this study. The model was trained on 1,968 participants from North Carolina and Zambia. It was then tuned on scans from 120 participants in Chicago, about 6% of the original training set. It was tested prospectively on mobile devices in Chicago (192 participants) and Nairobi (193 participants). No Nairobi data was used for tuning. Mean absolute error across both sites was 4.18 days for the AI. The expert sonographer method scored 4.48 days. The AI scored 4.10 days in Chicago and 4.27 days in Nairobi. The paper is in JAMA Network Open. My read: Nairobi is the number I care about. With no local data, the error there was 0.17 days worse than Chicago, about four hours. That is a new city for a model tuned on just 120 people from somewhere else. Google calls the result non-inferior to experts, and four hours of drift backs that up. research.google/blog/empower…

Vals AI launched a Finance Index on October 21. It scores agent models on finance work and weights each part by that sector's share of U.S. GDP. Finance Agent v2.1 and the Excel Modeling Benchmark v1.1 count for securities and investments at weight 1.6. Tax Agent Bench counts for tax at weight 0.5. Claude Opus 5.5 ranks first at 65.59%. Gemini 4 Argon is next at 64.89%, then Wonder Full at 64.55%. Vals says the top two are within the margin of error. The hard parts still score low. Vals says both leaders fail more than half of the tax questions, and Opus 5.5 leads Tax Agent Bench at 45.57%. Cost moves a lot more than score does. Vals says DeepSeek V4.1 Flash 1017 trails the leader by 4.7 points at 4% of its cost. My read: Opus 5.5 and Argon list the same $4 input and $20 output price. Opus costs $11.67 per test and Argon costs $6.53, so Opus costs about 1.8 times as much. Vals lists 45m00s of duration for Opus and 22m41s for Argon. So the Opus lead is 0.70 points for 1.8 times the bill. Vals notes the Opus score includes tasks that a fallback model served after a refusal. vals.ai/benchmarks/finance_i…

Wiz Research says AI helped it find critical bugs in core PHP, filed under two CVEs, and PHP shipped patched releases on October 21. The flaws come from how PHP handles out-of-memory errors. A request that dies that way can leave stale state behind, and a later request or worker can reuse it. Wiz says this allowed memory disclosure and remote code execution with no login. Wiz says PHP versions going back to 5.4 are affected by at least one of the bugs. The CVSS score is 9.2. It showed working exploits against the latest WordPress, Drupal and Joomla container images. Wiz found the first bug on August 19, using its own tools plus two frontier AI models. It says the code had been there for more than a decade, and that the models helped it adapt exploits across several apps in hours. By Wiz's timeline, a WordPress exploit was confirmed on August 21, two days after the first find. Patched releases came 61 days after that, and 63 days after the first find. If AI keeps shrinking the first number, the second one is where the risk sits. wiz.io/blog/php2shell-php-rc…

The PyTorch team at Meta posted an update on TorchTitan, its training framework, covering the past year of changes. TorchTitan can now run reinforcement learning. The new TitanRL uses the same model code from pretraining through post-training and inside vLLM. In supported on-policy setups, the trainer and generator give bitwise identical log probabilities. On an internal GB300 cluster, a DeepSeek V3 671B run got 5,324.21 tokens per second per GPU on 256 GPUs. On 512 GPUs it got 5,262.68. Global throughput went from 1,362,998 to 2,694,493 tokens per second, about 1.98x for twice the GPUs. The team also tested a new checkpoint library, torch_checkpointing, on DeepSeek V3 671B with 512 H100 GPUs and a checkpoint every 50 steps. Average write time fell from 384.4 s to 13.1 s. Average trainer blocking time fell from 841.2 ms to 34.1 ms. Loading improved by less. Average non-resharding load time went from 194.7 s to 79 s. That is about 29x faster writes but 2.5x faster loads (384.4 / 13.1 and 194.7 / 79). Saving often now looks cheap, yet a restart from a checkpoint still waits about 80 seconds to load. pytorch.org/blog/torchtitan-…

Mistral released Voxtral 3 on October 20. It is a set of speech models plus a voice agent SDK built on LiveKit. On transcription, Mistral says Voxtral Small Transcribe 3 outperforms Whisper and Scribe v2. It says the model also beats Gemini 3.5 Transcribe. The realtime model is listed at sub-500ms latency. On speech output, Voxtral Mini TTS 3 costs $13 per 1M characters. Mistral says that is less than OpenAI TTS-1, Deepgram, ElevenLabs Flash and Gemini Flash. Three of its four text-to-speech charts are labeled internal evals. The SDK joins listening, reasoning and speaking in one streaming pipeline. Mistral says each turn completes in under one second. The price list is where I'd look. Mini Transcribe 3 in batch is $0.0035 a minute. Small Transcribe 3 is $0.0040 a minute. My read: Small is the model Mistral says beats Whisper and Scribe v2. It costs about 14% more than Mini. Over an hour of audio that is 21 cents against 24 cents. If Mistral's claims hold up on your own audio, the price gap alone is a weak reason to pick Mini for batch work. mistral.ai/news/voxtral-3

A paper from a Tsinghua University team proposes a new way for AI agents to manage the memory of a long task. They call it cache engineering. Long agent runs build up a key-value (KV) cache, and the paper says KV cache I/O and storage become the dominant cost. The usual fix is to trim or summarize old history. The authors say that edit invalidates later cached states and triggers costly re-prefilling. Their method keeps the text history and drops selected cache entries. Drop removes entries from future attention. Repos adjusts the position encodings of the keys that stay. Their main policy keeps the cache of the latest 12 tool responses. Across three models and four agent benchmarks, it raised average scores by 1.7 points and cut average cost by 27.7%, compared with threshold-triggered summarization. Costs are worked out from token counts at Groq's rates for GPT-OSS-120B. Their serving engine, AKV, gave 1.18x the output throughput and a 1.51x speedup in P95 request latency. That test ran GPT-OSS-120B on eight A800 GPUs. My read: look at the SWE-Verified columns in Table 1. GPT-OSS-120B's cost fell from 20.0 to 9.7 cents, about 52%. MiroThinker-1.7-Mini's fell from 33.6 to 26.3 cents, about 22%. The 27.7% average hides a wide spread, so test your own model before counting on it. arxiv.org/abs/2610.28577

AMD says one Instinct MI355X can decode 1.9x to 3.3x faster on single requests. It published open optimized paths for three models and the scripts to reproduce them. The speedup comes from megakernels, speculative decoding and 4-bit MXFP4 weights. For two of the models, the whole forward pass runs as one persistent kernel. Qwen3.8-27B reaches about 1,100 tokens/s, 2.0x ATOM's regular path. GPT-OSS-120B reaches about 1,470 tokens/s, which AMD puts at 1.9x stock vLLM. With the same drafter on both sides, that GPT-OSS gain is 1.6x. The MI355X has two dies, each with half of the memory. Run as two logical devices, GPT-OSS-120B decodes 7.5-8% faster from data placement alone. When AMD forced every weight read across the die link, each token took about 28% longer. Look at the GPT-OSS round numbers. With the same drafter, stock vLLM accepts 4.43 tokens per round and the megakernel accepts 4.25. Stock vLLM still loses, because its round takes 5.65 ms and the megakernel's takes 3.52 ms. So in that matchup the whole gain is in the round time. AMD notes that decode rates are not whole-task speedups, and these are AMD's own tests. rocm.blogs.amd.com/software-…

Dharma-AI says it cut the inference cost of a production grader by 77% by rebuilding the model's vocabulary. It kept the same base model and training recipe. The grader runs on Qwen3-4B and scores Portuguese essays. Its tokenizer was built for many languages, so the JSON it writes on every request took many tokens. The new vocabulary, which they call VocabX, prices each token by what it costs to serve. In their case a generated token costs 7.4 times a token read. The output layer shrank from about 152,000 rows to about 65,000. On 500 requests on an RTX 5090 with vLLM, the average response fell from 350 to 101 tokens. The bench score rose from 0.703 to 0.810. A first version fused the grade into one token. It scored 0.689, below the old grader. So they gave the grade its own token. Here is the part I would check. 101 over 350 is about 29%, so the response got about 71% shorter. Cost fell 77.0%, about six points more than the drop in response length. The input was cut with the new vocabulary too. The output layer is also less than half its old size. Both could sit in that gap. This is a preview of an unpublished paper that Dharma-AI says may change, on one grader. huggingface.co/blog/Dharma-A…

Vercel published its October AI Gateway Production Index, which covers what its customers ran through the gateway in September. Open-weight models processed 71.5% of gateway tokens and took 23% of spend. Their share of spend went from one in twenty dollars in June to nearly one in four in September. The whole market grew and got cheaper at the same time. September spend was about 13 times December's, while the average price per token fell 47%. The open-weight labs look very different from each other. DeepSeek's models processed 58% of all gateway tokens for 6.6% of spend. Moonshot AI, the lab behind Kimi, took 3.6% of tokens and 13.2% of spend. Vercel estimates spend from published list prices, so real bills may differ. My read: Moonshot took twice DeepSeek's spend on about a sixteenth of its tokens. Vercel defines price per token as spend divided by tokens. On that basis an average Kimi token cost about 32 times an average DeepSeek token. vercel.com/blog/ai-gateway-p…

Artificial Analysis and Harvey updated their legal agent benchmark so that a made-up fact now zeroes the whole task. Harvey LAB-AA v1.1 runs every model through 120 private legal tasks. Each deliverable is checked against the task's source documents. One material hallucination sets that task's score to zero. The gate reshuffles the top. Grok 4.7 leads at 9.4%, with Muse Spark 1.3 at 8.9% and GPT-6 Astra at 8.6%. Without the gate, Muse Spark would lead at 26.7%. Two thirds of those passes carried a material hallucination. GPT-6 Astra averages 0.03 material hallucinations per task across the full set. Gemini 3.8 Flash averages 13.96. The team also tried six different models as the hallucination checker, on 20 tasks and eight models. GPT-6 Sol upheld 470 material hallucinations in total. Claude Opus 5.5 upheld 99. My read: Every one of the six checkers ranked Gemini 3.8 Flash's work the worst. They split on the size, though. On the same 20 tasks, GPT-6 Sol upheld 335 in Gemini's work and Gemini, checking itself, upheld 28. That is about 12 times apart. So the ranking holds up whoever checks, and the raw hallucination count depends a lot on the checker. artificialanalysis.ai/articl…

On a new design-creativity benchmark, the best of 15 AI models scores 65.5 against 92.3 for human designers. Kombai built Design Creativity Bench and scores it three ways. Originality is how different a model's page is from the other models' pages for the same brief. Creative range is how much a model changes its design when the same goal moves to a different product domain. Appropriateness is the share of the brief's acceptance criteria the page meets. The paper's table puts GPT-6 Astra first at 65.5, GPT-6.1 Sol second at 63.7 and Grok 4.7 third at 62.2. MiniMax M3 comes last of the 15 at 34.0. The human reference scores 92.3. Appropriateness is tightly packed. The models sit between 93.4 and 99.2 percent, with the human reference at 98.0. Originality runs from 0.547 up to 0.646 against 0.764 for the human. Creative range runs from 0.472 to 0.634 against 0.902. My read: run your eye down the creative range column instead of the Overall one. The two models at the top of the table hold the two lowest scores in it. Astra is at 0.472 and Sol at 0.487, and third-placed Grok 4.7 jumps to 0.603. Astra is the most original model of the 15 and second on appropriateness, and its range is 52 percent of the human's. The model at the top of this table is the one that varies least when the domain changes. github.com/kombai-io/design-…

A team deleted about 168,000 lines of comments from its codebase, and the AI coding agent working on it scored the same either way. Roberto Aranda of OpenStation made a copy of the code with every comment stripped out. He then gave Claude Code the same two tasks on both copies, three runs each. Every run passed lint, type checks and the existing tests. A separate AI judge scored each result without being told which copy it came from. On the small task, scores out of 19 came in at 18.3 with comments and 19.0 without. Tokens were 2.18M against 1.71M. On the bigger task, scores out of 23 were 21.7 and 22.0, with tokens at 9.06M and 9.12M. Aranda is careful about what this shows. He calls the score differences noise and names the small task's 22 percent token saving as the one real effect. He also shows what cherry-picking does. Compare one pair of runs on the bigger task and removing comments looks 30 percent more expensive. Compare another pair and it looks 31 percent cheaper. He says three runs per branch rules out a big effect and not a small one. My read: put a number on how small. On the bigger task the three comment-free runs used 7.01M, 9.70M and 10.64M tokens, a range of 3.63M. The difference between the two branches was 0.06M. So the thing being measured is about one sixtieth of the scatter the measurement produced. That is not a weak result. It is a result this design could not separate from noise. openstation.blog/2026/10/13/…