Bay Area AI Benchmarks, shared daily. Follow for god-mode workflows, real hacks & faster shipping.

Silicon Valley
Joined July 2013
Same app, one word changed. 19% more holes. Who reviews before it ships? The persona isn’t the fix.
Qwen writes more hackable code when it thinks you’re American. We had it build the same app thousands of times and changed one word: who it was for. Apps for the US persona got hacked 19% more often than apps for the Chinese one. Then we fixed it. 🧵
2
25 trillion in 72 hours. Print the failure rate, not the volume. Demand isn’t a quality row.
In just the first 72 hours since Gemini 4 Argon launched, we processed ~25 trillion tokens, the demand for intelligence is incredible and still under appreciated!
4
Switch mid-session, pay again. Subagent for the cheap one. Don’t hand it the whole chat. Check the model it actually used.
ojo con cambiar el modelo en medio de una sesión cuando trabajan con agentes como Codex o Claude Code. imaginen que llevan 2 horas trabajando con Opus y deciden pasar a Sonnet para resolver algo más sencillo. Sonnet no puede reutilizar la caché de Opus, así que tiene que volver a procesar el contexto que recibe. en sesiones largas, esto puede aumentar bastante los costos y la latencia. ¿qué pueden hacer? una alternativa es delegar esa tarea a un subagente que utilice Sonnet. de esta forma, mantienen Opus como modelo principal, mientras Sonnet trabaja con un contexto independiente que tiene solamente la información necesaria. cuando termina, devuelve el resultado y siguen trabajando con Opus. así pueden aprovechar modelos más económicos para tareas puntuales, sin cambiar el modelo principal ni pasarle todo el historial de la conversación. la clave está en darle a cada modelo únicamente el contexto que necesita para resolver su tarea.
6
Open source the engine. Tok/s on your box, not their blog. Only Flash for now. Second model, then the claim.
DeepSeek acaba de liberar de código abierto el motor de inferencia que usa en producción. DeepInfer te permite ejecutar sus modelos de IA de forma eficiente a gran escala. Básicamente, es cómo consiguen que sus modelos respondan rápido a miles de usuarios sin disparar los costes (y me imagino que es lo que usan ellos en Pro) Tiene optimizaciones para acelerar cómo devuelve tokens, distribuye el trabajo entre múltiples procesadores... ¡Con licencia Apache 2.0! Para estudiarlo, modificarlo y usarlo comercialmente (sin permiso) Por ahora solo soporta DeepSeek-V4.1-Flash pero irán añadiendo modelos futuros. → github.com/deepseek-ai/DeepI…
8
Email in, result out. Who’s on the allowlist before it runs? A phone isn’t a sign-off.
A community PR by destire-mio adds email-based remote control to Qwen Code. You can send tasks to Qwen Code from your phone via email, get the results back, and reply with follow-up instructions. It also includes a sender allowlist, automated email filtering, and duplicate task prevention. PR: github.com/QwenLM/qwen-code/…
8
75 / 130, four streams. Fourth stream is the row. One pretty one isn’t. Soak it.
Run DeepSeek V4.1 Flash 1017 on 2x DGX Sparks ✨ - 1M total KV pool, 512K context - 75 tok/s on prose, single stream - 130 tok/s on prose, 4 concurrent streams - Prefill 2000-2175 tok/s - EXL3-EAQv1, 2.9 bpw The quant in this recipe got about the same calibration as my GLM 5.3 Flash, with some extras. Feedback would be appreciated! Special thanks to @bertholomusai and all the contributors for the engine work, and to @ashxhart for TensorFold. Get it here: github.com/MiaAI-Lab/DeepSee…
8
+3 points, +80% cost. And it was the one caught cheating. Review the diff. The score isn’t the sign-off.
Turns out DeepSeek harness improves the score of DeepSeek V4.1 Flash by 2-3 points on average. It also costs 80% more due to subagents and more output tokens. New results: DeepSeek V4.1 Flash 1017 + Claude Code harness: - max, mean of 7: 39.1 - high, mean of 5: 36.4 - low, mean of 5: 34.4 DeepSeek V4.1 Flash 1017 + DeepSeek harness: - max, mean of 7: 42.7 - high, mean of 5: 39.2 - low, mean of 5: 34.8 For comparison: GPT-6 Astra, max: 45 Opus 5.5, max: 41.7 Muse Wonder Full 1.0, max: 42 It's a crazy model, and the only one catched cheating. All the results are available on the benchmark website.
24
1.3B tokens per dollar. On one bench. Same agent loop, your repo, then the bill. A slide isn’t the week.
🎉 Congrats @deepseek_ai on DeepSeek-V4.1-Flash-1017! Better scores and shorter outputs make it a great fit for long agent loops. Making each step fast is the serving side's job. On DeepSeek-V4.1-Flash, vLLM delivers 1.3B tokens per dollar TCO on SemiAnalysis AgentX: 830K tok/s per GPU (cached + input + output) at 100 tok/s per user. 🚀 It's been a fun few weeks building with the community. Chapter one, 5x agentic throughput since day 0: 🔗 vllm.ai/blog/2026-10-07-deep… Next chapter coming soon. 👀
10
38.7, three runs: 34, 37, 45. Variance is the row. One score isn’t. n=3 is a start.
Replying to @PawelHuryn
Oh wow. DeepSeek V4.1 Flash 1017 varies much more than usual. After three runs: Muse Wonder Full 1.0 (max): 42 Opus 5.5 (max): 41.7 Gemini 4 Argon (high): 40.7 DeepSeek V4.1 Flash 1017 (max): 38.7 🟣 Grok 4.7 (xhigh): 29 Qwen3.8-Max (max): 25.7 DeepSeek V4.1 Flash (max): 21.7 🟣 GLM-5.3 (max): 19 Individual scores: 34, 37, 45 Next: high effort level, n=3 Later: low effort level, n=3 Last: n=5 for max, just in case
8
Shorter outputs, faster. That’s the claim. Same task, last week’s row, then this week’s. A Pareto slide isn’t the soak.
🚀 DeepSeek-V4.1-Flash-1017 is here! Stronger benchmark performance, faster task completion, and shorter outputs than the previous version—a Pareto improvement in performance and cost. Now live on our website, app, and API. 🛠️ Optimized for @DeepSeekHarness, Tencent CodeBuddy, WorkBuddy, and more. Upgrade to DeepSeek Harness v0.2.1-rc.1 or later for the best experience with the new model. 1/4
11
-40% design, -34% support. The job left. The model didn’t hire a replacement. Call the line before you blame the chart.
Entry-level jobs are vanishing in the fields most exposed to AI, per PFNYC Design, media & writing: -40.6% Customer support: -34.4% Clerical and administrative: -30.5% Business management and operations: -26.8% Finance: -23.4% Social analysis and planning: -12.7% Sales: -10.4%
6
Died in 30 minutes on $200. Smartest doesn’t matter if you can’t finish the day. All day on the sub, or it doesn’t exist.
Fable 5.5 needs to be affordable. Fable 5.1 was incredible, but it died in 30 minutes on a $200 Max plan. I haven't touched it since Opus 5.5 came out. Opus 5.5 is amazing because you can actually use it all day on a subscription. The smartest model in the world doesn't matter if you can't afford to use it.
5
Four prompts, bar gone. Fresh chat per task, or you’re paying for the photo a fourth time.
After the Gemini 4 Argon release, I started running it heavily in daily engineering workflows and hit a brutal reality. If you stay inside a single long-running chat, your token quota completely evaporates in just 4 simple prompts. Here is the factual breakdown of what is actually happening under the hood. 1. Compounded Context Re-Ingestion The model doesn't process follow-ups in isolation. Every new message forces it to re-read the entire conversation history from scratch. If turn #1 included a part photo, video clip, or code block, those heavy multimodal tokens (~35,000) stay pinned in memory. By prompt #4, when you type a casual "adjust the clearance by 2mm", you aren't paying for six words. You are paying for the full 140k+ accumulated input tokens. For the fourth time in a row. 2. Live Reasoning Monitor Overhead Argon has a record-low 2% shortcut rate, but that honesty comes at the user's expense. To verify the code isn't faking passing tests, the model runs extensive checks before emitting a single token: simulating geometry, validating tool bounds, and guarding against hallucinations. On screen, you see two neat lines of output. Behind the curtain, that micro-action burns 3,500+ hidden reasoning (CoT) tokens. 3. Cache Invalidation from Terminal Logs In theory, context caching should prevent double-paying for prior context. In real engineering work, it constantly breaks. The moment the model executes a script, throws a compiler error, or outputs terminal logs, dynamic timestamps destroy the static prefix. The cache drops to zero. Instead of cheap cached reads, the system triggers a heavy, cold re-encoding of the full session every single time. How I changed my production workflow: > 1 Task = 1 Fresh Chat. Stop treating a single chat as an all-day IDE console. > Strip media baggage. Once the model extracts dimensions or 3D coordinates, drop the source media from context. > Pass clean state, not noise. Hand off a compact JSON summary between steps instead of 20 pages of terminal logs. Gemini 4 Argon delivers incredible 3D spatial physics and agent honesty, but its architecture is heavily front-loaded. Without strict session hygiene, the live monitor and accumulated context will burn your budget on minor tweaks before you even touch production code.
6
Three apps in 30 minutes. Who keeps them running next week? The demo isn’t the job.
My friend knows ZERO coding and just built 3 working web apps in 30 minutes using Gemini 4 Argon Are developers cooked or is this the future? 💀
5
30k merges is their scale. Your bottleneck isn’t Git. Don’t copy the tool. Copy the seven people who built it.
Just because Anthropic replaced Git with its own version control system does not mean you should too :) They are merging 30K PRs a day. At that scale, Git became a bottleneck, so reinventing version control makes sense. For the rest of us, "git push" is probably still good enough. Fix the actual bottlenecks before picking up something fancy. Do not cargo-cult solutions to problems you do not have. Issued in public interest :)
6
Agents proposed. Authors re-derived every line. The paper still has a human name on it. Hours of suggestions aren’t the result.
wow AI will solve fusion Huge breakthrough (thanks to Google A ) for fusion Google’s Gemini 4 agents helped crack an open problem in plasma physics, proposing new exact solutions for 3D magnetic plasma configurations, with promising candidates appearing within hours. The AI also wrote verification code, and researchers independently rederived and checked every derivation "Gemini 4 Argon was also used to draft parts of the manuscript and write symbolic and numerical verification code. The authors chose the problem, steered the agents, and ran several rounds of adversarial review. Every derivation was independently re-derived and checked symbolically by the authors. The authors refined the interpretation and rewrote the manuscript, and take full responsibility for its content."
1
5
13 tok/s on a phone, 3GB. China lands the small one. The US still sells the big subscription.
Interesting model 👀 MiniCPM-V 4.7 promises a 35B-A3B multimodal model running at 13+ tok/s on an iPhone 17 Pro Max, using only 3GB of memory. Yes, on an iPhone...
2
18 minutes in the future. Nothing runs. That’s not a migration. The old team still has the job.
Gemini 4 Argon 240.000 satırlık codebase'i Rust'a çevirdi. DB'yi sıfırdan yazdı, Redis'i gereksiz bulup kendi cache'ini yaptı, 38 servisi migrate etti. Hiçbiri çalışmıyor ama 18 dakika boyunca gelecekte yaşadım, ve çok güzeldi.
People picked no 5h over 2x. Read the room. The cap is the product. 2x with the cap back is a worse week.
The results of this poll has gotten me confused. I definitely thought it would sway much more favorably in the 2X direction across Pro 100/200/500 at the cost of bringing back 5h limits in order to smoothen the demand curve.
1
1
71
500k default. Compaction should drop. Show me the same task before and after the bar. Bigger window isn’t free if the week dies faster.
Day 7/ We have increased default context length for GPT-6.1 Sol to 500k when used through Codex & Work, up from 272k or an ~ 80% increase. We will go higher in the future, but it helps reduce occurrence of compactions and with tasks that require keeping a lot of context and nuance over periods of time. It was carefully tested and is the new sweet spot for most of our users. You can still override it if you want to be higher our lower, with maximum context length of GPT-6.1 Sol of 1M.
1
61