I was contacted by the CEO of @CommandCodeAI and I'm not really sure how I feel about it... I'm not a journalist, I'm just a regular user. I wanted to compare services on completely neutral benchmarks. I do not have a stake in the success or failure of any of these companies. However, I'm now being blamed for trying to release benchmarks because I believed that the community deserves real numbers on neutral grounds. I've acted in good faith to get to a solution and offered to correct ANY incorrect configuration I may have had and also re-run any tests with corrected configurations and post those. I've offered to run future tests in other agents (Pi), and also native agents (future tests). I offered to wait for backend issues to be patched. I've offered to install workaround plugins. That's more than I've done for any provider. I'm accused of being "outrageously wrong", my methodology being "bad faith", "publicly bashing" the company, and not showing empathy (disheartening, because I offered hours of my time to help when I had no obligation to). Their claim: the backend has safeguards "against memory leaks or memory overloading sessions," and those blocked my requests. We can all agree the harness may be inefficient, and if you believe that your guards are right, then let's leave it there. I use Pi day to day. I don't mind saying the harness is unoptimized, it's not even v1 yet. But don't say the benchmark is bad faith, don't say it's "still your issue". github.com/CommandCodeAI/com… Every provider ran the same harness, the same prompt, at the same time. Only one failed. If your service supports using API keys, expect others to use different clients. It's your feature. Disable the use of third party SDKs and clients if you think they're unoptimized and paint your service in a bad light. Else, it IS your responsibility to optimize for them. Especially a harness with 250k+ stars on github and the official harness for the model. After all, I'm testing the model, not the harness. So I'm done. Command Code won't be part of my future benchmarks. The sessions are public for anyone who wants to check my work, and the community is free to @ me and tell me that I've incorrectly done these benchmarks so I can fix them: github.com/nazzysar/commandc… (FYI, I sanitized out any sensitive info, you can use agents to just read through). My conscience is clear.

I want the end result to be that everyone wins, regardless of the provider we use.

I am thinking about also doing an experiment that tests each provider in their own respective harness to see how well optimized their harness is to their own inference setup. It will be near the end. But to be explicit this will become a test of harnesses and not API/model.

Same harness (DSH), Same reasoning effort (Max), Same prompt, different providers! Prompt: Build a single-file index.html (inline CSS and JS, no build step, no external libraries or network requests) containing an evolving ecosystem simulation of plants, prey, and predators on a 2D grid. It must auto-start on page load and run for N generations on its own. Each creature carries a gene vector (speed, vision, metabolism, reproduction threshold) or a tiny neural net that mutates on reproduction. Use a seeded PRNG for all randomness so the same seed always reproduces the same run, and keep the simulation logic separate from rendering so it can also run headless and faster than real time. The design is the main deliverable. Treat the simulation as a living specimen under glass, and make the controls part of the composition rather than a form beside it. Pick one distinctive visual direction suited to the subject (for example a naturalist’s field journal, a bioluminescent deep-sea observatory, or a scientific plate), with a deliberate palette, a considered type pairing, and one memorable signature element. Give each species its own color and glyph language. Put every variable (seed, mutation rate, food regrowth, generations, speed, catastrophe intervals) in an integrated control surface, such as a collapsible side drawer, a docked instrument panel, or controls anchored to the world itself. Use custom-styled sliders, dials, steppers, and toggles with live value readouts, units, and sensible ranges. Add a small sparkline next to each knob showing its effect, and give each control tooltips, keyboard support, and visible focus states. Changes should apply live with smooth feedback. Add a scheduled catastrophe system (drought, plague, and one of your choosing) that fires automatically at configurable intervals. Announce each event with a restrained, on-theme visual moment, such as a screen tint, a banner, or a timeline marker, and mark it on the charts. When the run ends, transition into a polished dashboard with population curves and trait-drift charts (mean and spread per gene per species), all drawn on canvas in the same visual language.

Also, in the interest of absolute transparency: OpenCode (@thdxr ) reached out to me personally to work through and fix some of errors and throughput issues they were facing that showed during the previous experiment. The result of that may or may not show through on the next experiments. However, as I did not give OpenCode a heads up on my initial experiment, it is only fair I continue the experiment as-is live with CommandCode (@CommandCodeAI ). However, if they do want to collaborate on any fixes that may arise during experiments, I am happy to help and also post before-afters. If i'm reached out quickly, those results may even show up as I continue multiple rounds :) I want every provider to win and the end result should always be what is best for consumers. So if everyone optimizes and fixes their platforms, EVERYONE wins.

Before I start posting the experiment results (whew, they take very long if you want to do it right! bare with me), I want to discuss some of the methodologies for the experiments i'm running so that I can be consistent across all tests. - I only test live and I only test 1 run. When doing real work, you do not re-run the same prompt twice to see what results you get. Every turn and interaction is "unique" in real workloads. - Model output quality is noisy and can be non-deterministic. I want to avoid subjectivity so I will stop commenting on what I personally think is better or not. That's for the eye of the beholder. - All stats on token count spans lead + subagents within a session as DSH is very agent team trigger friendly. - I will post any errors that I receive for every run. - This time, anything that is web based, I will try and get a demo of it up on github pages so that you can demo yourself. Renders and videos will stay here.

DeepSeek Harness team cooked. As i'm doing these experiments for you guys (and for myself!), i'm incredibly impressed by the agent teams and how the harness orchestrates tasks out. It's truly a joy to watch.

The people have spoken. CommandCode GOAT vs OpenCode Go vs DeepSeek API it is. I'm now going to try something different. MAX reasoning so we can test capabilities to their fullest potential. Tests are going to be slightly more complex. Follow and stay tuned!

Say it out loud for the people in the back. THERE IS NO MOAT. I've never seen democratization of technology across individuals AND companies happen this fast.

Yes! TTFT divergence is huge between DeepSeek API and OpenCode Go currently. Here's the stats: Round - DeepSeek API - OpenCode Go Round 1 - 2.2s - 56.3s Round 2 - 2.2s - 31.3s Round 3 - 2.5s - 31.5s Round 4 - 2.0s - 33.9s All stats taken from DeepSeek Harness.