Sergey | AI Builds & Tests (@SergeForge) header
Sergey | AI Builds & Tests (@SergeForge) profile picture

I make AI agents do real work, test what actually works, and show what it costs. Building tools along the way. Claude Code, Codex & more.

New York, USA
Joined March 2011
Photo by Sergey | AI Builds & Tests (@SergeForge)Photo by Sergey | AI Builds & Tests (@SergeForge)Photo by Sergey | AI Builds & Tests (@SergeForge)Photo by Sergey | AI Builds & Tests (@SergeForge)Photo by Sergey | AI Builds & Tests (@SergeForge)Photo by Sergey | AI Builds & Tests (@SergeForge)Photo by Sergey | AI Builds & Tests (@SergeForge)Photo by Sergey | AI Builds & Tests (@SergeForge)Photo by Sergey | AI Builds & Tests (@SergeForge)Photo by Sergey | AI Builds & Tests (@SergeForge)
one lazy lidar scan of my junk-filled basement + one prompt to claude fable 5.2 it found the real room under the mess (15.7 × 21.5 ft), designed 4 shelf setups and priced them: $420 to $5,000 and caught that a 77 in rack would've had under 4 in to the ceiling at the window end. i'd have found that out halfway through assembly
72
153K people on reddit watched me get roasted for this. top comment: "buddy couldn't figure out how to clean a garage without ai lol" fair. here's what claude fable 5.2 did with one lazy lidar scan of my basement, junk and all: found the real room under the mess: 15.7 × 21.5 ft, ceiling dropping from 85 to 81 in designed 4 shelf setups, 3 views each, and priced them: sam's club plastic: $420–490 steel racks: $1,100–1,350 wire on wheels: $1,330–1,570 plastic on wheels: $4,000–5,000, for people whose junk is worth more than mine and caught a 77 in rack that would've had under 4 in to the ceiling one prompt
1
76
anthropic says one agent with skills beats a team of subagents. i tested it in claude code on 2 real builds, 35 runs, sonnet 5.5 for every agent same quality both times. on the harder build every site passed all 40 hidden tests, and blind design scores tied the team cost 38% more and took twice as long: $1.31 and 29 min vs $0.81 and 15 min per site on the first build its 2 failures came from a handoff: the tester called a warning a false alarm, the orchestrator sent the fix anyway, and it broke keyboard booking
Anthropic is now recommending that companies REMOVE unnecessary subagents from their AI systems. This is a pretty significant shift in agent architecture, with their team finding that a single agent equipped with skills can frequently outperform complex multi-agent systems, and one customer seeing roughly 2x faster execution. The emerging rules of thumb: → Skills: Teach the agent HOW to do something, including procedures, expertise, and instructions. → Tools / MCP: Give the agent the ability to DO things through APIs, actions, and live data. → Subagents: Use for parallel work, isolated context, or tasks a single agent struggles with. The interesting part is progressive disclosure. Instead of spawning specialized agents or stuffing everything into a giant prompt, one agent discovers and loads skills and tools as needed. Their advice is to fold existing subagents into the main loop, turn their instructions into skills, expose their actions as tools, and run evals to see if performance holds up. As models have gotten smarter, improving agent architecture is becoming less about what you can add and more about what you can delete. claude.com/resources/article…
5
3
99
setup: claude code 2.1.319, sonnet 5.5 at medium effort for every agent, fresh container per run, hidden tests the agents never saw, costs at api list prices. the team was designed blind, and the single agent got the exact same role prompts and helper scripts as skills build 2: 7 runs each, 40 hidden tests, design scored blind by separate claude sessions from shuffled screenshots. the cost and time gaps hold in all 7 pairs. one test had a bug i fixed after (one site per side, no change to the result) 11 of 14 sites showed every slot struck through until you pick a technician. no reviewer on either side caught it
1
35
your coding agent can pass the tests by editing the tests so i don't trust "all tests passed" anymore. i ran agents against tests they never get to see, and in one batch 12 out of 12 said done. 3 had actually passed before i accept anything i run this: git diff HEAD -- tests/ if a test got skipped, an assert got loosened, or an expected value now matches the bug, that's not a fix the thing that worked best for me: keep the real tests away from the agent. i copy them in after it's done. it can't game a test that wasn't there
5
1
47
can claude opus 5.5 build smarter traffic lights? it built a pretend city full of cars and tested 2 kinds of lights: normal ones on a timer, and smart ones that react to waiting cars quiet roads: smart lights were a bit faster rush hour: smart lights made trips 6.6% longer. every single test left side normal, right side smart 👇
1
63
just noticed xai's grok build can pick up your codex session was working in codex, opened grok build for something else, and it offered to resume where codex left off (ctrl+u, or /resume-codex <session-id>) → reads the codex transcript to find where it stopped → checks the machine before touching anything → keeps going on grok 4.7 (high) tried it on a real task. it found the spot and kept going. handy when codex hits its limit
1
1
43
this holds up in claude code. i tested it at 380k tokens same 4-step job, sonnet 5.5, 3 runs each: compact once: $2.98 never compact: $3.44 new chat every step: $3.42 same hidden tests passed every time last week on a small session, compacting lost. size decides
New NVIDIA paper finds that AI agents barely lose accuracy when they keep contexts short by compacting their history often. Each new token costs more to generate as the context grows. Serving open models at a 256K context window instead of 16K cost 3.6x to 7x more per generated token. They had agents stop at a set limit, write a short handoff note, and restart with only the task and that note. For long documents, the agent searched files with Bash tools instead of reading everything at once. On 6 long-horizon benchmarks, at most 5 of 22 compaction settings per model lost more than 2 points versus full history. Compacting after every action kept Claude Opus 4.5 at 70.2% on SWE-bench Verified, versus 70.0% with full history. Before paying for long context, give agents a structured handoff note and file search tools, then check if short contexts match your accuracy.
1
1
48
setup: claude read 950kb of the codebase first (~380k tokens), then built a 4-step feature graded by hidden tests. compacting once right after the read made the coding steps 25% cheaper weird bonus: the "never compact" runs dropped to ~140k on their own when coding started. claude code clears old file reads by itself effort high, api list prices from token logs
26
GO ONE STEP LOWER. OPUS 5.5 ON LOW PASSED THE SAME HIDDEN TESTS AS MEDIUM claude --effort low → 8-file feature: $0.73 vs $1.82 → bug fix: $0.29 vs $0.51 → xhigh on a 3d scene: $6.74, never finished on real code, medium is already more thinking than most jobs need
Announcing the next version of Artificial Analysis API Performance Benchmarking: Time per Task is now our headline performance metric, measuring how long a model takes to complete a task rather than how fast it streams The update applies to the ~1,000 serverless endpoints we benchmark multiple times a day for speed, latency and caching. Key changes: ➤ Time per Task is now the headline metric to compare models: as workloads become more agentic, the time to complete a task matters more than speed per token. A model that is fast per token but verbose can take longer than a slower model that uses fewer tokens. Time per Task divides a model's output tokens per Intelligence Index task by its End-to-End Output Speed. ➤ End-to-End Output Speed covers the full request: speed is now measured from sending the request to receiving the final token. That includes the wait before the first token (prefill, and reasoning for models that hide it) as well as generation. Output tokens now come from each provider's API-reported counts instead of the o200k_base tokenizer, and we monitor for discrepancies in reported counts. ➤ Agentic workloads are now part of every measurement: we benchmark short, document-length and long-context prompts, plus multi-turn sessions across coding and knowledge work. Headline figures are the average across these workloads over the past 5 days. ➤ API default effort is now tracked alongside highest effort: most developers run models at the effort level each lab sets as its API default, so we now benchmark key models at that setting as well as at their highest effort. Effort names are not comparable across labs, so we use each lab's own default rather than one shared setting. Key takeaways: ➤ OpenAI models make up most of the Intelligence vs. Time per Task Pareto frontier despite slower output speeds: across effort levels, GPT-6 Astra and GPT-6.1 Sol hold the frontier from ~45 to ~55 on the Intelligence Index. At max effort, GPT-6 Astra runs at 51 tokens/s but finishes a task in ~17 minutes, against ~26 for Claude Opus 5.5 (max, 111 tokens/s) and ~29 minutes for Gemini 4 Carbon (high, 71 tokens/s), because it uses ~50k output tokens per task against ~120k to ~170k. ➤ API default effort cuts Time per Task by ~3 to 5x and Cost per Task by ~2 to 4x for GPT-6 Astra, GPT-6.1 Sol and Claude Opus 5.5: all three default to medium effort, which scores ~5 to 11 points lower on the Intelligence Index in exchange. Claude Opus 5.5 falls from ~26 minutes and $10.04 per task at max to ~6.8 minutes and $2.45 at medium effort. ➤ Endpoints that optimize for decode speed alone see the largest drop in End-to-End Output Speed: high decode speeds can come at the cost of latency, for example by queuing requests to run at lower batch sizes, or by holding output and sending it in a single burst. Both lengthen the wait before the first token, which End-to-End Output Speed now counts.
2
69
how i tested: same prompt, fresh container per run, graded by hidden tests the agent never saw. costs at api list prices from token logs. transcripts confirm every run ran at the effort i set
14
"track tokens per task" is the right call. i did, and the fix most people reach for made it worse same 4-step job in claude code: one long session: $1.12 /compact between steps: $1.43 new chat every step: $1.42 same tests passed. on a normal-size job, keep the session going
CLAUDE CODE USERS ARE BURNING TOKENS 3.8X FASTER....SAME PLAN, SAME PRICE Turns out the model might not be the only problem.... It's the subagents. Lessons from one dev's token analysis: - Track tokens per task, not just your bill - Subagents re-read the same files again and again - More subagents means more steps, not better results - Past 350K context, agents keep working after they're done - Start a fresh session around 300K tokens More agents doesn't mean more work done. 👀
7
1
3
247
why: earlier context is re-read from cache at ~10% of the normal price. /compact and fresh chats throw that away and the model re-reads your files at full price restarting around 300k probably still makes sense, my job never got that big. sonnet 5.5, effort high, 3 runs per method, hidden tests
19
opus 5.5 wins the hard benchmarks. fair. but do you really need it to build a plumber's booking website? i tested 4 models, same prompt, 3 runs each: haiku: 6¢ — 32/33 tests sol: 17¢ — 32/33 sonnet: 35¢ — 33/33 opus: $1.74 — 33/33 29x cost gap. all 4 websites side by side 👇
Super interesting Benchmark. GDPx measures how well AI agents handle 206 challenging tasks across 11 economically valuable fields, including chip design, software engineering, finance, and biology. But here's the interesting part: Opus 5.5 wins with 12%, closely followed by Gemini 4 Carbon with 10.9%. But while performing just 1.1% worse, Gemini 4 is about 4x cheaper than Opus 5.5. So cost-performance wise Gemini 4 wins by a lot.
1
42
Sergey | AI Builds & Tests retweeted
Opus 5.5 in October btw:
97
260
68
4,387
355,158
opus 5.5 built this ship for $6.94. gpt-6.1 sol changed the sails for $0.20 one edit: "make the sails triangular, change nothing else" sol did it in 2 minutes. hull, ocean, light and camera untouched building is the expensive part. changing it doesn't have to be
2
2
81
all 18 edits worked. each model editing its own ship: terra $0.02, sol $0.13, sonnet 5.5 $0.42, opus 5.5 $1.40 the reverse costs more: opus editing sol's ship was $1.95 all on high effort, 3 runs each, same one-line prompt
1
25
tested exactly this. same prompt, both on high, 3 runs each: opus 5.5 $6.94, sol $0.53 opus does look better. the light, the haze, the detail. but sol gets you most of the way for 13x less, and opus hit my 40 min cap in 2 of 3 runs better, yes. 13x better, not on this one
Note that 5.5 on high is a lot better than GPT 6.1 Sol on high, so though the cost gap is a lot smaller when you consider subsidization, you're getting a better model. If you compare a lower reasoning effort on Opus to a higher one on Sol to be more fair, the Claude advantage widens. Surprisingly competitive, tho
1
88
gpt-6.1 sol built this galleon for 53 cents. opus 5.5 cost $6.94 same prompt, 4 models, all on high effort, 3 runs each: terra: 1m 25s, $0.07 sol: 16 min, $0.53 sonnet 5.5: 40 min*, $4.97 opus 5.5: 37 min, $6.94 *sonnet hit my 40 min cap in all 3 runs, opus in 2 of 3 opus has the best light. sol is 13x cheaper and close enough that it's the one i'd pick
1
1
149
setup: every model on high effort, and the logs confirm every turn ran at high. fresh container per run, one prompt pointing at a spec file, costs at api list prices from token logs. each tile is that model's median-cost run
23