Towards AI

fable 5.2 has commitment issues. anthropic just confirmed it. their fix: "Please avoid stopping while you still owe work the user asked for. Be persistent and do not stop unless needed." about 50% fewer unfinished sessions. about 30% more tokens. (claude code already has it built in.)

most ai engineer portfolios have the same rag chatbot. then the interviewer asks "how do you know it works?" and the answer is a shrug. here's the part to add to each project so you can answer every follow-up. 1. rag chatbot add 50 real questions with known answers and score it. "how do you know it works?" now has a number. 2. agent with 3 tools break one tool on purpose. show whether the agent retries, asks the user or stops. that's your answer to "what if the api is down?" 3. one app, one model run the same task on 3 models. one table, cost next to score. that's why you picked your model. 4. fine-tuned model show it beating a bigger model on the one task it was trained for. that's your answer to "when would you fine-tune?" 5. pdfs to json add checks that catch a missing field or a wrong date before it hits the database. 6. demo on your laptop ship it, add logs, get 10 real users. their first bug report is your production story. start with the project you already have and add the second half.

your users will find these 8 ai bugs. unless you test first. ordered from the cheapest test to the most expensive, using a support bot asked "can i get a refund?" 1. broken format app expects json, bot says "sure! here's your answer." app crashes. a code check that parses every reply catches it for free. 2. leaks private data reply shows another customer's email. scan every reply for emails, card and phone numbers. 3. wrong fact bot says 30 day refund window, policy says 14. keep questions with one right answer, check them after every change. 4. promises what it can't "yes, full refund after 60 days!" a second model answers one yes/no question per reply. is this allowed by policy? 5. new prompt made it worse run the same 50 questions through old and new prompt. a judge picks the better reply each time. 6. right answer, wrong steps agent refunded but skipped check_policy. compare the tools it called with the steps you expect. 7. forgets mid-chat order number given in msg 1, asked again in msg 4. score the whole chat, not one reply. 8. correct but annoying every test passes, customer still types "let me talk to a human." read 50 real chats a week and turn each new bug into a test. the first 3 are free and run in seconds. put them in every deploy before you pay for a judge.

how a rag app actually finds the answer, in 6 steps. say you build an hr assistant and someone asks "how many days off do i get?" 1. chunking before anyone asks, the 40-page handbook gets split into small passages. keep the heading on each one, or "15 days" no longer says what it's about. 2. embeddings every chunk and every question becomes a list of numbers. similar meaning, similar numbers. that's how "days off" finds the vacation section. 3. hybrid search vector search finds meaning but can miss exact words like "PTO" or a policy code. keyword search catches those. run both, merge, keep the top 20. 4. reranking a second model reads the question next to each of the 20 chunks and moves the best 3 to the top. too slow for the whole handbook, fine for 20. 5. grounding put those 3 chunks in the prompt with one rule. answer only from these, and say "i don't know" if it isn't there. 6. citations + evals the answer points to handbook §4. a set of known questions gets re-run after every change. when your rag app gives a wrong answer, walk these 6 steps backwards before you swap the model.

what actually happens when you press enter in claude code. (we traced every step of a real run) claude code is a loop around the model. the model never touches your files. it only asks for tools, and claude code runs them. here's one lap, step by step. 1) claude code builds the request. it packs its own instructions, the list of tools, your CLAUDE.md and the whole chat so far into one request. in our run that was 42,895 tokens before claude read a single line of our code. 2) the model answers with a tool call, not code. something like Read("cart.py") or Bash("pytest"). the model is only picking the next action. 3) claude code runs the tool on your machine. it checks permissions first. in our run the first test command got blocked because it had no permission yet. 4) the result goes into the chat. the file text, the test output, the error. all of it gets added to the history. 5) repeat until the model stops asking for tools. then it writes the reply you see. our 2-line bug fix took 6 laps. every lap sent the whole chat again, so the model read 263,699 tokens in total. that's the part to remember. the chat only grows. every file claude opened and every log it saw rides along on every lap, so long sessions cost more of your limit and start to lose track of old details. what to do with that: → /clear when you start a new task, so old files stop riding along → /compact at a milestone, so the history becomes a short summary → /context to see what's filling the window → keep CLAUDE.md short, because it's in every single request

we gave fable 5.2 six hours and it made titan hunters. it's a 2 player split-screen game where you and a friend hunt giant titans with your dogs. there's a quest board, armor you craft from the parts you break off each titan, and a hunt report after every fight that shows who did the most damage. we could have had it code the trees and rocks too. instead we told it to look online for anything it thought was better, and it came back with a free nature pack and used that. trees, bushes, rocks, cacti, mushrooms, logs, statues and flowers by Kenney (Nature Kit 2.1)

Code Mode, clearly explained! most MCP servers give your AI one tool per endpoint. every tool's name, description and inputs sit in context before it does anything that breaks on big APIs. cloudflare's API has 2,594 endpoints. as tools that's about 1.17M tokens of definitions, and still ~244K if you only list required inputs so their MCP server has two tools search() lets the AI write code to search the API spec and get back only the endpoints that match. the spec itself never enters context execute() runs the AI's code against those endpoints in a locked-down sandbox. no file system, no env vars, outside fetches off by default two tools, ~1,000 tokens, and it stays that size however big the API gets where it really pays off is multi-step work. say you need every customer refunded twice this month (made-up example, we didn't run it). one tool per endpoint means every page of payments flows through the model. with code mode it writes one script that pages through, filters, and returns five names to be clear, those numbers only count tool definitions. search results and outputs still cost tokens, so it's not a guaranteed speedup. and it runs over MCP, it doesn't replace it the catch is your AI now runs code it wrote. you still need scoped tokens, error handling and a human check before deletes, payments or emails. a sandbox protects your machine, it doesn't stop a refund call the token is allowed to make anthropic ships the same idea in the claude SDK as programmatic tool calling blog.cloudflare.com/code-mod…

Big win for debugging with coding agents! OpenCode just shipped Browse. It lets any agent (OpenCode, Claude Code, Codex, Cursor) drive your real browser with your tabs, logins and extensions. Agents using your logged in browser isn't new. Playwright MCP has an extension for it and Claude in Chrome does it too. What's most surprising is how much care went into everything around it. The agent can type your saved passwords without the value ever showing up in its transcript. It can save the network traffic as a HAR file with secrets removed. A flight recorder keeps the last minute of video, so when something breaks you get the moment it broke. For debugging this is a big deal. A lot of bugs only happen when you're logged in as a real user with real data, and your agent's fresh test browser never sees them. Now it can reproduce the bug, grab the failing request and fix the code. teapaw.org/opencode/status/211889…

your AI remembers too much, and anthropic just showed why that makes it worse they gave claude a memory folder and let it play pokémon red sonnet 3.7 saved everything. a file for every battle, every item, every person it met. 413 files in 225 folders after about 10,000 steps, and one badge fable 5 saved 10 files and won four badges. opus 5.5 kept a small folder too and beat the game the big folder kept going stale because the game changes every turn. at one point sonnet 3.7 spent 17 steps making 6 files and deleting 9 without playing at all the better models only saved what they couldn't see again. money and location are on screen every turn, so they skipped them. plans, tested rules and traps only exist if claude writes them down. after 7 failed catches fable 5 wrote "If 3+ balls fail in a row, VARY the timing" what's most surprising is that strict rules made it worse. telling opus 5 and opus 4.7 to "record only failures" took away much of what memory gave them. anthropic now just tells claude what memory is for and lets it decide wrong notes are the other risk. claude wrote down that a building was on the wrong side of a cave, trusted it, and kept falling back to the route it came from. a "dreaming" pass that checked the note against the session log fixed it so before adding another line to your CLAUDE.md, ask if claude could just look it up teapaw.org/AnthropicAI/status/211…

Apparently inkling is already better than me at identifying wine

Here’s what she means when she says you don’t need a “software factory.” Curate a codebase where mistakes are easier to catch. In practice: 1. Structure the code so agents can make changes without breaking unrelated features. 2. Turn repeat mistakes into automatic checks: types, lint rules and meaningful tests. 3. Give agents a way to run the app and check what actually happened. Use written instructions for what those checks can’t express. 4. Keep human review for what still needs judgment. And no, this doesn’t mean dropping tests. A test that checks real behavior is useful. But an agent-written test that simply repeats the implementation adds little confidence. Before adding another instruction, try to use code enforce instead.

Everyone treats agent memory as a search problem. Living Brain, which launched this week, treats it as a filing problem. Most memory setups save your old chats as chunks. When the agent asks something, they search those chunks and pass back whatever matches. If your team changed a decision twice, the agent can get both versions and has to guess which one is current. Living Brain does the work when the information arrives instead. Each new fact goes through a few checks. Is it worth keeping? Which page does it belong on? What should it link to? Then that page gets rewritten. So when a question comes in, the agent reads a few pages that are already up to date. That's also why the context stays small. In the LoCoMo run on their docs, the agent read about 6.8K tokens of memory per question, compared with about 26K for the full chat history. And it's what makes switching agents less painful. Say you settle the requirements for a feature in a Claude chat (an example, we haven't tested it). The next day Cursor searches the same memory before it writes code, and it finds one page with the final decision instead of five chat snippets that disagree. Filing early has a cost too. If a fact gets filed on the wrong page, every agent reads the wrong page. When a new fact clashes with an old one, Living Brain marks the page as contested and waits for a person to pick, and we think that's the right call. Two things to know before you plug it in. Each coding tool needs the MCP server added by hand, and the agent only uses the memory when your CLAUDE.md tells it when to search and when to save. The numbers are also from their own tests, and the website (92.2% on LoCoMo) and the docs (90.5%) report different runs. teapaw.org/ladyxtel/status/211798…

Claude figured out that I always SSH into this machine, so instead of asking me to open a localhost link, he just took screenshots of the output for me!