The change in question came from a refactor. I have a separate “parity test” suite for any and all refactoring, and those have proven very good at preventing bad regressions, both for refactors involving code reorganization, code modifications, and code deletions.

There’s some overlap, but I’ve also been using new ideas for testing code not covered in the testing series, like getting agents to deliberate plant bugs at every layer and see how tests behave, mutation testing etc; which are new (to me) techniques I haven’t covered before.

Integration tests are great, but they don’t replace unit tests; they come later in the process, whereas bugs that can be caught by a good unit test should be. It allows for speedier iteration. As for the guidelines, I’ll try to make my Claude config public, including testing.

We’ll still need end to end tests, and frankly, we’re going to need all the forms of testing I’ve been talking about for a decade. But “unit tests are now slop” doesn’t have to be the foregone conclusion. Unit tests can be very useful even if a human will never again read them.

5 months and many experiments later, this was totally a case of “skill issue” on my part 😅 Like with code, agents don’t magically write great tests If your unit tests are slop (as mine were), that’s because you don’t have proper testing guidelines for your agents (as I didn’t)

Truly, she’s one of the vilest. Just astonishingly and unfailingly awful. She admits she gets a torrent of racist abuse, but engages in it herself. Despicable piece of shit. She and Indian_Bronson, both cut from the same cloth.

I don’t know if I’m the best person to answer this. Build new stuff. There’s still a world of software that’s yet to be built. Start with any tool you use, and see how you can make it better. Ideas are cheap to try out. The more you do, the better you get at picking better ideas.

lol, I refuse to pay for this app, so typos shall remain. It’s also how you know I’m rawdogging every tweet myself, rather than get AI to write it (which I learnt is a thing, esp with ultra long tweets some influencers and execs post)

Unfinished coding side projects are now way easier to complete than unfinished blog posts. I’m perfectly OK with AI writing my code, even the docs. But I flinch at the idea of AI writing my blog posts, which feels like an egregious violation of a fundamental ethnical contract.

Ideally the folks building software factories or harnesses should be evaluating how best to get agents to debug, because using Claude code out of the box isn’t going to cut it.

All super valid criticisms. I’m not a researcher or on a team at work that’s tasked with building better debugging primitives using evals, testing different mechanisms etc. I’m just an infrastructure engineer who has access to claude, and am chronicling how I use it to debug.

That should be covered by the data integrity axis. How data integrity is handled for stateful services which rely on replicated, partitioned data under different scenarios should answer most questions.

There are probably more patterns, but I need to go read more papers lol. Doing it will take time, hence the early 2027 (more like Feb or March) goal.

Lot of it holds up pretty well, actually. The main changes in the last ~7 years have been in: - move towards more flexible consistency - use of replicated logs for all manner of things - move away from strict consensus and synchronous replication - control plane design choices

I started this series on testing nearly a decade ago, but never published the final part copyconstruct.medium.com/tes… There’ve been so many great advances in the storage space in the ~7y I feel this is worth completing. Have a draft from years ago I hope I can wrap up in early 2027.

Ok, so now I guess I have to do this lol. Will try to get this done on Sunday, but if not, will have to wait until next weekend. Might even blog, explaining where some of these ideas came from (specific senior devs I worked with, conf talks of the past, oss projects, etc)

It has both open and closed source test suites. SQLite is a library with multiple layers; all but one of the layers has oss test suites, iirc. There’s plenty of great ideas in the open source test suites, covering a host of domains.

If that was your takeaway from that article, there’s no point engaging with you in good faith anymore. 👋

I’m literally that weirdo who wrote (all by hand) a 15K word blog post on: - why one cant treat models as oracles - how models gaslight you all the time - what it takes to build confidence in the model’s output copyconstruct.medium.com/how… So yeah sure, I believe in magical woo-woo

None of this is true IME. People who expect models to automatically do the right thing are bound to be disappointed. Give it good instructions, constraints, context and multiply verification by tenfold, and you get way better results than any human can achieve on their own.