Third infrastructure incident we've written up this August, after GitHub & Monzo. This time, how Namecheap's outage took down sites it wasn't even hosting, because DNS shared infrastructure with hosting and the control panel. statuscake.com/blog/namechea…

Monzo's "Stand-in" activated during their outage on Wednesday: a fully separate backup bank on GCP (primary runs on AWS) running 18 services vs. primary 3k, and switched manually by their engineering team rather than failover. It did roughly what it was built to do; though some customers reported transfers and balances not behaving as expected while stand-up was running. More in our blog: statuscake.com/blog/monzos-s…

Worth reading GitHub's RCA in full. The part we found most interesting is that their fix for the original failure moved traffic to a second region, where a separate, unrelated retry bug then spiked one service's traffic from 7–9K requests/second to 70–100K. We wrote up how it happened: statuscake.com/blog/what-bro…

There's always room for another 15% spent hitting refresh on GitHub Actions' own status page, which still says everything's fine.

I'd start by comparing the execution paths, not just the request types. Reads may be cached or served from a replica. Writes often have to hit the primary & wait on the commit.

Every dashboard can still say green while the customer experience is already on fire. Your customers don't experience your architecture component by component, they experience the whole request path.

finding out your monitoring, support tool, & status page is hosted in us-east-1 too?!* *we're not, I hasten to add, for exactly that reason.

At 3am, mid-incident, staring at a function you apparently wrote, not recognising a word of it. That's when you find out how much you care about clean code. 💀

It's not just infrastructure, it's teams, process & time too. Increasingly, there's no human in the review or merge loop at all. Even then the bottleneck hasn't disappeared, it's just moved downstream - to whichever poor soul is on call when the unreviewed change breaks something.

System design, but push it past your own service. Most production incidents aren't caused by bad code, but by perfectly reasonable components (often owned by different teams) interacting in ways nobody designed for.

An alert tells you where the problem became visible, not where it started. Confusing the two is how a DNS issue ends in a Kubernetes restart.

Secret option E: 30 seconds hoping it fixes itself, resist the urge to restart it for luck, then actually check what broke and what changed. A) gets updated the next morning as an apology to your future self.

The bit nobody's mentioned yet. It's not just a DNS curiosity, it's an operational blind spot. Apex and www can sit behind entirely different infrastructure. One being healthy tells you nothing about the other. If users can reach both, monitor both, or verify the redirect actually holds rather than assuming it does.

Both can be true at once. It can be a perfectly legitimate bug fix from the maintainer’s perspective, while still being a breaking change from the user’s perspective if production systems have come to depend on that behaviour. Intent absolutely matters, but so does impact.

I'm sorry to hear that’s been your experience. If you’re happy to DM me your email address on your account, I’ll take a look at the tests & arrange a call so we can go through what’s happening with you - JB

GitHub is having a rough afternoon. 🦄 If your builds, deploys or repos are looking a little unhappy, you’re not alone. Sending some DevOps hugs to the @github team while they work through it. 💚 Status: githubstatus.com

Want to explore StatusCake? 🟢 Start monitoring → app.statuscake.com/Try/ 💻 Developer docs → developers.statuscake.com 🐙 GitHub → github.com/StatusCakeDev 💼 LinkedIn → linkedin.com/company/statusc…

Monitor what matters. Communicate what’s happening. Website, API and infrastructure monitoring for engineering teams. Uptime · SSL · Domains · Page Speed · Servers Monitoring since 2012. statuscake.com