For the last attempt, the traffic team at slack went back to strengthen run books (especially for very risky rollbacks), increasing observability on DNS (route53 logs for full visibility of dns requests, breakdown by resolvers). @rdelvira #QConLondon

When encountering issues with the rollout, the traffic team decided to rollbck DNSSEC, with confidence of having done it many times in testing. That wasn't taking into account the 24h cache DNS resolvers worldwide. @rdelvira #QConLondon

Using netlog capture to debug issues in the chromium engine used behind the Slack client. @rdelvira #QConLondon

The challenge on DNSSEC is that it needs to be applied per domain, cannot be split per subdomain or be progressively rolled out. The rollout was done per domain, with slack.com being the last one...causing 3 different outages @rdelvira #QConLondon

"We planned DNSSEC carefully, with the necessary changes and replicated most of our DNS use cases... And you'll see later why I said 'most'..." @rdelvira #QConLondon

Here we go, last talk of the day on the production track at #QConLondon, with @rdelvira and "an entertaining outage story" (his own words) when slack rolled out DNSSEC

"If we have proper visualisation and better metaphors, we set much better conditions for our operators to be comfortable in understanding and responding to variations in our systems." @yurynino #QConLondon

Visual metaphors as alternative solutions to hard to read bar charts. For traffic, using nodes and edges of variable size - and for latency using color coded gauges. @yurynino #QConLondon

When surveyed, only 55% of engineers answered correctly when looking at a standard line graph - but they all got it right when viewing it in a city metaphor @yurynino #QConLondon

Introducing visual metaphors: displaying data and metrics in real world shapes and analogies. @yurynino #QConLondon