Staff SRE @Google. Distributed systems / databases / reliability / correctness. Views my own. Retweet / like is not an endorsement. https://xitter.cf/t.co/UI9PS85hBx
San Francisco Bay Area
Joined July 2009
- Tweets3.4K
- Following528
- Followers2.3K
- Likes11.8K
Pinned Tweet
For quite some time I've been putting together a list of resources on testing distributed systems
asatarin.github.io/testing-d…
Andrey Satarin retweeted
We've been testing Raft implementations, and have found bugs in every implementation we've tested. Here's an in-depth report on one set of issues.
Andrey Satarin retweeted
PSA: The most important part in "We have rewritten our project from language/platform/library X to language/platform/library Y and got tremendous performance boosts" is "rewritten", for any mature X and Y. Doubly so if you dislike X and like Y.
Great recent find “Epic: The Musical”., based on The Odyssey.
Gods, monsters, magic, love, death, betrayal, revenge, drama. It has It all. **So quotable**.
music.youtube.com/playlist?l…
Andrey Satarin retweeted
Moore’s law still applies in the sense that the number of transistors in our processors increases exponentially.
What broke around 2005 is Dennard scaling. Our clock frequencies are no longer increasing. But you don’t want to buy a processor from 2005: it is going to be much slower.
Bandwidth and storage are increasing all around.
Our compute throughput is not stagnating. Don’t let anyone tell you otherwise.
Many current programmers remember the Moore’s law curve. That was the period from the 70s through the 90s where clock rates and memory doubled every year or so. But that doubling stopped in 2005. Memory continued to get denser for a while, but even that has stopped. The current way that we increase computing power is by doubling the number of GPUs in server farms every year or so. But that can’t continue much longer. The power requirements, the space requirements, the capital requirements, and the sheer mass of it all, will exceed feasibility within a few more doublings.
So don’t expect AIs to continue to improve exponentially. They’re headed for a brick wall, and that brick wall is not that far away.
Andrey Satarin retweeted
I designed Dropbox's storage system and modeled its durability. Durability numbers (11 9's etc) are meaningless because competent providers don't lose data because of disk failures, they lose data because of bugs and operator error.
Yes S3 has lost data. No it wasn't because some disks failed.
If you're building your own infrastructure you should heavily invest in release process and validation testing (link in reply). You're not going to do a better job than a major cloud provider though.
The best thing you can do for your own durability is to choose a competent provider and then ensure you don't accidentally delete or corrupt own data on it:
1. Ideally never mutate an object in S3, add a new version instead.
2. Never live-delete any data. Mark it for deletion and then use a lifecycle policy to clean it up after a week.
This way you have time to react to a bug in your own stack.
"Correctness at Feldera" talks about various correctness techniques, including:
- machine proof of the underlying DBSP algorithm
- differential testing of the implementation
feldera.com/blog/correctness…
More on:
- reusing corpora of tests from existing systems
- metamorphic tests with SQLancer
- verifying control plan with fault injection and fuzzing
Added to the list
asatarin.github.io/testing-d…
Andrey Satarin retweeted
Buckle up because we're crashing into the new year with my annual database retrospective: License change blowbacks! @databricks vs. @SnowflakeDB gangwar! @DuckDB shotgun weddings! Buying a college quarterback with database money for your new lover! cs.cmu.edu/~pavlo/blog/2025/…
MariaDB is improving their isolation level guarantees following the recent @jepsen_io report.
"Isolation level violation testing and debugging in MariaDB" post talks about how they specialize Jepsen tests for InnoDB, make it reproducible and debuggable.
mariadb.com/resources/blog/i…
Jepsen report on MySQL jepsen.io/analyses/mysql-8.0…
Post references Alexey Gotsman's talk on isolation levels, which I commented on before
xitter.cf/asatarin/status/176691…
Burnout is an occupational phenomenon from *chronic* stressors that have *not been successfully managed*.
Can cause heath issues (mental and physical), but it's not a medical condition. It should not be treated as a medical condition. There is no diagnosis or cure per se.
ALT WORLD HEALTH ORGANIZATION STATEMENT ON BURNOUT • Burnout is a syndrome conceptualized as resulting from chronic workplace stressors that have not been successfully managed. • It is characterized by three dimensions: • feelings of energy depletion or exhaustion • increased mental distance from one’s job, or feelings of negativism or cynicism related to one's job • reduced professional efficacy • Burnout is an occupational phenomenon. • It is not classified as a medical condition.
Another great video on burnout
"You Are Burned Out And Don't Even Know It"
youtu.be/jqONINYF17M
Signs of burnout is lack of:
- energy
- empathy
- fulfillment
Which leads to cynicism.
Checking out if the grass is greener or the sky is bluer on the other other site.
bsky.app/profile/asatarin.bs…
Andrey Satarin retweeted
Getting made fun of because you cover laptop webcam with a sticker? 😭
Here are materials from my talk about controlling ThinkPad X230 webcam LED over USB presented at POC by @POC_Crew 😎
Use these as a comeback 😁
Slides: docs.google.com/presentation…
Code: github.com/xairy/lights-out
Jepsen 15: What Even Are Transactions? by Kyle @aphyr Kingsbury
Talks about safety of MySQL, RavenDB, Datomic.
This is the first and only (so far) Jepsen talk I've seen in person. It was great, of course.
youtu.be/ecZp6cWhDjg
Andrey Satarin retweeted
The @ClickHouseDB paper is now out vldb.org/pvldb/vol17/p3731-s…
Lightening Fast Analytics for Everyone!