Deepak Kumar retweeted
The evergreen resource - An Introduction to Distributed Systems by Kyle Kingsbury
github.com/aphyr/distsys-cla…
For an impromptu discussion, the Google File System (GFS) came into the picture. And from there, I remembered this MIT course 6.824 Distributed Systems, where there is a lecture on it.
Many folks have covered GFS in depth, but this course also provides an in-depth view of Distributed Systems in general. Recommend this if you are interested in this domain.
nil.csail.mit.edu/6.824/2021…
(Note: the site is HTTP though)
Deepak Kumar retweeted
🐘 Introducing Postgres Internals in 3D.
🌟 pgrun.dev/pi
Stop reading boring diagrams. Start exploring PostgreSQL.
An interactive 3D map to understand how Postgres actually works.
100% free. Built for developers.
xitter.cf/alxshp/status/21100494…
Introducing PGSync 🐘
Real-time PostgreSQL replication. FREE. OSS
Built in Rust. 🦀
Postgres 16 / 17 / 18.
No extensions. No triggers. No complex setup.
→ snapshot
→ replicate
→ stream changes
→ track WAL
→ recover from crashes
→ resume automatically
→ keep databases in sync
~15 MB RAM during streaming.
Built for developers. Built for AI agents.
Open source.
xitter.cf/alxshp/status/21078790…
Deepak Kumar retweeted
banger read on infra and workflows side from stripe's blog
stripe.dev/blog/run-recurrin…
Deepak Kumar retweeted
The 194th issue of the Polymathic Engineer is out.
This week, we discuss how to build a cache:
- The API contract
- Memory alone is not enough
- Building an LRU cache
- When fresher data isn’t better: the LFU cache
- Implementation
Read it here:
newsletter.francofernando.co…
Deepak Kumar retweeted
went through this blog as i was curious how Talim used Postgres exactly
and man very good articulation Talim @talimbuilds
Deepak Kumar retweeted
Replying to @VaibhavSisinty
The project is real and open source.
Repo: github.com/GLinnik21/plx-nat…
Site: plxnative.com/
Deepak Kumar retweeted
Very underrated tip from @karpathy.
Just tried ASD-STE100 rule for model responses, and I found it to be surprisingly good at making explanations clearer. It can be bit too strict, so I agree that "80% of the way to ASD-STE100" feels like a better default.
A prompt you can try with your agent:
"Use ASD-STE100 as a guide for your responses. Write short sentences, use active voice, and keep terminology consistent. Relax the vocabulary rules when they make explanations awkward. Preserve technical precision and uncertainty."
@0xpili_ also made a nice skill for it:
github.com/0xpili/simplified…
We'll be spending a lot more time trying to understand the outputs of language models. A few thoughts, tips & tricks:
Writing. Something I've had success with: Ask your LLM to explain something in ASD-STE100, it's a controlled language specification originally developed for aerospace maintenance documentation. LLMs well-versed in this language and it comes with heavy constraints on clean writing style that I often find a lot more readable. Sometimes I've tried to soften it a bit e.g. ask for "80% of the way to ASD-STE100" because the spec is quite stringent. But even better:
Diagrams / images. Instead of writing, ask your LLM to create a diagram. These can be a lot easier to process, parse, and understand. But even better:
Web pages. Ask for output "in HTML" to get a beautiful, interactive webpage. LLMs are getting really good at frontend and can create beautiful experiences, animations, etc. But even better:
Explainer videos. The output format I am most bullish on is fully custom / bespoke explainer videos generated on any arbitrary topic. Experiment with things like "Create a 3b1b style video explainer on X. Use my ElevenLabs API key for audio narration". (you'd need an API key for the latter or you can ask your LLM to find you decent free alternatives that use your local compute). This is actually starting to work!
In summary:
- As LLMs get better, they will do more and more of the legwork autonomously, and a lot more of our work will rise up the abstractions into oversight and understanding.
- Luckily, LLMs can help here too because as intelligence and code are increasingly abundant, you can ask for large, custom, discardable software artifacts (e.g. web apps, video explainers) that would have never made sense to create before. Push the boundaries here and you'll be surprised.
Deepak Kumar retweeted
Found the best resource to learn Harness Engineering. 😭
walkinglabs.github.io/learn-…
Deepak Kumar retweeted
Wow: @petermattis has quite the history in shipping: co-created GIMP, built Gmail’s original backend, shipped Google’s Colossus, and is the cofounder and CTO at Cockroach Labs. We talked distributed DBs:
Timestamps:
00:00 Intro
02:42 Peter’s path into tech
04:00 Building GIMP
09:30 Working on Gmail at Google
14:51 Google’s infra: google3, build files, Bazel, and Colossus
21:30 Distributed storage bottlenecks
23:59 Latency, throughput, and availability
30:04 Contributing to libraries
41:52 Google Spanner
46:10 CockroachDB
52:00 Manual vs. automatic sharding
55:28 Consistency models and strong consistency
1:00:03 Raft consensus
1:06:15 How AI brought Peter back to coding
1:19:12 Peter’s tools and agentic workflows
1:23:08 How AI can improve quality
1:26:39 Code reviews: are they done?
1:29:17 100x engineers
1:35:33 Peter’s advice for leveling up your engineering skills
Brought to you by:
• @turbopuffer – the team are completely redesigning their storage architecture from first principles. And they’re documenting all of it! Follow along,: turbopuffer.com/v3
• @linear – most of us work with agents in a “single-player” setting. Linear’s take is agent work should be teamwork, and they built it so: linear.app/pragmatic
• @WorkOS – fresh from the WorkOS oven: Airlock, the authorization layer for AI agents. It evaluates every request against the agent’s intent and your rules and either allows it, denies it, or routes it to a human for approval. workos.com/airlock
Three fascinating parts from this convo:
1. Peter was recruited to Google because of... GIMP!
The very first version of the famous Google logo was made in GIMP, the free, open-source raster graphics and image editing software created by Peter and his college roommate Spencer Kimball.
In 2001, Sergey reached out to Peter, invited him to an interview and then made an offer. While Peter said no back then (thanks to the commute), he joined a year later, in 2002.
2. Peter twice “beat” standard library data structures in performance terms.
At Google, one of his colleagues noticed that std::map showed up in memory profiles. Peter looked closer and figured out that the std::map implementation is a red-black tree, meaning every node has two pointers. Peter built a B-tree with nearly the same semantics which was both faster, due to spatial locality, and smaller because it used fewer pointers. They used this data structure inside Google.
Peter also did something similar with Go’s map: it was performant, but he built a Swiss Table implementation that was faster. That implementation later made it into the Go library, with the Go team helping to finish it!
3. If you squint hard, everything in distributed databases and storage systems starts looking like a B-tree.
These B-trees are a recurring theme in this podcast episode: the backend of Gmail, the std::map replacement, CockroachDB’s range index, etc. There’s even a paper on this phenomenon, The Ubiquitous B-Tree
Deepak Kumar retweeted
Rust has one serialization library that is Serde. Over the years I collected a bucket list of all the things that I do not enjoy about it. Finally did something about it and finished Deser. Here is a small summary of the why, what and how: lucumr.pocoo.org/2026/9/29/d…
Deepak Kumar retweeted
study.devatlas.site
wtf this is free??? 💀 Your one-stop place to revise tech for interviews.
Deepak Kumar retweeted
A CS undergrad asked me where he could have most effect in the AI age. I said probably at either extreme: either close to the technology, actually making LLMs, or close to the customer, using AI to give them exactly what they want. Or maybe both if you can stretch that far.
Deepak Kumar retweeted
Jev is now available to everyone. No waitlist.
Start using it here: console.typesafe.ai
Deepak Kumar retweeted
i just open-sourced it for those curious! github.com/jarrodwatts/jev-t…
Deepak Kumar retweeted
one CPU core can look at 28.8 million packets a second before the Linux kernel even wakes up
that's XDP. it runs inside the network driver, before the kernel allocates memory for the packet, before there's a socket, before there's a stack.
it just gets to decide: keep this, or throw it away, right now
most backend engineers never touch this layer in their whole career. I think that's a shame, because the stuff built on top of it is wild:
- Cloudflare drops over 10 million packets a second per core with it. they've auto-mitigated 100+ attacks above 2 billion packets per second, one of them a 3.8 Tbps flood, and nobody on their team touched a keyboard
- Meta's load balancer (Katran) runs this at every Facebook data center. the old way, iptables-style, tops out around a million packets a second per core and pins the CPU at 90% doing it. Katran does 10-100x that
- Cilium uses the same trick to rip iptables out of Kubernetes entirely. their own benchmarks show p99 network latency dropping from 45ms to under 2ms after the switch
all three are the same idea wearing different hats: read the packet before the OS does, decide fast, skip everything you don't need
I got curious enough to build a tiny version myself, an XDP load balancer with Maglev hashing, no shortcuts.
on one core of a normal laptop it pushed 9 million+ packets a second with zero drops. that's a laptop, not a data center NIC
code's here if you want to see how it works (written in Zig btw):
github.com/immanuwell/ringze…
what's the most "the kernel isn't supposed to be able to do that" thing you've seen eBPF pull off?
Deepak Kumar retweeted
How do you manage database connections when you have 1M+ clients hitting 500k+ database hosts? We’re sharing a deep dive into ZGateway, the stateless proxy tier handling traffic for ZippyDB (>1B ops/sec).
The direct-access model created an unbounded TLS connection mesh, wasting CPU and causing OOM crashes during reconnection storms. ZGateway decoupled the fleets, dropping per-host connections by 97–98% with just a 6% compute overhead.
Read the full engineering deep dive here: engineering.fb.com/2026/09/0…
Deepak Kumar retweeted
Day 2 lectures:
@AndrewLewisPye on 2-round finality
youtube.com/watch?v=dlvJRXrQ…
@SashaSpiegelman on Everlasting BFT: How Changing Requirements Keep Reinventing Consensus
youtube.com/watch?v=y_yiNkfJ…
@fradamt and @yannvon on Introduction to Ethereum Consensus
-Part 1: youtube.com/watch?v=6_EsGMv1…
-Part 2: youtube.com/watch?v=VA_ukotk…
Day 1 lectures from the summer school are now posted:
@ittaia on Many Variants of Simplex
youtube.com/watch?v=Z68hfZe_…
@TheWattenhofer on Reinventing Solana: Alpenglow and Constellation
youtube.com/watch?v=Jp1_UZ5p…
Dongning Guo on Longest-Chain Consensus: Safety under Network Delays
youtube.com/watch?v=Hz7lI0qx…
@maria__apo on The Network Beneath the “Computer in the Sky”
youtube.com/watch?v=lIEi6s0Q…
Deepak Kumar retweeted
Distributed Systems series: done.
Added the final 6 articles to the repo today, closing out a series that covers consensus algorithms (Paxos, Raft), replication strategies, consistency models, partitioning, failure detection, and real-world case studies like Dynamo, Cassandra, Spanner, and Kafka.
That brings the repo to 66 articles total.
For anyone new here: it's a GitHub repo where I index every technical article I write, sorted by category so you're not stuck scrolling my timeline to find something. Right now it covers:
HLD (full system design breakdowns: YouTube, Netflix, Spotify, Instagram, Amazon, and more) LLD (parking lots, vending machines, music systems, and other classics) Backend Engineering (APIs, databases, caching, concurrency, security) Distributed Systems (now complete, 9 parts start to finish) Microservices (decomposition, communication, sagas, data ownership) AI (just getting started)
360 stars and counting. If you haven't checked it out:
github.com/harshit3011/Techn…






































