Nejc

Rewriting Debezium in Rust sounds like the obvious move. We didn’t do it. And that decision matters more than you think. An in-depth look at our CDC strategy. Covering: Architecture decisions Tradeoffs Ecosystem realities Custom engineering RisingWave’s core compute and storage engine is written in Rust. So people ask: “Why not build a Rust-native CDC engine and replace Debezium?” We tried prototypes and tests. Our decision: Do not rewrite Debezium. Embed and customize it instead. CDC is more than a parser. CDC is ecosystem-heavy: Debezium is the de facto standard Connectors for Postgres, MySQL, SQL Server, etc. Years of fixes for edge cases, encodings, DDL, and versions Active community tracking database evolution A Rust rewrite would try to rebuild that ecosystem and its maintenance. Why Embedded Debezium, not Debezium Server? We chose the Embedded Engine: Server mode Separate process that reads from DB and writes to Kafka or Pulsar Generic forwarder, hard to deeply customize Embedded mode Java library inside our process We pull CDC events via API and control buffering, transforms, backpressure, and errors CDC runs inside RW with the same scheduler, checkpoints, monitoring, and alerts This keeps CDC aligned with RisingWave’s streaming DAG and recovery model. What we built on top of Debezium? Embedding Debezium lets us upgrade behavior where it matters most: Lock-free snapshot + incremental stitching Record a snapshot boundary, stream snapshot rows and WAL or binlog events together, then deduplicate downstream for a consistent initial state without long locks. Parallel backfill for large tables Slice big tables by primary key ranges, backfill in parallel, and adapt thread count to upstream DB load for faster copy without overload. Cross-cloud schema history Pluggable history for S3, GCS, Azure Blob, etc, fitting multi-cloud deployments, migrations, and failovers. Memory-aware schema history Segment history by size or time and load only what is needed, reducing memory peaks and stabilizing long-running CDC. TOAST column completion for Postgres When Debezium sends __debezium_unavailable_value, we fetch the historical TOAST value and merge it back so rows stay complete. Instant schema change detection Use PostgreSQL Relation messages from the replication protocol for near real-time schema sync with fewer metadata polls and less DB load. All of this is possible because Debezium runs inside RW, not as an external black box. Pragmatism over rewrites! Debezium gives us maturity, widely used connectors and battle-tested log parsing. A Rust rewrite would be multi-year, high-risk maintenance with limited gain over embedding and extending Debezium. What we needed was a stable, controllable, deeply integrated CDC entry point for RW, not a brand new CDC ecosystem. So our choice: Use Debezium Embedded Engine as the foundation Customize where our workloads demand it Focus on engineering efficiency, stability, and maintainability instead of rewriting everything in Rust for its own sake.