Skip to content

Repository files navigation

PostgreSQL Logical Replication & CDC Pipeline Automation

PostgreSQL Logical Replication & CDC Pipeline Automation

A production-grade, open reference for running real-time PostgreSQL data movement at scale.
Logical decoding, replication slots, publications & subscriptions, and change-data-capture pipelines with Python & Debezium — including monitoring, failover, and schema-change discipline.

🌐 Read it at logicalreplication.org →


What this is

logicalreplication.org is a free, deeply cross-linked technical reference for the engineers who actually operate PostgreSQL logical replication and CDC in production. Every page assumes you are already in production: concrete GUC values, real system-view queries, byte and time thresholds, zero-downtime rollout sequences with revert paths, and version-specific callouts for PostgreSQL 14 through 17.

It is written for database engineers, data-platform teams, Python ETL developers, and DevOps — the people who get paged when a replication slot pins the write-ahead log at 3 a.m.

Who it's for

  • You run logical replication or CDC on PostgreSQL and need answers that go past the manual's happy path.
  • You are building a Debezium or Python change-data-capture pipeline and want the failure modes documented before you meet them.
  • You need copy-ready SQL and Python, diagnostic queries with alert thresholds, and safe deployment sequences you can hand to on-call.

What it covers

The reference is organized into four connected areas:

Area What you'll find
Architecture & Fundamentals How the write-ahead log, logical decoding, replication slots, and publication/subscription models actually work — plus the trust boundaries around them.
Setup & Management Creating publications, initializing and recovering replication slots, subscription sync procedures, and monitoring & alerting with Prometheus, Grafana, and Alertmanager.
CDC Pipelines with Python & Debezium Debezium connector configuration and snapshot modes, parsing the pgoutput stream, async consumers, JSON-to-Avro transformation with a schema registry, Kafka event routing, and dead-letter-queue patterns.
Topologies & Failover Operations Failover and slot recovery (including PostgreSQL 17 synchronized failover slots), bidirectional replication and conflict resolution, and online schema change under an active CDC slot.

Every page carries hand-authored diagrams, structured data for search, and inline links to the exact adjacent topic you need next.

Highlights

  • 51 in-depth guides spanning first principles to advanced failover and schema-change operations.
  • Runnable SQL & Python targeting PostgreSQL 14–17, with version-specific behavior called out inline.
  • Diagnostic queries with real thresholds against pg_replication_slots, pg_stat_replication, and pg_stat_subscription.
  • Original, accessible SVG diagrams for the hardest concepts — LSN timelines, slot state machines, and end-to-end pipeline flows.
  • Fast, installable, offline-capable — a static site that works as a progressive web app.

Built with

  • Eleventy — static site generation
  • Hand-authored inline SVG, structured data (JSON-LD), and a service worker for offline use
  • Deployed on Cloudflare Workers
npm install
npm run build     # build the static site into _site/
npm run serve     # local dev server with live reload

License

Content © the site owner. Code is released under the MIT License.


Explore the full reference at logicalreplication.org →

About

Production-grade open reference for PostgreSQL logical replication & CDC pipelines — replication slots, publications, Debezium, monitoring, and failover.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages