$ cat "Building an Enterprise High-Availability PostgreSQL 16 Cluster: A Complete Guide.md"
Databases DevOps

High-Availability PostgreSQL 16 Cluster with Patroni

When your business operates 24/7, database downtime translates directly into financial losses and blocked customer operations. While PostgreSQL native streaming replication mirrors data reliably, it does not handle automated failover or split-brain protection on its own.

In this guide, the Noxet team details our battle-tested enterprise architecture for high-availability database infrastructure using PostgreSQL 16, distributed consensus on etcd v3, lifecycle manager Patroni, and HAProxy load balancing.

Target Architecture

To guarantee quorum and eliminate split-brain hazards, a minimum of three distributed consensus nodes (DCS) is required. The production topology:

               +----------------------------------+
               |  Client Applications & Services  |
               +-----------------+----------------+
                                 |
                     +-----------v-----------+
                     |  HAProxy (VRRP / VIP) |
                     +-----+-----------+-----+
         Port 5432 (Write) |           | Port 5433 (Read-Only)
        (Health: /primary) |           | (Health: /replica)
                           |           |
             +-------------+-----------+-------------+
             |                                       |
    +--------v--------+     Streaming Repl      +--------v--------+
    |   pg-node-01    |========================>|   pg-node-02    |
    | (Primary/Leader)|                         |  (Replica/Sync) |
    |   Patroni:8008  |                         |   Patroni:8008  |
    +--------+--------+                         +--------+--------+
             |                                       |
             +-------------------+-------------------+
                                 |
                 +---------------v---------------+
                 |    etcd Cluster (3 Quorum)    |
                 |  [etcd-01] [etcd-02] [etcd-03]|
                 +-------------------------------+

Core Architecture Components:

  1. PostgreSQL 16: Relational DBMS configured with streaming replication and checksums.
  2. etcd v3: Distributed key-value store maintaining cluster consensus and leader lease.
  3. Patroni: Daemon managing the PostgreSQL lifecycle on each node, tracking health metrics, and orchestrating zero-data-loss failover.
  4. HAProxy: TCP load balancer querying Patroni HTTP health endpoints (:8008/primary and :8008/replica).

1. Patroni Configuration (patroni.yml)

Patroni handles cluster bootstrap, dynamic parameter management, and pg_hba.conf generation. Below is a production-ready configuration for primary node pg-node-01:

scope: pg-ha-cluster
namespace: /service
name: pg-node-01

# Distributed Configuration Store (etcd3)
etcd3:
  hosts:
    - 192.168.10.11:2379
    - 192.168.10.12:2379
    - 192.168.10.13:2379

# Patroni REST API for health checks and patronictl CLI management
restapi:
  listen: 192.168.10.21:8008
  connect_address: 192.168.10.21:8008

# Global cluster parameters (persisted in etcd on first bootstrap)
bootstrap:
  dcs:
    ttl: 30
    loop_wait: 10
    retry_timeout: 10
    maximum_lag_on_failover: 1048576 # 1 MB maximum WAL lag for automated promotion
    synchronous_mode: true
    synchronous_mode_strict: false

    postgresql:
      use_pg_rewind: true
      use_slots: true
      parameters:
        wal_level: replica
        max_wal_senders: 10
        max_replication_slots: 10
        hot_standby: "on"
        wal_keep_size: 2048MB
        max_connections: 500
        shared_buffers: 8GB
        effective_cache_size: 24GB
        maintenance_work_mem: 2GB
        checkpoint_completion_target: 0.9
        wal_compression: "on"

  initdb:
    - encoding: UTF8
    - data-checksums

  pg_hba:
    - host replication replicator 192.168.10.0/24 md5
    - host all all 192.168.10.0/24 md5
    - host all all 127.0.0.1/32 md5

# Local node settings
postgresql:
  listen: 192.168.10.21:5432
  connect_address: 192.168.10.21:5432
  data_dir: /var/lib/postgresql/16/main
  bin_dir: /usr/lib/postgresql/16/bin
  pgpass: /var/lib/postgresql/.pgpass
  authentication:
    replication:
      username: replicator
      password: "StrongReplicationSecret2026!"
    superuser:
      username: postgres
      password: "StrongSuperuserSecret2026!"

tags:
  nofailover: false
  noloadbalance: false
  clonefrom: false
  nosync: false

For standby nodes (pg-node-02, pg-node-03), only the node name and local IP addresses in restapi and postgresql sections are adjusted.

2. Load Balancing with HAProxy

HAProxy decouples applications from the physical database instances. Traffic is segregated into two dedicated listeners:

  • Port 5432: Routes write transactions strictly to the current Leader/Primary.
  • Port 5433: Distributes read-only queries across healthy standby replicas.

Production snippet for /etc/haproxy/haproxy.cfg:

global
    log /dev/log local0
    maxconn 4096
    user haproxy
    group haproxy

defaults
    log     global
    mode    tcp
    option  tcplog
    timeout connect 5000ms
    timeout client  50000ms
    timeout server  50000ms

# --- Master / Primary (Read-Write) ---
frontend pg_cluster_write
    bind 0.0.0.0:5432
    default_backend pg_cluster_primary

backend pg_cluster_primary
    mode tcp
    option httpchk GET /primary
    http-check expect status 200
    default-server inter 3s fall 3 rise 2 on-marked-down shutdown-sessions
    server pg-node-01 192.168.10.21:5432 maxconn 300 check port 8008
    server pg-node-02 192.168.10.22:5432 maxconn 300 check port 8008
    server pg-node-03 192.168.10.23:5432 maxconn 300 check port 8008

# --- Standby Replicas (Read-Only) ---
frontend pg_cluster_read
    bind 0.0.0.0:5433
    default_backend pg_cluster_replicas

backend pg_cluster_replicas
    mode tcp
    balance roundrobin
    option httpchk GET /replica
    http-check expect status 200
    default-server inter 3s fall 3 rise 2
    server pg-node-01 192.168.10.21:5432 check port 8008
    server pg-node-02 192.168.10.22:5432 check port 8008
    server pg-node-03 192.168.10.23:5432 check port 8008

3. Split-Brain Mitigation and Safe Recovery

Split-brain occurs when network partitioning causes two instances to claim primary status simultaneously, causing divergence of database state.

Defensive Mechanics in Patroni:

  1. TTL-based Leases: The leader must renew its lease key in etcd every loop_wait (10 seconds). If communication with the etcd quorum is severed, the lease naturally expires.
  2. Hardware/Kernel Watchdog Fencing: Patroni integrates with /dev/watchdog. If the Patroni process is suspended or loses quorum contact, the OS is forcibly reset before the lease expires, preventing split writes.
  3. Automated pg_rewind: When a partitioned primary rejoins the cluster, Patroni leverages pg_rewind to roll back divergent WAL segments to the timeline branch point, transforming the old leader into a healthy replica without a full rebuild.

4. Daily Operations & Observability

Cluster administration is streamlined via the patronictl CLI:

# Inspect topology and synchronization health
patronictl -c /etc/patroni/patroni.yml list

# Graceful zero-downtime planned switchover
patronictl -c /etc/patroni/patroni.yml switchover --master pg-node-01 --candidate pg-node-02

# Restart a node following parameter reconfiguration
patronictl -c /etc/patroni/patroni.yml restart pg-ha-cluster pg-node-01

Monitoring via Prometheus:

Patroni provides built-in metrics via its /metrics endpoint. Prometheus tracks patroni_replication_lag, instance role changes (patroni_is_leader), and etcd quorum health.

Conclusion

This architecture delivers a Recovery Time Objective (RTO) under 15 seconds and a Recovery Point Objective (RPO) of 0 in synchronous replication mode. It reliably withstands node crashes, network splits, and scheduled maintenance without application downtime.

Need an architecture audit or high-load database cluster deployment for your business? Contact the Noxet team.