PostgreSQL High Availability with Patroni: Complete Implementation Guide

Introduction
In today's 24/7 business environment, database downtime can cost organizations thousands of dollars per minute in lost revenue, damaged reputation, and operational disruption. PostgreSQL, while incredibly robust and reliable, requires careful architecture and configuration to achieve true high availability (HA). Traditional PostgreSQL replication solutions often lack the automation and intelligence needed for seamless failover in production environments.
Patroni emerged as a game-changing solution that transforms PostgreSQL high availability from a complex, manual process into an automated, intelligent system. Developed by Zalando, Patroni is a template for creating custom PostgreSQL high availability solutions using Python and a distributed configuration store like etcd, Consul, or ZooKeeper.
This comprehensive guide will walk you through implementing PostgreSQL high availability with Patroni, covering everything from initial setup to advanced configuration scenarios. Whether you're a database administrator responsible for mission-critical systems, a DevOps engineer implementing infrastructure automation, or a business leader evaluating HA solutions, this guide provides the knowledge needed to build resilient PostgreSQL environments.
You'll learn how to set up Patroni clusters, configure automatic failover, implement monitoring and alerting, and apply best practices that ensure your PostgreSQL databases remain available even during hardware failures, network partitions, and planned maintenance.
Table of Contents
Understanding PostgreSQL High Availability Challenges
Traditional PostgreSQL high availability implementations face several fundamental challenges that Patroni elegantly addresses.
Traditional HA Limitations
Manual Failover Complexity
- Requires human intervention during outages
- Potential for human error during high-stress situations
- Slow recovery times impacting business operations
- Complex scripts and procedures to maintain
Split-Brain Scenarios
- Multiple nodes believing they're the primary
- Data inconsistency and corruption risks
- Difficult to detect and resolve automatically
- Can lead to permanent data loss
Configuration Management
- Manual synchronization of configuration changes
- Inconsistent settings across cluster nodes
- No centralized configuration management
- Difficult to maintain as clusters scale
Monitoring and Alerting Gaps
- Limited visibility into cluster health
- Reactive rather than proactive monitoring
- Difficulty correlating metrics across nodes
- No automated health checks and remediation
How Patroni Solves These Challenges
Patroni addresses these limitations through several key innovations:
- Distributed Consensus: Uses etcd, Consul, or ZooKeeper for cluster coordination
- Automatic Failover: Intelligent leader election and failover without human intervention
- Configuration Management: Centralized, dynamic configuration updates
- Health Monitoring: Continuous health checks and automatic remediation
- REST API: Programmatic cluster management and monitoring
Patroni Architecture and Components
Understanding Patroni's architecture is crucial for successful implementation and troubleshooting.
Core Components
Patroni Agent
- Runs on each PostgreSQL node
- Manages local PostgreSQL instance lifecycle
- Communicates with distributed configuration store
- Implements health checks and failover logic
Distributed Configuration Store (DCS)
- Stores cluster state and configuration
- Provides distributed locking mechanisms
- Enables leader election and consensus
- Common options: etcd, Consul, ZooKeeper
HAProxy/Load Balancer
- Routes application traffic to current primary
- Provides health checking for application connections
- Enables transparent failover for applications
- Can be configured for read/write splitting
REST API
- Provides cluster status and control endpoints
- Enables integration with monitoring systems
- Allows programmatic cluster management
- Supports custom health checks
Cluster Topology
A typical Patroni cluster consists of multiple PostgreSQL nodes managed by Patroni agents, coordinated through a distributed configuration store, and accessed via a load balancer:
Setting Up Your First Patroni Cluster
Let's walk through setting up a three-node Patroni cluster with etcd as the distributed configuration store.
Prerequisites
System Requirements:
- 3 servers for PostgreSQL nodes (minimum 4GB RAM, 2 CPU cores each)
- 3 servers for etcd cluster (can be co-located with PostgreSQL in test environments)
- Network connectivity between all nodes
- Synchronized system clocks (NTP recommended)
Software Requirements:
# On each PostgreSQL node sudo apt-get update sudo apt-get install -y postgresql-14 postgresql-contrib-14 sudo apt-get install -y python3-pip python3-dev sudo pip3 install patroni[etcd]
Step 1: Setting Up etcd Cluster
etcd Node 1 (etcd1.example.com):
# Download and install etcd
ETCD_VER=v3.5.9
wget https://github.com/etcd-io/etcd/releases/download/${ETCD_VER}/etcd-${ETCD_VER}-linux-amd64.tar.gz
tar xzf etcd-${ETCD_VER}-linux-amd64.tar.gz
sudo mv etcd-${ETCD_VER}-linux-amd64/etcd* /usr/local/bin/
# Create etcd user and directories
sudo useradd -r -s /bin/false etcd
sudo mkdir -p /var/lib/etcd
sudo chown etcd:etcd /var/lib/etcdCreate systemd service:
sudo tee /etc/systemd/system/etcd.service > /dev/null <<EOF [Unit] Description=etcd key-value store Documentation=https://github.com/etcd-io/etcd After=network.target [Service] User=etcd Type=notify Environment=ETCD_DATA_DIR=/var/lib/etcd Environment=ETCD_NAME=etcd1 Environment=ETCD_INITIAL_ADVERTISE_PEER_URLS=http://etcd1.example.com:2380 Environment=ETCD_LISTEN_PEER_URLS=http://0.0.0.0:2380 Environment=ETCD_LISTEN_CLIENT_URLS=http://0.0.0.0:2379 Environment=ETCD_ADVERTISE_CLIENT_URLS=http://etcd1.example.com:2379 Environment=ETCD_INITIAL_CLUSTER=etcd1=http://etcd1.example.com:2380,etcd2=http://etcd2.example.com:2380,etcd3=http://etcd3.example.com:2380 Environment=ETCD_INITIAL_CLUSTER_STATE=new Environment=ETCD_INITIAL_CLUSTER_TOKEN=patroni-cluster ExecStart=/usr/local/bin/etcd Restart=always RestartSec=10s LimitNOFILE=40000 [Install] WantedBy=multi-user.target EOF sudo systemctl daemon-reload sudo systemctl enable etcd sudo systemctl start etcd
Repeat similar configuration for etcd2 and etcd3 nodes, adjusting the ETCD_NAME and ETCD_INITIAL_ADVERTISE_PEER_URLS accordingly.
Step 2: Configuring Patroni on PostgreSQL Nodes
Create Patroni configuration file (/etc/patroni/patroni.yml) on Node 1:
scope: postgres-cluster
namespace: /patroni/
name: postgres1
restapi:
listen: 0.0.0.0:8008
connect_address: postgres1.example.com:8008
etcd:
hosts:
- etcd1.example.com:2379
- etcd2.example.com:2379
- etcd3.example.com:2379
bootstrap:
dcs:
ttl: 30
loop_wait: 10
retry_timeout: 30
maximum_lag_on_failover: 1048576
postgresql:
use_pg_rewind: true
use_slots: true
parameters:
wal_level: replica
hot_standby: "on"
max_connections: 100
max_worker_processes: 8
wal_keep_segments: 8
max_wal_senders: 10
max_replication_slots: 10
max_prepared_transactions: 0
max_locks_per_transaction: 64
wal_log_hints: "on"
track_commit_timestamp: "off"
archive_mode: "on"
archive_timeout: 1800s
archive_command: 'mkdir -p ../wal_archive && test ! -f ../wal_archive/%f && cp %p ../wal_archive/%f'
recovery_conf:
restore_command: 'cp ../wal_archive/%f %p'
initdb:
- encoding: UTF8
- data-checksums
pg_hba:
- host replication replicator 127.0.0.1/32 md5
- host replication replicator 10.0.0.0/8 md5
- host all all 0.0.0.0/0 md5
users:
admin:
password: admin_password
options:
- createrole
- createdb
replicator:
password: replicator_password
options:
- replication
postgresql:
listen: 0.0.0.0:5432
connect_address: postgres1.example.com:5432
data_dir: /var/lib/postgresql/14/main
bin_dir: /usr/lib/postgresql/14/bin
config_dir: /etc/postgresql/14/main
pgpass: /tmp/pgpass
authentication:
replication:
username: replicator
password: replicator_password
superuser:
username: postgres
password: postgres_password
parameters:
unix_socket_directories: '/var/run/postgresql'
tags:
nofailover: false
noloadbalance: false
clonefrom: false
nosync: falseStep 3: Starting Patroni Services
Create systemd service file (/etc/systemd/system/patroni.service):
[Unit] Description=Runners to orchestrate a high-availability PostgreSQL After=syslog.target network.target [Service] Type=simple User=postgres Group=postgres ExecStart=/usr/local/bin/patroni /etc/patroni/patroni.yml KillMode=process TimeoutSec=30 Restart=no [Install] WantedBy=multi-user.target
Start Patroni on all nodes:
sudo systemctl daemon-reload sudo systemctl enable patroni sudo systemctl start patroni
Step 4: Verifying Cluster Status
Check cluster status via Patroni REST API:
# Check cluster members curl -s http://postgres1.example.com:8008/cluster | jq # Check node status curl -s http://postgres1.example.com:8008/patroni | jq # Check primary node curl -s http://postgres1.example.com:8008/primary
Use patronictl command-line tool:
# Install patroni command line tools pip3 install patroni[etcd] # List cluster members patronictl -c /etc/patroni/patroni.yml list # Show cluster topology patronictl -c /etc/patroni/patroni.yml topology
Configuration Deep Dive
Understanding Patroni's configuration options is crucial for optimizing your cluster for specific requirements.
DCS (Distributed Configuration Store) Settings
bootstrap:
dcs:
ttl: 30 # Leader key TTL in seconds
loop_wait: 10 # Main loop sleep time
retry_timeout: 30 # Retry timeout for DCS operations
maximum_lag_on_failover: 1048576 # Max lag in bytes for failover
master_start_timeout: 300 # Timeout for master start
synchronous_mode: false # Enable synchronous replication
synchronous_mode_strict: false # Strict synchronous mode
postgresql:
use_pg_rewind: true # Use pg_rewind for faster recovery
use_slots: true # Use replication slots
recovery_conf:
restore_command: 'cp ../wal_archive/%f %p'PostgreSQL Parameter Tuning
Critical parameters for high availability:
postgresql:
parameters:
# Replication settings
wal_level: replica
max_wal_senders: 10
max_replication_slots: 10
hot_standby: "on"
# Performance settings
shared_buffers: 1GB
effective_cache_size: 3GB
work_mem: 4MB
maintenance_work_mem: 256MB
# Reliability settings
fsync: "on"
synchronous_commit: "on"
wal_sync_method: fdatasync
full_page_writes: "on"
wal_log_hints: "on"
# Monitoring settings
log_destination: 'stderr'
logging_collector: "on"
log_directory: '/var/log/postgresql'
log_filename: 'postgresql-%Y-%m-%d_%H%M%S.log'
log_min_duration_statement: 1000Advanced Failover Configuration
Synchronous Replication Setup:
bootstrap:
dcs:
synchronous_mode: true
synchronous_mode_strict: true
postgresql:
parameters:
synchronous_standby_names: '*'
synchronous_commit: "on"Custom Failover Conditions:
bootstrap:
dcs:
maximum_lag_on_failover: 1048576 # 1MB max lag
check_timeline: true # Verify timeline consistency
postgresql:
use_pg_rewind: true
remove_data_directory_on_rewind_failure: trueFailover Scenarios and Testing
Proper testing ensures your HA setup will perform correctly during real outages.
Planned Failover (Switchover)
Manual switchover using patronictl:
# Perform graceful switchover to specific node patronictl -c /etc/patroni/patroni.yml switchover postgres-cluster --master postgres1 --candidate postgres2 # Verify new primary patronictl -c /etc/patroni/patroni.yml list
REST API switchover:
# Initiate switchover via REST API
curl -X POST http://postgres1.example.com:8008/switchover \
-H "Content-Type: application/json" \
-d '{"candidate": "postgres2"}'Automatic Failover Testing
Simulate primary node failure:
# Stop Patroni service (simulates planned maintenance) sudo systemctl stop patroni # Stop PostgreSQL service (simulates database crash) sudo systemctl stop postgresql # Network partition simulation (using iptables) sudo iptables -A INPUT -s postgres2.example.com -j DROP sudo iptables -A INPUT -s postgres3.example.com -j DROP
Monitor failover process:
# Watch cluster status during failover watch -n 1 'patronictl -c /etc/patroni/patroni.yml list' # Check logs for failover events sudo journalctl -u patroni -f
Split-Brain Prevention Testing
Test etcd network partition:
# Isolate one etcd node sudo iptables -A INPUT -s etcd2.example.com -j DROP sudo iptables -A INPUT -s etcd3.example.com -j DROP # Verify cluster maintains quorum etcdctl endpoint health --endpoints=etcd1.example.com:2379,etcd2.example.com:2379,etcd3.example.com:2379
Monitoring and Maintenance
Comprehensive monitoring is essential for maintaining a healthy Patroni cluster.
Health Check Endpoints
Patroni provides several REST API endpoints for monitoring:
# Primary endpoint (returns 200 if node is primary) curl -f http://postgres1.example.com:8008/primary # Replica endpoint (returns 200 if node is healthy replica) curl -f http://postgres1.example.com:8008/replica # Read-only endpoint (returns 200 if node can serve read queries) curl -f http://postgres1.example.com:8008/read-only # Health endpoint (returns detailed cluster health) curl -s http://postgres1.example.com:8008/health | jq
Prometheus Integration
Create Patroni exporter configuration:
# patroni-exporter.yml
global:
scrape_interval: 15s
scrape_configs:
- job_name: 'patroni'
static_configs:
- targets:
- postgres1.example.com:8008
- postgres2.example.com:8008
- postgres3.example.com:8008
metrics_path: /metrics
scrape_interval: 5sKey metrics to monitor:
patroni_postgres_running: PostgreSQL process statuspatroni_master: Primary node indicatorpatroni_replica_lag_in_bytes: Replication lagpatroni_postgres_streaming: Streaming replication statuspatroni_dcs_last_seen: Last successful DCS communication
Log Analysis and Alerting
Important log patterns to monitor:
# Failover events grep -i "failover\|switchover" /var/log/postgresql/postgresql-*.log # Replication lag warnings grep -i "lag" /var/log/postgresql/postgresql-*.log # Connection issues grep -i "connection\|timeout" /var/log/postgresql/postgresql-*.log # Patroni health check failures sudo journalctl -u patroni | grep -i "failed\|error"
Sample alerting rules (Prometheus AlertManager):
groups:
- name: patroni
rules:
- alert: PatroniNodeDown
expr: up{job="patroni"} == 0
for: 30s
labels:
severity: critical
annotations:
summary: "Patroni node {{ $labels.instance }} is down"
- alert: PostgreSQLReplicationLag
expr: patroni_replica_lag_in_bytes > 16777216 # 16MB
for: 2m
labels:
severity: warning
annotations:
summary: "High replication lag on {{ $labels.instance }}"
- alert: NoPostgreSQLPrimary
expr: sum(patroni_master) == 0
for: 1m
labels:
severity: critical
annotations:
summary: "No PostgreSQL primary node available"Real-World Implementation Case Study
Challenge: Financial Services High Availability
A financial services company required 99.99% uptime for their core trading platform database. Their existing PostgreSQL setup used manual failover procedures that typically took 10-15 minutes during outages, causing significant revenue loss and regulatory compliance issues.
Requirements Analysis
Business Requirements:
- Maximum 30 seconds downtime during failover
- Zero data loss tolerance
- 24/7 operation with minimal maintenance windows
- Regulatory compliance for financial data
Technical Requirements:
- 3-node PostgreSQL cluster across multiple availability zones
- Automatic failover with health checking
- Real-time monitoring and alerting
- Backup and point-in-time recovery capabilities
Implementation Details
Infrastructure Setup:
Synchronous Replication Configuration:
bootstrap:
dcs:
synchronous_mode: true
synchronous_mode_strict: true
maximum_lag_on_failover: 0 # Zero data loss
postgresql:
parameters:
synchronous_standby_names: 'postgres2'
synchronous_commit: 'remote_apply'
max_wal_senders: 10
wal_keep_segments: 32Advanced Monitoring Setup:
#!/usr/bin/env python3
# Custom health check script
import requests
import sys
import time
def check_cluster_health():
nodes = [
'https://postgres1.company.com:8008',
'https://postgres2.company.com:8008',
'https://postgres3.company.com:8008'
]
primary_count = 0
healthy_replicas = 0
for node in nodes:
try:
# Check if node is primary
response = requests.get(f"{node}/primary", timeout=5)
if response.status_code == 200:
primary_count += 1
# Check replica health
response = requests.get(f"{node}/replica", timeout=5)
if response.status_code == 200:
healthy_replicas += 1
except requests.exceptions.RequestException:
continue
# Validate cluster state
if primary_count != 1:
print(f"ERROR: Expected 1 primary, found {primary_count}")
sys.exit(1)
if healthy_replicas < 1:
print(f"ERROR: No healthy replicas available")
sys.exit(1)
print("Cluster health check passed")
return True
if __name__ == "__main__":
check_cluster_health()Results and Outcomes
Performance Achievements:
- Failover time: Reduced from 10-15 minutes to 25-30 seconds
- Data loss: Achieved zero data loss through synchronous replication
- Uptime: Improved from 99.9% to 99.995% (exceeding 99.99% target)
- Operational overhead: Reduced by 80% through automation
Business Impact:
- Revenue protection: Eliminated $2M+ annual losses from extended outages
- Compliance: Achieved regulatory requirements for data availability
- Operational efficiency: Reduced DBA on-call incidents by 90%
- Confidence: Enabled aggressive business growth knowing infrastructure could scale
Advanced Configurations
Multi-Region Setup
For geographically distributed applications:
# Primary region configuration
bootstrap:
dcs:
postgresql:
parameters:
archive_mode: "on"
archive_command: 'aws s3 cp %p s3://company-wal-archive/%f'
# Secondary region (async replica)
tags:
nofailover: true # Prevent automatic failover to remote region
noloadbalance: true
bootstrap:
method: restore_or_initdb
restore_or_initdb:
command: pg_basebackup -h primary.region1.company.com -D ${PGDATA} -U replicator -v -P -W
keep_existing_recovery_conf: falseCustom Callback Scripts
Implement custom actions during failover events:
postgresql:
callbacks:
on_start: /etc/patroni/callbacks/on_start.sh
on_stop: /etc/patroni/callbacks/on_stop.sh
on_role_change: /etc/patroni/callbacks/on_role_change.shExample callback script (/etc/patroni/callbacks/on_role_change.sh):
#!/bin/bash
ACTION=$1
ROLE=$2
CLUSTER=$3
case $ACTION in
on_role_change)
if [ "$ROLE" = "master" ]; then
# Actions when becoming primary
echo "$(date): Node became primary" >> /var/log/patroni-callbacks.log
# Update load balancer
curl -X POST http://loadbalancer.company.com/api/update-primary \
-d "host=$(hostname -f)"
# Send notification
curl -X POST http://alerts.company.com/webhook \
-d "message=PostgreSQL primary role changed to $(hostname -f)"
elif [ "$ROLE" = "replica" ]; then
# Actions when becoming replica
echo "$(date): Node became replica" >> /var/log/patroni-callbacks.log
fi
;;
esacSecurity Hardening
SSL/TLS Configuration:
restapi:
listen: 0.0.0.0:8008
certfile: /etc/ssl/certs/patroni.crt
keyfile: /etc/ssl/private/patroni.key
cafile: /etc/ssl/certs/ca.crt
verify_client: required
postgresql:
parameters:
ssl: "on"
ssl_cert_file: '/etc/ssl/certs/server.crt'
ssl_key_file: '/etc/ssl/private/server.key'
ssl_ca_file: '/etc/ssl/certs/ca.crt'
ssl_ciphers: 'ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256'Authentication and Authorization:
restapi:
authentication:
username: admin
password: secure_password
postgresql:
authentication:
replication:
username: replicator
password: replication_password
superuser:
username: postgres
password: postgres_passwordBest Practices for Production
Capacity Planning
Hardware Recommendations:
| Component | Minimum | Recommended | Enterprise |
|---|---|---|---|
| CPU Cores | 4 | 8 | 16+ |
| RAM | 8GB | 32GB | 64GB+ |
| Storage | SSD 100GB | NVMe 500GB | NVMe 1TB+ |
| Network | 1Gbps | 10Gbps | 25Gbps+ |
etcd Resource Requirements:
| Cluster Size | CPU | RAM | Storage |
|---|---|---|---|
| 3 nodes | 2 cores | 4GB | 20GB SSD |
| 5 nodes | 2 cores | 8GB | 20GB SSD |
| 7 nodes | 4 cores | 8GB | 50GB SSD |
Network Configuration
Firewall Rules:
# PostgreSQL -A INPUT -p tcp --dport 5432 -s 10.0.0.0/8 -j ACCEPT # Patroni REST API -A INPUT -p tcp --dport 8008 -s 10.0.0.0/8 -j ACCEPT # etcd client communication -A INPUT -p tcp --dport 2379 -s 10.0.0.0/8 -j ACCEPT # etcd peer communication -A INPUT -p tcp --dport 2380 -s 10.0.0.0/8 -j ACCEPT
Network Latency Considerations:
- Same DC: < 1ms latency for optimal performance
- Cross-AZ: < 10ms acceptable for synchronous replication
- Cross-Region: > 50ms requires careful configuration
Backup and Recovery
Continuous Archiving Configuration:
postgresql:
parameters:
archive_mode: "on"
archive_command: 'test ! -f /backup/wal_archive/%f && cp %p /backup/wal_archive/%f'
archive_timeout: 300Point-in-time Recovery Setup:
# Create base backup pg_basebackup -h postgres1.example.com -D /backup/base -U replicator -v -P # Recovery configuration echo "restore_command = 'cp /backup/wal_archive/%f %p'" > /backup/recovery.conf echo "recovery_target_time = '2025-07-08 14:30:00'" >> /backup/recovery.conf
Performance Optimization
Connection Pooling with PgBouncer:
# /etc/pgbouncer/pgbouncer.ini [databases] mydb = host=postgres-vip.example.com port=5432 dbname=mydb [pgbouncer] listen_port = 6432 listen_addr = * auth_type = md5 auth_file = /etc/pgbouncer/users.txt pool_mode = transaction max_client_conn = 1000 default_pool_size = 100 reserve_pool_size = 10
Load Balancing Configuration:
# HAProxy configuration for read/write splitting backend postgres_primary option httpchk GET /primary server postgres1 postgres1.example.com:5432 check port 8008 server postgres2 postgres2.example.com:5432 check port 8008 backup server postgres3 postgres3.example.com:5432 check port 8008 backup backend postgres_replicas balance roundrobin option httpchk GET /replica server postgres2 postgres2.example.com:5432 check port 8008 server postgres3 postgres3.example.com:5432 check port 8008
FAQ Section
Conclusion
Patroni represents a significant advancement in PostgreSQL high availability, transforming complex manual processes into automated, intelligent systems. By leveraging distributed consensus, automated failover, and comprehensive monitoring, Patroni enables organizations to achieve enterprise-grade availability for their PostgreSQL deployments.
The key to successful Patroni implementation lies in understanding your specific requirements, properly configuring the system for your environment, and implementing comprehensive monitoring and testing procedures. Regular failover testing, performance monitoring, and capacity planning ensure your HA system remains effective as your needs evolve.
Remember that high availability is not just about technology—it requires organizational processes, documentation, and training to be truly effective. Invest in understanding the system, training your team, and establishing clear procedures for both routine operations and emergency scenarios.
Next Steps
- Start with a test environment: Set up a Patroni cluster in a non-production environment to gain experience
- Develop runbooks: Create detailed procedures for common operations and emergency scenarios
- Implement monitoring: Set up comprehensive monitoring and alerting before going to production
- Plan for disasters: Develop and test backup and recovery procedures
- Consider professional services: For critical production deployments, consider engaging experienced consultants
About UduLabs
UduLabs brings over 25 years of database expertise to help organizations implement robust, scalable PostgreSQL high availability solutions. Our team has successfully deployed Patroni clusters for enterprises across various industries, from financial services to e-commerce platforms.
We offer comprehensive services including HA architecture design, implementation, monitoring setup, and ongoing support. Our proven methodologies ensure your PostgreSQL infrastructure meets the highest availability requirements while optimizing for performance and cost-effectiveness.
Contact UduLabs today to learn how we can help you implement a bulletproof PostgreSQL high availability solution tailored to your specific requirements.
*The code snippets provided in this blog are intended as conceptual examples or framework overviews. They are representative and not the complete source code.