Introduction
As part of the HA (High Availability) investigation, the decision was made to focus on the use of certain tools in testing and implementation.
As a database HA solution for Postgres, the Patroni project was chosen. Patroni's solution involves a primary-replica structure for replication. Writes and reads are made to the primary and the data is streamed asynchronously to the replicas. The primary is referred to as the leader in the documentation and this database instance becomes leader by way of a leader election. The elected leader holds a lock. All other instances see this and assume a role as a secondary. If the leader node dies, a new election is done and a new leader is selected. This process is extremely quick.
For backup and restore a tool called Velero was chosen. Velero is a widely used and comprehensive tool for backup of Kubernetes resources, including persistent volumes. The CLI provided by Velero is easy to use and adopts a good amount of the syntax used by kubectl.
The remainder of this document will show a deployment of a 5 pod Patroni cluster inside a Kubernetes environment. The database cluster will then be backed-up, destroyed and then restored - in that order.
Notes
- All installation takes place on Ubuntu 20.04
- This demonstration assumes that you have a running Kubernetes cluster and a helm repo (The name used in this case for the repo is "local").
- A tool called "minio" is used to simulate connection to AWS. In a real world situation, minio would not be needed because a Kubernetes cluster would be backed by cloud provider storage.
- This demo is based on kubernetes server version 1.22.6 and client (kubectl) version 1.23.3
Running minio
Firstly, we will install minio on the machine you are using with Docker. This command pulls the minio image and creates the container with a volume called "data" mapped to the /data directory.
docker run --name minio -p 9000:9000 -v data:/data minio/minio server /data
The output should be similar to the below. Pay particular attention to the credentials and the address of the web ui.
Navigate to the web ui and enter in the credentials.
Once the credentials are entered, you can create an S3 bucket called "kubedemo"
That is all the setup required for minio.
Velero Installation
Firstly we download the required version of the velero client, untar, put the binaries in the appropriate place and delete the tar.gz
wget https://github.com/vmware-tanzu/velero/releases/download/v1.8.0-rc.1/velero-v1.8.0-rc.1-linux-amd64.tar.gz tar zxf velero-v1.0.0-linux-amd64.tar.gz sudo mv velero-v1.0.0-linux-amd64/velero /usr/local/bin/ rm -rf velero*
This completes the installation of the Velero client.
Then run the following to create a credentials file that Velero will use to access minio. Change the credentials accordingly.
cat <<EOF>> minio.credentials [default] aws_access_key_id=minioadmin aws_secret_access_key=minioadmin EOF
Finally, we can install the Velero components in the cluster with the following. Change the IP of your minio installation accordingly and make sure you provide the correct path to "minio.credentials".
velero install --provider aws --bucket kubedemo \ --secret-file ./minio.credentials \ --backup-location-config region=minio,s3ForcePathStyle=true,s3Url=http://172.17.0.2:9000 \ --plugins velero/velero-plugin-for-aws:v1.4.0-rc1 \ --use-volume-snapshots false --use-restic
Once this executes successfully, Velero is fully installed. If you check your Kubernetes namespaces you should now see one named "velero" with both Velero and Restic pods running. Restic enables us to backup persistent volumes that are stored in our local file system.