Introduction

As part of the HA (High Availability) investigation, the decision was made to focus on the use of certain tools in testing and implementation.

As a database HA solution for Postgres, the Patroni project was chosen. Patroni's solution involves a primary-replica structure for replication. Writes and reads are made to the primary and the data is streamed asynchronously to the replicas. The primary is referred to as the leader in the documentation and this database instance becomes leader by way of a leader election. The elected leader holds a lock. All other instances see this and assume a role as a secondary. If the leader node dies, a new election is done and a new leader is selected. This process is extremely quick.

For backup and restore a tool called Velero was chosen. Velero is a widely used and comprehensive tool for backup of Kubernetes resources, including persistent volumes. The CLI provided by Velero is easy to use and adopts a good amount of the syntax used by kubectl.

The remainder of this document will show a deployment of a 5 pod Patroni cluster inside a Kubernetes environment. The database cluster will then be backed-up, destroyed and then restored - in that order.

Notes

• All installation takes place on Ubuntu 20.04
• This demonstration assumes that you have a running Kubernetes cluster and a helm repo (The name used in this case for the repo is "local").
• A tool called "minio" is used to simulate connection to AWS. In a real world situation, minio would not be needed because a Kubernetes cluster would be backed by cloud provider storage.
• This demo is based on kubernetes server version 1.22.6 and client (kubectl) version 1.23.3

Running minio

Firstly, we will install minio on the machine you are using with Docker. This command pulls the minio image and creates the container with a volume called "data" mapped to the /data directory.

docker run --name minio -p 9000:9000 -v data:/data minio/minio server /data

The output should be similar to the below. Pay particular attention to the credentials and the address of the web ui.

Image Added

Navigate to the web ui and enter in the credentials.

Image Added

Once the credentials are entered, you can create an S3 bucket called "kubedemo"

Image Added

That is all the setup required for minio.

Velero Installation

Firstly we download the required version of the velero client, untar, put the binaries in the appropriate place and delete the tar.gz

wget https://github.com/vmware-tanzu/velero/releases/download/v1.8.0-rc.1/velero-v1.8.0-rc.1-linux-amd64.tar.gz
tar zxf velero-v1.8.0-rc.1-linux-amd64.tar.gz
sudo mv velero-v1.8.0-rc.1-linux-amd64/velero /usr/local/bin/
rm -rf velero*

This completes the installation of the Velero client.

Then run the following to create a credentials file that Velero will use to access minio. Change the credentials accordingly.

cat <<EOF>> minio.credentials
[default]
aws_access_key_id=minioadmin
aws_secret_access_key=minioadmin
EOF

Finally, we can install the Velero components in the cluster with the following. Change the IP of your minio installation accordingly and make sure you provide the correct path to "minio.credentials".

velero install --provider aws --bucket kubedemo \
--secret-file ./minio.credentials \
--backup-location-config region=minio,s3ForcePathStyle=true,s3Url=http://172.17.0.2:9000 \
--plugins velero/velero-plugin-for-aws:v1.4.0-rc1 \
--use-volume-snapshots false --use-restic

Once this executes successfully, Velero is fully installed. If you check your Kubernetes namespaces you should now see one named "velero" with both Velero and Restic pods running. Restic enables us to backup persistent volumes that are stored in our local file system.

Patroni Setup and Installation

Once minio and Velero are set up, we can start to bring up the Patroni installation. Details about Patroni for Postgres and what it does can be found here and in out investigation wiki article. In brief, what Partoni does is make the Postgres distribution High Availability. It sets up a cluster of database instances (number is configurable). It will elect one as the leader  - this election is done in a quick and efficient manner to minimise downtime. Data is synchronised quickly across databases so that they always have the same data. Kubernetes will quickly spin up a new instance if one of the current instances fails.

We have provided a helm chart for Patroni that uses the OOM common helm named templates for some of its operations. In this way the Patroni chart will fit into the OOM repo quite easily. We have provided the Patroni chart and common chart:

These can be used in your local/remote setup. Here we will detail how to set this up in your environment.

Once these charts are pushed to your "local" repo and Velero and minio are running, you should create a new Kubernetes namespace called "patroni". You can do this with

kubectl create ns patroni

Make sure the charts are visible in your repo.

Image Added

All of the Patroni Postgres database instances will be created in that namespace. Next, we can simple install our helm charts in the Patroni namespace.

helm install patroni-velero patroni -n patroni

Once installation runs without error, you should be able to observe several things in your cluster. We look at the pods that have been created:

Image Added

We can take a quick look to see which of these pods has been elected the leader of the cluster currently

Image Added

Here we can see the pod patroni-velero-0 has been elected leader and the others are replicas. We can see what happens if we delete the leader.

kubectl delete po patroni-velero-0 -n patroni

Very quickly we will see that a new leader has been elected and that the deleted pod has been recreated as a replica

Image Added

So, at this stage we know the failover mechanism and leader election is working properly.

Persistent Volumes and Storage

The charts used have been setup to use persistent volumes and claims in the same way that this is set up in OOM. The volumes are mapped to a local drive. In a cluster, this will be the NFS and you may have to configure the chart to point to the NFS but on my local setup I have configured it to point to "/home/postgres". If we look under there we will see the backup directory and volumes for each of the database instances.

Image Added

In addition, you can see both the persistent volumes and the persistent volume claims in the Kubernetes namespace:

Image Added

I have set the capacity of each in the chart configuration to 1G. You can configure this in the Patroni values.yaml.

Add Some Data

The next step in the demo is to add some data to the databases. Once we add some data, we can create a backup.

First, we look at the patroni Kubernetes service so that we know which NodePort we can connect to from outside.

Image Added

The NodePort in this case is 31957. In this case we use Intellij's built-in database client to connect to the database. Once connected, we should see something like this:

Image Added

User is: "postgres". Password is: "tea". We use the interface here to create a new schema - called patroni and a new table called "patroni-db". The SQL to create the table is:

create table patroni."patroni-db"
(id bigint NOT NULL,
  name varchar(255) default NULL,
  phone varchar(100) default NULL,
  email varchar(255) default NULL,
  country varchar(100) default NULL,
  PRIMARY KEY (id)
);

Then we can add some data with the below:

INSERT INTO patroni."patroni-db" (id,name,phone,email,country)
VALUES
    (1, 'Juliet Oneal','(455) 278-4852','et.tristique@hotmail.edu','Austria'),
    (2, 'Yolanda Russo','1-355-528-2157','nisl.elementum@outlook.net','Colombia'),
    (3, 'August Sutton','1-259-524-1472','id.risus@outlook.com','New Zealand'),
    (4, 'Lana Mann','(732) 135-6552','sollicitudin.a@google.com','South Korea'),
    (5, 'Logan Harrison','1-703-292-0386','vulputate.risus@google.net','South Korea');

Once we have confirmed that the data we added is actually in the database, we can begin the Velero backup.

Velero Backup

This process should be very simple as the Velero cli is self-explanatory - we are going to backup the entire "patroni" namespace - this will include persistent volumes and persistent volume claims.

velero backup create patroni-backup --include-namespaces patroni --default-volumes-to-restic

Once this runs, you can view the logs/output to see if backup was successful.

velero backup describe patroni-backup

If you see that it was successful in the output, we can look in the minio "kubedemo" bucket to see the objects there.

Image Added

You should see something like the above - and the backup is complete.

Delete the Patroni Namespace & Create a Velero Restore

To delete the namespace and its' contents run:

helm uninstall patroni-velero -n patroni
kubectl delete ns patroni

Ensure that all pvs and pvcs are deleted.

To create the restore - it is again, a simple command:

velero restore create partoni-restore --from-backup patroni-backup

Check the result of the restore with:

velero restore describe partoni-restore

It should show everything completed successfully. We can check the pods, pvs and pvcs and we should see everything works as expected - including leader election:

kubectl get pods -l spilo-role -L spilo-role -n patroni
kubectl get pv -n patroni
kubectl get pvc -n patroni

We should now check the database to see if our data has been restored. The NodePort will have changed on restart, so we will have to check that again. Once we have that, we connect to the database. Checking the table we created, we should be able to see that the data is restored:

Image Added

This marks an end of the demo. To summarise:

• We ran minio to simulate AWS storage infrastructure
• We installed Velero in our cluster
• We used the charts that were provided to install Patroni in helm/kubernetes
• We added a schema, a table and some data on the Patroni cluster
• We created a Velero backup of the patroni namespace including persistent volumes
• We deleted the namespace and volumes.
• We used velero to restore the namespace and volumes
• We confirmed the data was restored