User Guide

Learn how to work with the Postgres Operator in a Kubernetes (K8s) environment.

Create a manifest for a new PostgreSQL cluster

Make sure you have set up the operator. Then you can create a new Postgres cluster by applying manifest like this minimal example:

apiVersion: ""
kind: postgresql
  name: acid-minimal-cluster
  teamId: "acid"
    size: 1Gi
  numberOfInstances: 2
    # database owner
    - superuser
    - createdb

    # role for application foo
    foo_user: # or 'foo_user: []'

  #databases: name->owner
    foo: zalando
    version: "11"

Once you cloned the Postgres Operator repository you can find this example also in the manifests folder:

kubectl create -f manifests/minimal-postgres-manifest.yaml

Make sure, the spec section of the manifest contains at least a teamId, the numberOfInstances and the postgresql object with the version specified. The minimum volume size to run the postgresql resource on Elastic Block Storage (EBS) is 1Gi.

Note, that the name of the cluster must start with the teamId and -. At Zalando we use team IDs (nicknames) to lower the chance of duplicate cluster names and colliding entities. The team ID would also be used to query an API to get all members of a team and create database roles for them.

Watch pods being created

kubectl get pods -w --show-labels

Connect to PostgreSQL

With a port-forward on one of the database pods (e.g. the master) you can connect to the PostgreSQL database. Use labels to filter for the master pod of our test cluster.

# get name of master pod of acid-minimal-cluster
export PGMASTER=$(kubectl get pods -o jsonpath={} -l application=spilo,version=acid-minimal-cluster,spilo-role=master)

# set up port forward
kubectl port-forward $PGMASTER 6432:5432

Open another CLI and connect to the database. Use the generated secret of the postgres robot user to connect to our acid-minimal-cluster master running in Minikube. As non-encrypted connections are rejected by default set the SSL mode to require:

export PGPASSWORD=$(kubectl get secret postgres.acid-minimal-cluster.credentials -o 'jsonpath={.data.password}' | base64 -d)
export PGSSLMODE=require
psql -U postgres -p 6432

Defining database roles in the operator

Postgres Operator allows defining roles to be created in the resulting database cluster. It covers three use-cases:

  • manifest roles: create application roles specific to the cluster described in the manifest.
  • infrastructure roles: create application roles that should be automatically created on every cluster managed by the operator.
  • teams API roles: automatically create users for every member of the team owning the database cluster.

In the next sections, we will cover those use cases in more details.

Manifest roles

Manifest roles are defined directly in the cluster manifest. See minimal postgres manifest for an example of zalando role, defined with superuser and createdb flags.

Manifest roles are defined as a dictionary, with a role name as a key and a list of role options as a value. For a role without any options it is best to supply the empty list []. It is also possible to leave this field empty as in our example manifests. In certain cases such empty field may be missing later removed by K8s due to the null value it gets (foobar_user: is equivalent to foobar_user: null).

The operator accepts the following options: superuser, inherit, login, nologin, createrole, createdb, replication, bypassrls.

By default, manifest roles are login roles (aka users), unless nologin is specified explicitly.

The operator automatically generates a password for each manifest role and places it in the secret named {username}.{team}-{clustername} in the same namespace as the cluster. This way, the application running in the K8s cluster and connecting to Postgres can obtain the password right from the secret, without ever sharing it outside of the cluster.

At the moment it is not possible to define membership of the manifest role in other roles.

Infrastructure roles

An infrastructure role is a role that should be present on every PostgreSQL cluster managed by the operator. An example of such a role is a monitoring user. There are two ways to define them:

  • With the infrastructure roles secret only
  • With both the the secret and the infrastructure role ConfigMap.

Infrastructure roles secret

The infrastructure roles secret is specified by the infrastructure_roles_secret_name parameter. The role definition looks like this (values are base64 encoded):

user1: ZGJ1c2Vy
password1: c2VjcmV0
inrole1: b3BlcmF0b3I=

The block above describes the infrastructure role 'dbuser' with password 'secret' that is a member of the 'operator' role. For the following definitions one must increase the index, i.e. the next role will be defined as 'user2' and so on. The resulting role will automatically be a login role.

Note that with definitions that solely use the infrastructure roles secret there is no way to specify role options (like superuser or nologin) or role memberships. This is where the ConfigMap comes into play.

Secret plus ConfigMap

A ConfigMap allows for defining more details regarding the infrastructure roles. Therefore, one should use the new style that specifies infrastructure roles using both the secret and a ConfigMap. The ConfigMap must have the same name as the secret. The secret should contain an entry with 'rolename:rolepassword' for each role.

dbuser: c2VjcmV0

And the role description for that user should be specified in the ConfigMap.

  dbuser: |
    inrole: [operator, admin]  # following roles will be assigned to the new user
    - createdb
    db_parameters:  # db parameters, applied for this particular user
      log_statement: all

One can allow membership in multiple roles via the inrole array parameter, define role flags via the user_flags list and supply per-role options through the db_parameters dictionary. All those parameters are optional.

Both definitions can be mixed in the infrastructure role secret, as long as your new-style definition can be clearly distinguished from the old-style one (for instance, do not name new-style roles userN).

Since an infrastructure role is created uniformly on all clusters managed by the operator, it makes no sense to define it without the password. Such definitions will be ignored with a prior warning.

See infrastructure roles secret and infrastructure roles configmap for the examples.

Teams API roles

These roles are meant for database activity of human users. It's possible to configure the operator to automatically create database roles for lets say all employees of one team. They are not listed in the manifest and there are no K8s secrets created for them. Instead they would use an OAuth2 token to connect. To get all members of the team the operator queries a defined API endpoint that returns usernames. A minimal Teams API should work like this:

/.../<teamname> -> ["name","anothername"]

A "fake" Teams API deployment is provided in the manifests folder to set up a basic API around whatever services is used for user management. The Teams API's URL is set in the operator's configuration and enable_teams_api must be set to true. There are more settings available to choose superusers, group roles, PAM configuration etc. An OAuth2 token can be passed to the Teams API via a secret. The name for this secret is configurable with the oauth_token_secret_name parameter.

Resource definition

The compute resources to be used for the Postgres containers in the pods can be specified in the postgresql cluster manifest.

      cpu: 10m
      memory: 100Mi
      cpu: 300m
      memory: 300Mi

The minimum limit to properly run the postgresql resource is 256m for cpu and 256Mi for memory. If a lower value is set in the manifest the operator will cancel ADD or UPDATE events on this resource with an error. If no resources are defined in the manifest the operator will obtain the configured default requests.

Use taints and tolerations for dedicated PostgreSQL nodes

To ensure Postgres pods are running on nodes without any other application pods, you can use taints and tolerations and configure the required toleration in the manifest.

  - key: postgres
    operator: Exists
    effect: NoSchedule

How to clone an existing PostgreSQL cluster

You can spin up a new cluster as a clone of the existing one, using a clone section in the spec. There are two options here:

  • Clone directly from a source cluster using pg_basebackup
  • Clone from an S3 bucket

Clone directly

    cluster: "acid-batman"

Here cluster is a name of a source cluster that is going to be cloned. The cluster to clone is assumed to be running and the clone procedure invokes pg_basebackup from it. The operator will setup the cluster to be cloned to connect to the service of the source cluster by name (if the cluster is called test, then the connection string will look like host=test port=5432), which means that you can clone only from clusters within the same namespace.

Clone from S3

    uid: "efd12e58-5786-11e8-b5a7-06148230260c"
    cluster: "acid-batman"
    timestamp: "2017-12-19T12:40:33+01:00"

Here cluster is a name of a source cluster that is going to be cloned. A new cluster will be cloned from S3, using the latest backup before the timestamp. In this case, uid field is also mandatory - operator will use it to find a correct key inside an S3 bucket. You can find this field in the metadata of the source cluster:

kind: postgresql
  name: acid-test-cluster
  uid: efd12e58-5786-11e8-b5a7-06148230260c

Note that timezone is required for timestamp. Otherwise, offset is relative to UTC, see RFC 3339 section 5.6) 3339 section 5.6.

For non AWS S3 following settings can be set to support cloning from other S3 implementations:

    uid: "efd12e58-5786-11e8-b5a7-06148230260c"
    cluster: "acid-batman"
    timestamp: "2017-12-19T12:40:33+01:00"
    s3_access_key_id: 0123456789abcdef0123456789abcdef
    s3_secret_access_key: 0123456789abcdef0123456789abcdef
    s3_force_path_style: true

Setting up a standby cluster

Standby clusters are like normal cluster but they are streaming from a remote cluster. As the first version of this feature, the only scenario covered by operator is to stream from a WAL archive of the master. Following the more popular infrastructure of using Amazon's S3 buckets, it is mentioned as s3_wal_path here. To start a cluster as standby add the following standby section in the YAML file:

    s3_wal_path: "s3 bucket path to the master"

Things to note:

  • An empty string in the s3_wal_path field of the standby cluster will result in an error and no statefulset will be created.
  • Only one pod can be deployed for stand-by cluster.
  • To manually promote the standby_cluster, use patronictl and remove config entry.
  • There is no way to transform a non-standby cluster to a standby cluster through the operator. Adding the standby section to the manifest of a running Postgres cluster will have no effect. However, it can be done through Patroni by adding the standby_cluster section using patronictl edit-config. Note that the transformed standby cluster will not be doing any streaming. It will be in standby mode and allow read-only transactions only.

Sidecar Support

Each cluster can specify arbitrary sidecars to run. These containers could be used for log aggregation, monitoring, backups or other tasks. A sidecar can be specified like this:

    - name: "container-name"
      image: "company/image:tag"
          cpu: 500m
          memory: 500Mi
          cpu: 100m
          memory: 100Mi
        - name: "ENV_VAR_NAME"
          value: "any-k8s-env-things"

In addition to any environment variables you specify, the following environment variables are always passed to sidecars:

  • POD_NAME - field reference to
  • POD_NAMESPACE - field reference to metadata.namespace
  • POSTGRES_USER - the superuser that can be used to connect to the database
  • POSTGRES_PASSWORD - the password for the superuser

The PostgreSQL volume is shared with sidecars and is mounted at /home/postgres/pgdata.

Note: The operator will not create a cluster if sidecar containers are specified but globally disabled in the configuration. The enable_sidecars option must be set to true.

InitContainers Support

Each cluster can specify arbitrary init containers to run. These containers can be used to run custom actions before any normal and sidecar containers start. An init container can be specified like this:

    - name: "container-name"
      image: "company/image:tag"
        - name: "ENV_VAR_NAME"
          value: "any-k8s-env-things"

initContainers accepts full v1.Container definition.

Note: The operator will not create a cluster if initContainers are specified but globally disabled in the configuration. The enable_init_containers option must be set to true.

Increase volume size

Postgres operator supports statefulset volume resize if you're using the operator on top of AWS. For that you need to change the size field of the volume description in the cluster manifest and apply the change:

    size: 5Gi # new volume size

The operator compares the new value of the size field with the previous one and acts on differences.

You can only enlarge the volume with the process described above, shrinking is not supported and will emit a warning. After this update all the new volumes in the statefulset are allocated according to the new size. To enlarge persistent volumes attached to the running pods, the operator performs the following actions:

  • call AWS API to change the volume size

  • connect to pod using kubectl exec and resize filesystem with resize2fs.

Fist step has a limitation, AWS rate-limits this operation to no more than once every 6 hours. Note, that if the statefulset is scaled down before resizing the new size is only applied to the volumes attached to the running pods. The size of volumes that correspond to the previously running pods is not changed.

Logical backups

You can enable logical backups from the cluster manifest by adding the following parameter in the spec section:

  enableLogicalBackup: true

The operator will create and sync a K8s cron job to do periodic logical backups of this particular Postgres cluster. Due to the limitation of K8s cron jobs it is highly advisable to set up additional monitoring for this feature; such monitoring is outside the scope of operator responsibilities. See configuration reference and administrator documentation for details on how backups are executed.