PostgreSQL as stateful k8s application - issue with mounted volumes ( in regards to ownership)

I am trying to adopt a stateful k8s PostgreSQL cluster based on this article to the local environment in my company.

EDIT
this is a vmware tanzu cluster, which I do not have not set up myself, so I do not have any further details on the natrue of the cluster itself. I have added a StorageClass which I am referencing to

> kubectl version
Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.8", GitCommit:"5575935422cc1cf5169dfc8847cb587aa47bac5a", GitTreeState:"clean", BuildDate:"2021-06-16T13:00:45Z", GoVersion:"go1.15.13", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.8+vmware.1", GitCommit:"3e397df2f5dadadfa35958ec45c14b0e81abc25f", GitTreeState:"clean", BuildDate:"2021-06-21T16:59:40Z", GoVersion:"go1.15.13", Compiler:"gc", Platform:"linux/amd64"}

PostgreSQL is running as the postgres user, not root. This may be part of my problem.

end EDIT

There is a custom PostgreSQL image which mounts 3 volumes as

• /opt/db/data/postgres/data
• /opt/db/backup/postgres/backups
• /opt/db/backup/postgres/archives

When applying those files (in the order as they are listed below) up the cluster the postgres pod does not spin up and the logs report a problem with access rights.

> kcl logs pod/postgres-stateful-0
starting up postgres docker image:
postgres -D /opt/db/data/postgres/data
+ echo 'starting up postgres docker image:'
+ echo postgres -D /opt/db/data/postgres/data
+ '[' '!' -d /opt/db/data/postgres/data ']'
+ '[' '!' -O /opt/db/data/postgres/data ']'
+ mkdir -p /opt/db/data/postgres/data
+ chmod 700 /opt/db/data/postgres/data
chmod: changing permissions of '/opt/db/data/postgres/data': Operation not permitted

this stems from the docker-entrypoint.sh running on container creation.

the script checks whether the $PGDATA dir (/opt/db/data/postgres/data) exists and whether it is owned my the postgres user. Actually the Dockerfile from the custom image creates this correctly so the mkdir and chmod action should be skipped and the container should be started.

This works when you just run a single pod based on that image.

So I am guessing that mounting the Volums inside the container somehow srews up the ownership and I am wondering how to get around this or in other words define owner and acces rights for the mount paths inside the to be created container.

Can anybody point me to the right direction on how to solve this? I could not even say whether it's the statefulset.yml or the storage.yaml that needs to be adjusted

Image creation

• Dockerfile

ARG REGISTRY=docker-dev-local.intern.net
ARG BASE_IMAGE_REPO=scm
ARG BASE_IMAGE_NAME=debian-bullseye
ARG BASE_IMAGE_TAG=latest

# Second stage - create runtime image
# -----------------------------------
#FROM debian:11 as base
#FROM docker-dev-local.intern.net/scm/debian-bullseye:build-74 as base
FROM $REGISTRY/$BASE_IMAGE_REPO/$BASE_IMAGE_NAME:$BASE_IMAGE_TAG

# Maintainer
# ----------
LABEL org.opencontainers.image.authors="<[email protected]>"

# Build Environment variables, change as needed
# ---------------------------------------------
ARG PG_MAJOR=14
ARG PG_VERSION=14.1
ARG DIST_VERSION=deb11
ARG DVZ_BUILD=dvz1
ENV DVZ_REPO_URL=http://dvzsn-rd1115.dbmon.rz-dvz.cn-mv.de/scb-repo

# Environment variables required for this build (do NOT change)
# ---------------------------------------------
ENV PG_MAJOR=${PG_MAJOR}
ENV PG_VERSION=${PG_VERSION}
ENV PGUSER=postgres
ENV PGDATABASE=postgres
ENV PGPORT=5432
ENV DBBASE=/opt/db
ENV PGBASE=$DBBASE/postgres
ENV PGBIN=$PGBASE/bin
ENV PGHOME=$PGBASE/postgresql
ENV PGDATA=$DBBASE/data/postgres/data
ENV PGLOG=$PGDATA/log
ENV PGBACK=$DBBASE/backup/postgres/backups
ENV PGARCH=$DBBASE/backup/postgres/archives

ENV PATH=$PGHOME/bin:$PATH

ENV LANG=de_DE.UTF-8
ENV LC_MESSAGES=en_US.UTF-8
ENV TZ=Europe/Berlin

RUN env | sort

# Install additional packages and dependencies
# --------------------------------------------
RUN set -ex; \
    apt-get update && \
    apt-get upgrade && \
    apt-get install -y --no-install-recommends \
        ca-certificates \
        curl \
        dirmngr \
        gnupg \
        iproute2 \
        less \
        libnss-wrapper \
        libpam0g \
        libreadline8 \
        libselinux1 \
        libsystemd0 \
        libxml2 \
        locales \
        openssl \
        procps \
        vim-tiny \
        wget \
        xz-utils \
        zlib1g \
    && \
    apt-get clean

# create locales for en_US and de_DE
RUN localedef -i en_US -f UTF-8 en_US.UTF-8 && \
    localedef -i de_DE -f UTF-8 de_DE.UTF-8 && \
    locale -a

# Set up user and directories
# ---------------------------
RUN mkdir -p $PGBASE $PGBIN $PGDATA $PGBACK $PGARCH && \
    useradd -d /home/postgres -m -s /bin/bash --no-log-init postgres && \
    chown -R postgres:postgres $PGBASE $PGDATA $PGBACK $PGARCH $DBBASE/data && \
    chmod a+xr $PGBASE

# set up user env
# ---------------
USER postgres
COPY --chown=postgres:postgres ["files/.alias", "files/.bashrc", "files/postgresql.conf.${PG_MAJOR}", "files/conf.d/00-ina-default.conf", "/hom
COPY ["files/docker-entrypoint.sh", "/"]
ADD ["files/pg-docker-env.tar.gz", "$PGBASE/"]

# install postgres
# --------------------
# copy postgres package from builder stage
#RUN mkdir -p $PGBASE/postgresql-$PG_VERSION-$DIST_VERSION-$DVZ_BUILD
#COPY --from=build --chown=postgres:postgres ["$PGBASE/postgresql-$PG_VERSION-$DIST_VERSION-$DVZ_BUILD", "$PGBASE/postgresql-$PG_VERSION-$DIST_
# download build of postgres
WORKDIR $PGBASE
RUN curl -sSL $DVZ_REPO_URL/postgres/Linux/$DIST_VERSION/postgresql-$PG_VERSION-$DIST_VERSION-dvz1.tar.gz | tar xzf - -C $PGBASE
RUN ln -s $PGBASE/postgresql-$PG_VERSION-$DIST_VERSION-$DVZ_BUILD postgresql

# bindings
# --------
VOLUME ["$PGDATA", "$PGBACK", "$PGARCH"]
STOPSIGNAL SIGINT
EXPOSE 5432
HEALTHCHECK --interval=1m --start-period=5m \
   CMD pg_ctl status >/dev/null || exit 1

# Define default command to start Database.
ENTRYPOINT ["/docker-entrypoint.sh"]
CMD ["postgres", "-D", "/opt/db/data/postgres/data"]

• docker-entrypoint.sh

#!/bin/bash
set -xeEuo pipefail

echo "starting up postgres docker image:"
echo "$@"

# check PGDATA directory and create if necessary
if [ \! -d $PGDATA ] || [ \! -O $PGDATA ]
then
    mkdir -p $PGDATA
    chmod 700 $PGDATA
fi

# check database cluster in PGDATA directory and create new db cluster if necessary
if [ \! -s $PGDATA/PG_VERSION ] || ! pg_controldata
then
    POSTGRES_PASSWORD=${POSTGRES_PASSWORD:-"Start1234"}
    initdb -D $PGDATA --locale=de_DE.UTF-8 --lc-messages=en_US.UTF-8 --auth-local=trust --auth-host=md5 --pwfile=<(echo "$POSTGRES_PASSWORD")
    mv $PGDATA/postgresql.conf $PGDATA/postgresql.conf.orig
    cp ~/postgresql.conf.${PG_MAJOR} $PGDATA/postgresql.conf
    mkdir -p $PGDATA/conf.d
    cp ~/00-ina-default.conf $PGDATA/conf.d/
    {
        echo "# allow connections via docker gateway or bridge"
        echo "host    all             all             172.16.0.0/14           md5"
    } >> "$PGDATA/pg_hba.conf"
fi

# show PGDATA version and controldata
echo "PGDATA/PGVERSION=`cat $PGDATA/PG_VERSION`"

# start postgres rdbms now
exec "$@"

kubernetes declarations

• storage.yml

kind: PersistentVolume
apiVersion: v1
metadata:
  name: postgres-pgdata33
  labels:
    app: postgres
    type: local
spec:
  storageClassName: ina01
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/var/data"
---
kind: PersistentVolume
apiVersion: v1
metadata:
  name: postgres-pgbackup33
  labels:
    app: postgres
    type: local
spec:
  storageClassName: ina01
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  hostPath:    path: "/var/data"
---
kind: PersistentVolume
apiVersion: v1
metadata:
  name: postgres-pgarch33
  labels:
    app: postgres
    type: local
spec:
  storageClassName: ina01
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/var/data"
# #####################################################################################
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: pgdata33-pvc
  labels:
    app: postgres
spec:
  storageClassName: ina01
  capacity:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: pgbackup33-pvc
  labels:
    app: postgres
spec:
  storageClassName: ina01
  capacity:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: pgarch33-pvc
  labels:
    app: postgres
spec:
  storageClassName: ina01
  capacity:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi

• configmap.yml

apiVersion: v1
kind: ConfigMap
metadata:
  name: postgres-configuration
  labels:
    app: postgres
data:
  POSTGRES_DB: awesomedb
  POSTGRES_USER: amazinguser
  POSTGRES_PASSWORD: perfectpassword

• statefulset.yml

---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres-stateful
  labels:
    app: postgres
spec:
  serviceName: "postgres"
  replicas: 1
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
      - name: postgres
        image: docker-dev-local.intern.net/ina/postgresql:14.1-scm-debian-bullseye-build-74-4
        envFrom:
        - configMapRef:
            name: postgres-configuration
        ports:
        - containerPort: 5432
          name: postgresdb
        volumeMounts:
        - name: pv-data
          mountPath: /opt/db/data/postgres/data   # /var/lib/postgresql/data
        - name: pv-backup
          mountPath: /opt/db/backup/postgres
        - name: pv-arch
          mountPath: /opt/db/backup/postgres/arch
      securityContext:
        runAsUser: 1000
        runAsGroup: 1000
        fsGroup: 1000
      volumes:
      - name: pv-data
        persistentVolumeClaim:
          claimName: pgdata33-pvc
      - name: pv-backup
        persistentVolumeClaim:
          claimName: pgbackup33-pvc
      - name: pv-arch
        persistentVolumeClaim:
          claimName: pgarch33-pvc

• service.yml

apiVersion: v1
kind: Service
metadata:
  name: postgres-service
  labels:
    app: postgres
spec:
  ports:
  - port: 5432
    name: postgres
  type: NodePort
  selector:
    app: postgres