Vfrdtyky

I am working through the persistent disks tutorial found here while also creating it as a StatefulSet instead of a deployment.

When I run the yaml file into GKE the database fails to start, looking at the logs it has the following error.

[ERROR] --initialize specified but the data directory has files in it. Aborting.

Is it possible to inspect the volume created to see what is in the directory? Otherwise, what am I doing wrong that is causing the disk to be non empty?

Thanks

apiVersion: v1

kind: PersistentVolumeClaim

metadata:

  name: datalayer-pvc

spec:

  accessModes:

    - ReadWriteOnce

  resources:

    requests:

      storage: 100Gi

---

apiVersion: v1

kind: Service

metadata:

  name: datalayer-svc

  labels:

    app: myapplication

spec:

  ports:

  - port: 80

    name: dbadmin

  clusterIP: None

  selector:

    app: database

---

apiVersion: apps/v1beta2

kind: StatefulSet

metadata:

  name: datalayer

spec:

  selector:

    matchLabels:

      app: myapplication

  serviceName: "datalayer-svc"

  replicas: 1

  template:

    metadata:

      labels:

        app: myapplication

    spec:

      terminationGracePeriodSeconds: 10

      containers:

      - name: database

        image: mysql:5.7.22

        env:

        - name: "MYSQL_ROOT_PASSWORD"

          valueFrom:

            secretKeyRef:

              name: mysql-root-password

              key: password

        - name: "MYSQL_DATABASE"

          value: "appdatabase"

        - name: "MYSQL_USER"

          value: "app_user"

        - name: "MYSQL_PASSWORD"

          value: "app_password"

        ports:

        - containerPort: 3306

          name: mysql

        volumeMounts:

        - name: datalayer-pv

          mountPath: /var/lib/mysql

      volumes:

      - name: datalayer-pv

        persistentVolumeClaim:

          claimName: datalayer-pvc

This issue could be caused by the lost+found directory on the filesystem of the PersistentVolume.

I was able to verify this by adding a k8s.gcr.io/busybox container (in PVC set accessModes: [ReadWriteMany], OR comment out the database container):

- name: init

  image: "k8s.gcr.io/busybox"

  command: ["/bin/sh","-c","ls -l /var/lib/mysql"]

  volumeMounts:

  - name: database

    mountPath: "/var/lib/mysql"

There are a few potential workarounds...

Most preferable is to use a subPath on the volumeMounts object. This uses a subdirectory of the PersistentVolume, which should be empty at creation time, instead of the volume root:

volumeMounts:

- name: database

  mountPath: "/var/lib/mysql"

  subPath: mysql

Less preferable workarounds include:

• Use a one-time container to rm -rf /var/lib/mysql/lost+found (not a great solution, because the directory is managed by the filesystem and is likely to re-appear)


• Use mysql:5 image, and add args: ["--ignore-db-dir=lost+found"] to the container (this option was removed in mysql 8)


• Use mariadb image instead of mysql
  

More details might be available at docker-library/mysql issues: #69 and #186

You would usually see if your volumes were mounted with:

kubectl get pods # gets you all the pods on the default namespace

# and

kubectl describe pod <pod-created-by-your-statefulset>

Then you can these commands to check on your PVs and PVCs

kubectl get pv  # gets all the PVs on the default namespace

kubectl get pvc # same for PVCs

kubectl describe pv <pv-name>

kubectl describe pvc <pvc-name>

Then you can to the GCP console on disks and see if your disks got created: