How to persist pod logs on Minikube

illustrations illustrations illustrations illustrations illustrations illustrations

How to persist pod logs on Minikube

Published on Sep 17, 2024 by Sep Dehpour

TLDR 

TLDR; I had a brilliant idea to write logs to a file on Minikube and have those logs persist between the restarts. I had to jump through some hoops. The main one was realizing that I had to change the folder ownership of where the files will persist within Minikube itself and use hostPath volume type on the pod so that I could persist the logs in Minikube.

Why you don’t need this on a managed k8s cluster 

Normally, on a managed Kubernetes cluster, you write the logs to stdout/stderr. The logs automatically end up in a logs bucket in the case of Google Cloud, and you can view them on Cloud logging. You don’t “lose” the logs. But with Minikube, you will.

Overview of the problem 

  1. Minikube persists data that is written to specific addresses including /data/*
  2. We can use Kubernetes PersistentVolume to mount a volume at /data/logs-data
  3. Minikube will create a /data/logs-data that is owned by root.
  4. If you are running docker as a non-root user, it can’t write to /data/logs-data

The docker image 

In the Dockerfile, we want to explicitly assign a user_id to the user we want to run inside Docker. In this case, we create a nonroot user:

FROM ...
RUN adduser --disabled-password --gecos "" --uid 1000 --shell /bin/bash nonroot 

RUN mkdir -p /var/logs-data && \
    chown nonroot:nonroot /var/logs-data && \
    chmod 755 /var/logs-data
USER nonroot
...
CMD ["sleep", "infinity"]

The key here is assigning an explicit ID to the nonroot user by passing –uid 1000. The nonroot user won’t need a password to log in, hence the --disable-password flag. We won’t be passing name and other information when creating the user so we do --gecos "". While we are at it let’s assign a default shell to the user too: --shell /bin/bash

Minikube and file permissions  

minikube ssh "sudo mkdir -p /data/logs;sudo chown 1000:1000 /data/logs-data"

We create the /data/logs folder inside minikube proactively and assign the user_id 1000 as the owner. Note that we are using the same ID that we explicitly added in the Dockerfile.

Kubernetes yaml file 

We will use a hostPath volume type. A hostPath volume mounts a file or directory from the host node’s filesystem into your Pod. In this case directly from Minikube.

We will explicitly tell Kubernetes to use user_id 1000 for all operations in the docker image by passing fsGroup, runAsGroup, and runAsUser.

Example:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: logs-data
spec:
  accessModes:
    - ReadWriteOnce
  capacity:
    storage: 1Gi
  hostPath:
    path: /data/logs-data  # Host path for Minikube node
  storageClassName: standard
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: logs-data
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  volumeName: logs-data
---
apiVersion: batch/v1
kind: Job
metadata:
  name: myapp
  labels:
    app: myapp
spec:
  template:
    spec:
      securityContext:
        fsGroup: 1000  # Group ID from atlas docker image nonroot user
        runAsGroup: 1000  # Group ID from atlas docker image nonroot user
        runAsUser: 1000  # User ID from atlas docker image nonroot user
      containers:
      - name: main
        image: path_to_image
        command:
          - sleep
          - infinity
        volumeMounts:
          - name: logs-data
            mountPath: /var/logs-data
      restartPolicy: Never
      volumes:
        - name: logs-data
          persistentVolumeClaim:
            claimName: logs-data

When we apply the above yaml file, we can start writing logs to files inside /var/logs-data which will end up inside /data/logs-data on Minikube. These will persist when the pods restart, killed, and even when Minikube restarts!

If you want to know what fsGroup, runAsGroup, and runAsUser are, here is some more material to read:

Bonus material: fsGroup, runAsGroup, and runAsUser 

In Kubernetes, fsGroup, runAsGroup, and runAsUser are part of the securityContext for pods and containers. These fields are used to control the ownership and permissions for processes and files within the containers, helping you run containers with non-root users and ensuring proper access control. Here’s an explanation of each:

1. runAsUser: 

  • Definition: Specifies the user ID (UID) that the container’s process should run as.

  • Purpose: Ensures that processes inside the container run as a non-root user, improving security by avoiding the use of the default root user in the container.

  • Effect: All processes in the container will run as this specified user. If a container image defines a specific user, runAsUser will override it.

  • Example:

    securityContext:
  runAsUser: 1000  # Process will run as user with UID 1000

  • Use Case: If your container image is based on a Linux system, you can create users inside the image with specific UIDs, such as 1000 for a non-root user, and then use runAsUser to run the container process under this user. This is useful for avoiding privilege escalation or preventing containerized processes from running as root.

2. runAsGroup: 

  • Definition: Specifies the group ID (GID) that the container’s process should run as.

  • Purpose: Similar to runAsUser, but it controls the primary group under which the process will run.

  • Effect: All processes in the container will run under this group. If the image specifies a group, runAsGroup will override it.

  • Example:

    securityContext:
  runAsGroup: 1000  # Process will run with GID 1000

  • Use Case: You can ensure that the process in the container has the appropriate group access to certain files or directories. For instance, if your container process needs access to files owned by a particular group (like group 1000), runAsGroup ensures the process has the correct group permissions.

3. fsGroup: 

  • Definition: Specifies the group ID (GID) for file system access control.

  • Purpose: Allows you to set a group that will own the files created by the container and control access to shared volumes. This ensures proper group-based access to shared resources like volumes.

  • Effect:

    • When a volume is mounted inside the container, Kubernetes changes the group ownership of the volume and its files to the fsGroup specified.
    • New files created inside the mounted volumes will have this group ID.
    • This applies only to writable volumes (like PersistentVolume or EmptyDir), not to hostPath volumes.

  • Example:

    securityContext:
  fsGroup: 2000  # Files on mounted volumes will have GID 2000

  • Use Case: When using shared storage (like PersistentVolume), fsGroup helps ensure that files created by different containers or users can be accessed by other containers in the same group. It’s especially useful in scenarios where multiple containers need to write to the same volume but are running under different users.

How They Work Together: 

When you set runAsUser, runAsGroup, and fsGroup, these values control how processes and files are managed inside the container:

  1. runAsUser determines the user ID of the processes running inside the container.
  2. runAsGroup sets the primary group of those processes.
  3. fsGroup sets the group ownership of mounted volumes, ensuring that the container has access to shared files through group permissions.

Example Scenario:

Let’s say you have a container running a process as runAsUser: 1000 and runAsGroup: 1000, with fsGroup: 2000. The container’s process will:

  • Run as user 1000 and group 1000 (i.e., the process will execute as a non-root user and group).
  • Any volumes mounted in the container will have their group ownership changed to 2000 (as specified by fsGroup), ensuring that files written to those volumes are accessible by other containers or users in group 2000.

Conclusion 

This was a few lines of configurations at the end. What took too much time for me to realize was that the hostPath volumes in Kubernetes do not respect the fsGroup, runAsGroup, or runAsUser settings when mounted. The Kubernetes securityContext, including fsGroup, does not change the ownership or permissions of files on hostPath volumes. This is because hostPath volumes directly mount directories from the host node’s filesystem, and Kubernetes does not modify the file ownership or permissions of the host’s file system when doing so.

See Also