A How To Guide: Remotely Accessing Minikube Kubernetes on KVM

Note: After writing this article I talked to the maintainer of Minikube and turns out the recommended way to remotely access Minikube is to do SSH tunneling. In fact kubectl port forwarding and everything works fine over the ssh tunnel. For more details visit https://github.com/kubernetes/minikube/issues/4733

If you are like me, you have some older hardware sitting around catching dust. The other day I decided to install a lightweight Kubernetes cluster on an Ubuntu 20 box.

There are a few options when it comes to choosing a lightweight Kubernetes distributions:

• Minikube You will have many virtualization options.
• kind Similar to Minikube but only runs on Docker.
• k3s built for IoT & Edge computing.
• k3d K3d is K3s in Docker.

I’m sure there are even more options. These all sound fantastic. However, most of them run on Docker. The problem with Docker on Linux is that you have to run it as the root user (at least that is true at the time of writing this article.). That exposes you to a lot of security risks.

Out of all these options, Minikube supports many drivers from Virtualbox, KVM to Docker. You can even run Minikube without a driver, skipping the Virtual Machine creation.

The None driver obviously could cause a lot of damage and should be avoided. That leaves us with KVM and Virtualbox.

I have to tell you that you will have a much smoother experience with Virtualbox when it comes to remotely accessing Minikube. But Virtualbox and Minikube were not playing nicely together on my Ubuntu box. There were times that Virtualbox was using 200% CPU while just sitting idle. See here for an example of somebody complaining about this issue.

Our Setup 

Our setup will be the Ubuntu 20 LTS box, pc1 which will host a Minikube cluster and a separate computer called the pc2. We want to use kubectl from pc2 to hit the cluster on pc1 just as if we were using kubectl to hit a Kubernetes cluster on the cloud.

1. Install Minikube, kubectl, and KVM on the host 

Follow the official docs

2. Run Minikube with explicit apiserver ip on the host 

Find your computer’s ip address. Here 192.168.1.121 is the ip of the host that is running minikube and we are giving 8GB memory and 8 CPUs to the kubernetes cluster.

minikube start --vm-driver kvm2 --memory 8112 --cpus 8 --apiserver-ips=192.168.1.121

3. Port Forward 

Note: All these steps were needed because I was using KVM as the Minikube driver. If you are using Virtualbox, it is just a click of the button to add the port forwarding in the network section of the settings.

Unless you understand libvirt, forget about port forwarding via kvm/iptables. I will never get back the hours I wasted trying. I ended up using the good old Nginx for port forwarding.

3.a Install Nginx 

$ sudo apt update
$ sudo apt install nginx

3.b Reverse Proxy via NGINX 

Make NGINX do the reverse proxy. The important part here is that Minikube has its own SSL certificates. So we want NGINX to only pass through the traffic and not deal with SSL and the certs. Fortunately the more recent versions of NGINX allow you to do that via the stream directive.

Unlike the server directive, I could only add the stream directive directly to the nginx.conf

stream {
  server {
      listen 192.168.1.121:51999;
      #TCP traffic will be forwarded to the specified server
      proxy_pass 192.168.39.131:8443;       
  }
}

So in this example the ip of the host of minikube is 192.168.1.121 and the port we want to expose to the world is 51999.

You can find the ip of the Minikube instance in your ~/.kube/config file. The port should be 8443.

Before we do anything, let’s test the Nginx settings:

$ sudo nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

And restart the nginx:

$ sudo systemctl restart nginx

3.c Open the port on the firewall 

My Ubuntu box’s IP address on the local network is 192.168.1.121. We want to let the local network be able to reach this box on port 51999.

We are gonna run:

$ sudo ufw allow from 192.168.1.0/24 to any port 51999

And then run $ sudo ufw status to make sure that UFW (Ubuntu Firewall) is enabled. If not, please enable it UFW! You will probably want to add some rules to allow ssh to the host.

3.d Test from local 

Let’s first of all make sure that we have access to minikube.

$ kubectl get pods
No resources found in default namespace.

In this case I don’t have any pods already running. But we know that kubectl is working fine just from that response. Let’s load the ~/.kube/config and make sure that the Nginx reverse proxy works.

Change the server line to use the host ip and the port we just opened:

apiVersion: v1
clusters:
- cluster:
    certificate-authority: ../.minikube/ca.crt
    # server: https://192.168.39.131:8443  <-- this was the previous value there
    server: https://192.168.1.121:51999
  name: minikube
contexts:
- context:
    cluster: minikube
    user: minikube
  name: minikube
current-context: minikube
kind: Config
preferences: {}
users:
- name: minikube
  user:
    client-certificate: ../.minikube/profiles/minikube/client.crt
    client-key: ../.minikube/profiles/minikube/client.key

Now run $ kubectl get pods again. You should get exact the same response as before.

4. Certs 

2. Create a ~/.minikube_remote on pc2 and run the following commands. This is using scp which assumes the ssh configs are setup for your remote machine to access the host. If you don’t want to use scp, you can just use a USB to copy these files.

mkdir ~/.minikube_remote
cd ~/.minikube_remote
scp pc1:.minikube/ca.crt .
scp pc1:.minikube/profiles/minikube/client.crt .
scp pc1:.minikube/profiles/minikube/client.key .

cat <<EOF > ~/.kube/config
apiVersion: v1
clusters:
- cluster:
    certificate-authority: ../.minikube_remote/ca.crt
    server: https://192.168.1.121:51999
  name: minikube
contexts:
- context:
    cluster: minikube
    user: minikube
  name: minikube
current-context: minikube
kind: Config
preferences: {}
users:
- name: minikube
  user:
    client-certificate: ../.minikube_remote/client.crt
    client-key: ../.minikube_remote/client.key
EOF

5. Remote Access 

Now you should be able to use the kubectl on the pc2 to remote access the minikube on pc1! The best part about this is that kubectl port-forward should work as normal too in case you need to access services in your cluster.

Conclusion 

It is always fun to distribute the load from your machine to other pieces of hardware you may have. If you want to even get more serious, you can deploy a “real” kubernetes cluster on your machines. You may want to use a bare metal hypervisor, learn kops or even use a few old machines.

Have fun!