In this post, we will see how to use Minikube in a more advanced way.
In the previous posts about Kubernetes, we saw how to use Kubectl for deploying some resources in a local cluster.
Now, we will move one step forward.
We will see how to create a multi-node Minikube cluster, how to configure the network with Calico, how to install and fix the Kubernetes Dashboard, and how to deploy MongoDB using a StatefulSet with dynamic storage.
We will deploy the same yml files used in the previous post but, we will create a cluster with three nodes:
1 Controll Plane node
2 Worker nodes
[Creating a multi-node cluster]
By default, Minikube creates only one node.
If we want to simulate a more realistic environment, we can add two more nodes.
We can add the second and the third node with this command:
minikube node add
Now, we can check the nodes with Kubectl:
kubectl get nodes
At this point, we could see that the new nodes are in the NotReady state.
This happens because, in a multi-node cluster, Kubernetes needs a network plugin to manage communication between Pods running on different nodes.
This network plugin is called CNI.
[Installing Calico CNI]
CNI means Container Network Interface.
In a Kubernetes cluster, the CNI is responsible for managing the networking between Pods and nodes.
In this example, we will use Calico, which is one of the most used CNI plugins in Kubernetes environments.
To install Calico, we can run this command:
kubectl apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.28.0/manifests/calico.yaml
After some seconds, we can check again the status of the nodes:
kubectl get nodes
This means that the cluster network is working correctly and Kubernetes can schedule Pods on all the available nodes.
[Installing the Kubernetes Dashboard]
Now, we can install the Kubernetes Dashboard.
The Kubernetes Dashboard is a web interface that allows us to see and manage the resources inside the cluster.
To install it, we can run this command:
kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.7.0/aio/deploy/recommended.yaml
After the installation, we can expose the Dashboard using Kubectl proxy:
kubectl proxy
Then, we can open this URL in the browser:
http://127.0.0.1:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/
However, in a multi-node cluster, we could have a problem.
The Dashboard Pod could be scheduled on a Worker node, while the proxy is running through the Control Plane node.
In some local environments with Docker Desktop, this can generate networking problems such as ‘no route to host’.
In this case, one possible solution is to force the Dashboard Pod to run on the Control Plane node.
[Fixing the Dashboard]
To force the Dashboard to run on the Control Plane node, we can apply a patch to the Dashboard Deployment using a YML file called patch.yml:
spec:
template:
spec:
nodeSelector:
node-role.kubernetes.io/control-plane: ""
tolerations:
- key: "node-role.kubernetes.io/control-plane"
operator: "Exists"
effect: "NoSchedule"
Then, we can apply the patch with this command:
kubectl patch deployment kubernetes-dashboard -n kubernetes-dashboard --patch-file patch.yml
Now, we can check where the Dashboard Pod is running:
kubectl get pods -n kubernetes-dashboard -o wide
The Dashboard Pod should now be running on the ‘minikube’ node, which is the Control Plane node.
Now, we can start the proxy again:
kubectl proxy
And we can open the Dashboard using this URL:
http://127.0.0.1:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/
[Creating a token for the Dashboard]
The Dashboard requires a token to login.
For testing purposes, we can create an admin service account with these commands:
kubectl create serviceaccount admin-user -n kubernetes-dashboard
kubectl create clusterrolebinding admin-user -n kubernetes-dashboard --clusterrole=cluster-admin --serviceaccount=kubernetes-dashboard:admin-user
Finally, with this last command, we will generate the token that we can use to login into the Kubernetes Dashboard:
kubectl create token admin-user -n kubernetes-dashboard --duration=24h
This configuration is useful for a local lab environment.
In a real production environment, we should always configure permissions more carefully and avoid using a full ‘cluster-admin’ role unless it is really necessary.
[MongoDB and storage in a multi-node cluster]
In the previous post, we saw how to deploy MongoDB and Mongo-Express on Kubernetes.
In a single-node cluster, using a local volume can be enough for testing.
But in a multi-node cluster, we need to be more careful.
If a Pod is scheduled on node 1 and the volume exists only on node 2, the Pod will not be able to access the data.
For this reason, in a multi-node scenario, it is better to use a StatefulSet with ‘volumeClaimTemplates’.
With ‘volumeClaimTemplates’, Kubernetes creates a dedicated PersistentVolumeClaim for each replica of the StatefulSet.
This is the YAML file for MongoDB:
[mongodb.yml]
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: mongodb-statefulset
namespace: test-namespace
spec:
serviceName: "mongodb-service"
replicas: 1
selector:
matchLabels:
app: mongodb
template:
metadata:
labels:
app: mongodb
spec:
containers:
- name: mongodb
image: mongo
env:
- name: MONGO_INITDB_ROOT_USERNAME
valueFrom:
secretKeyRef:
name: mongodb-secret
key: mongodb-username
- name: MONGO_INITDB_ROOT_PASSWORD
valueFrom:
secretKeyRef:
name: mongodb-secret
key: mongodb-password
ports:
- containerPort: 27017
volumeMounts:
- name: mongodb-storage
mountPath: /data/db
volumeClaimTemplates:
- metadata:
name: mongodb-storage
spec:
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 1Gi
In this example, we are using only one MongoDB replica.
This is important because MongoDB replication requires a proper MongoDB Replica Set configuration.
So, for this post, we keep the configuration simple and focused on Kubernetes storage.
[Deployment order]
When we deploy MongoDB and Mongo-Express, the order is important.
First of all, we create the namespace:
apiVersion: v1
kind: Namespace
metadata:
name: test-namespace
kubectl apply -f namespace.yml
Then, we create the Secret with the MongoDB credentials:
apiVersion: v1
kind: Secret
metadata:
name: mongodb-secret
namespace: test-namespace
type: Opaque
data:
mongodb-username: YWRtaW4=
mongodb-password: UGFzczEyMw==
kubectl apply -f mongodb-secret.yml
Then, we create the MongoDB Service:
apiVersion: v1
kind: Service
metadata:
name: mongodb-service
namespace: test-namespace
spec:
type: NodePort
ports:
- port: 27017
targetPort: 27017
nodePort: 32000
selector:
app: mongodb
kubectl apply -f mongodb-service.yml
After that, we create the MongoDB StatefulSet:
kubectl apply -f mongodb.yml
Then, we create the Service for Mongo-Express:
apiVersion: v1
kind: Service
metadata:
name: mongodb-ui-service
namespace: test-namespace
spec:
type: NodePort
ports:
- port: 8081
targetPort: 8081
nodePort: 32001
selector:
app: mongodb-ui
kubectl apply -f mongodb-ui-service.yml
Finally, we deploy Mongo-Express:
apiVersion: apps/v1
kind: Deployment
metadata:
name: mongodb-ui-deployment
namespace: test-namespace
spec:
replicas: 2
selector:
matchLabels:
app: mongodb-ui
template:
metadata:
labels:
app: mongodb-ui
spec:
containers:
- name: mongo-express
image: mongo-express
env:
- name: ME_CONFIG_MONGODB_ADMINUSERNAME
valueFrom:
secretKeyRef:
name: mongodb-secret
key: mongodb-username
- name: ME_CONFIG_MONGODB_ADMINPASSWORD
valueFrom:
secretKeyRef:
name: mongodb-secret
key: mongodb-password
- name: ME_CONFIG_MONGODB_SERVER
value: mongodb-service
ports:
- containerPort: 8081
kubectl apply -f mongodb-ui.yml
At this point, MongoDB and Mongo-Express should be running inside our multi-node Kubernetes cluster.
We can check the Pods with this command:
kubectl get pods -n test-namespace -o wide
We can also check the PersistentVolumeClaims with this command:
kubectl get pvc -n test-namespace
[Monitoring the cluster with K9s]
Typing long Kubectl commands all the time can be annoying.
For this reason, we can use K9s.
K9s is a terminal UI that allows us to monitor and manage Kubernetes resources in a simple way.
On MacOs, we can install it with the command:
brew install k9s
and start it with the command:
k9s
With K9s, we can see Pods, Services, Deployments, StatefulSets, logs and many other resources.
For example, we can select a Pod and press ‘l’ to see the logs.
We can also press ‘e’ to edit a resource directly from the terminal.
[Conclusion]
In this post, we saw how to use Minikube in a more advanced way.
We started from a simple local Kubernetes environment and we created a multi-node cluster with one Control Plane node and two Worker nodes.
Then, we installed Calico to manage the networking between nodes.
After that, we installed the Kubernetes Dashboard and fixed a common networking problem that can happen with Docker Desktop.
Finally, we deployed MongoDB using a StatefulSet with dynamic storage and we saw how to monitor the cluster using K9s.
Obviously, Minikube is not a production tool.
In a real on-premises environment, we could use tools such as Kubeadm to create the cluster and storage solutions such as Longhorn to manage persistent volumes.
However, the concepts are the same: nodes, networking, scheduling, storage, StatefulSets and troubleshooting.
Minikube is a perfect tool for learning these concepts locally before moving to a real Kubernetes environment.