Kubernetes Cluster

Check Kubernetes

Check if all apps and pods are ready.

expeca@controller-01:/opt/chi-in-a-box$ kubectl get svc,deployment,pod -n kube-system
NAME                     TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)                  AGE
service/kube-dns         ClusterIP   10.43.0.10     <none>        53/UDP,53/TCP,9153/TCP   37h
service/metrics-server   ClusterIP   10.43.31.114   <none>        443/TCP                  37h

NAME                                     READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/metrics-server           1/1     1            1           37h
deployment.apps/local-path-provisioner   1/1     1            1           37h
deployment.apps/coredns                  1/1     1            1           37h

NAME                                         READY   STATUS    RESTARTS   AGE
pod/kube-multus-ds-kl2tk                     1/1     Running   0          37h
pod/metrics-server-9cf544f65-t86w5           1/1     Running   0          37h
pod/local-path-provisioner-64ffb68fd-qv5m4   1/1     Running   0          37h
pod/coredns-85cb69466-4glrk                  1/1     Running   0          37h

Commands to check the nodes

kubectl get nodes -o wide
kubectl describe nodes worker-01

Check if all services on a node are running healthy

kubectl get pods --all-namespaces -o wide --field-selector spec.nodeName=worker-01
kubectl logs -n <namespace> <pod-name>

Check network attachment definitions

kubectl get network-attachment-definitions
kubectl describe network-attachment-definition worker-02.ens1f1

Check K8S config file on the controller

cat $HOME/.kube/config

How to completely remove the K3S Server

Run the following on the controller to completly uninstall the k8s master:

#!/bin/sh
set -x
systemctl stop k3s
systemctl disable k3s
systemctl daemon-reload
rm -f /etc/systemd/system/k3s.service
rm -f /usr/local/bin/k3s
if [ -L /usr/local/bin/kubectl ]; then
    rm -f /usr/local/bin/kubectl
fi
if [ -L /usr/local/bin/crictl ]; then
    rm -f /usr/local/bin/crictl
fi
if [ -e /sys/fs/cgroup/systemd/system.slice/k3s.service/cgroup.procs ]; then
    kill -9 `cat /sys/fs/cgroup/systemd/system.slice/k3s.service/cgroup.procs`
fi
umount `cat /proc/self/mounts | awk '{print $2}' | grep '^/run/k3s'`
umount `cat /proc/self/mounts | awk '{print $2}' | grep '^/var/lib/rancher/k3s'`

rm -rf /var/lib/rancher/k3s
rm -rf /etc/rancher/k3s

rm -f /usr/local/bin/k3s-uninstall.sh

To install k8s master again, just use kolla:

./cc-ansible --site ../site-config/ deploy --tags k3s

NOTE: After bringing up a new K3S master, you have to re-deploy blazar,zun, and doni so they copy over the new kubeconfig file.

./cc-ansible --site ../site-config/ deploy --tags blazar
./cc-ansible --site ../site-config/ deploy --tags doni
./cc-ansible --site ../site-config/ deploy --tags zun