Update k3s Certificates

This happens every year around May 6th. The connection to the k3s service goes down and Kubectl command won't work. Here is how to fix it:

Make sure that Openstack authentication environment is enabled.

Check if the certificates are one year old in the following folder, also check the status of k3s service if it contains logs like: "x509: certificate has expired or is not yet valid".

systemctl status k3s
sudo ls -l /var/lib/rancher/k3s/server/tls/

For more information, check this issue on k3s repo: https://github.com/k3s-io/k3s/issues/5163

1. Delete the dynamic certificate and restart k3s (choose EXPECA user and password)

sudo rm -f /var/lib/rancher/k3s/server/tls/dynamic-cert.json
systemctl restart k3s

2. Copy and paste the new kubeconfig generated by k3s for kubectl and zun.

sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config 

Test that kubectl is back

kubectl get nodes -o wide
kubectl get svc,deployment,pod -n kube-system

3. Reset k3s secrets on kubectl and restart again by running

kubectl --insecure-skip-tls-verify delete secret -n kube-system k3s-serving
rm -f /var/lib/rancher/k3s/server/tls/dynamic-cert.json
systemctl restart k3s

4. Now copy the new k3s kubeconfig to zun and blazar config folders and restart them

sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config 
sudo cp /etc/rancher/k3s/k3s.yaml /etc/kolla/zun-compute-k8s/kubeconfig.yml
sudo cp /etc/rancher/k3s/k3s.yaml /etc/kolla/blazar-manager/kubeconfig.yml
docker restart zun_compute_k8s
docker restart blazar_manager

NOTE: if there are dangling zun containers left on the workers delete them. First check

kubectl get pods --all-namespaces -o wide | grep "zun"
kubectl delete pod <podname> -n <namespace>