Worker Nodes
Check the workers' status in Openstack
Check enrolled workers in Doni, from Openstack's CLI
openstack hardware listCheck if their states on both
blazerandk8sworkers areSTEADY:openstack hardware show <UUID>Check which workers are registered in Blazar (Openstack's reservation tool):
(venv) expeca@controller-01:/opt/chi-in-a-box$ openstack reservation device list +--------------------------------------+---------------+-----------+-------------+ | id | device_driver | name | device_type | +--------------------------------------+---------------+-----------+-------------+ | 95b195a2-ec63-45ee-8302-c4116de5d1b2 | k8s | worker-01 | container | | 56ffcf3e-516e-49ab-8494-7b60745350de | k8s | worker-02 | container | | b94c30b0-378a-4ced-867d-464f8c9d1562 | k8s | worker-03 | container | | 9326ba04-bbda-4f08-a407-2f8cbe1eb0a1 | k8s | worker-04 | container | +--------------------------------------+---------------+-----------+-------------+Check a worker's information in Blazar:
(venv) expeca@controller-01:/opt/chi-in-a-box$ openstack reservation device show 95b195a2-ec63-45ee-8302-c4116de5d1b2 +------------------+--------------------------------------+ | Field | Value | +------------------+--------------------------------------+ | created_at | 2023-05-04 13:30:01 | | device_driver | k8s | | device_name | Dell PowerEdge R650xs | | device_type | container | | id | 95b195a2-ec63-45ee-8302-c4116de5d1b2 | | machine_name | worker-01 | | model | PowerEdge | | name | worker-01 | | platform_version | 2 | | reservable | True | | uid | 51997f0d-1f5e-4993-a508-2a7f3affdf20 | | updated_at | | | vendor | Dell | +------------------+--------------------------------------+On Horizon (Openstack's web interface), go to "Reservations"->"Leases"->"Device Calendar", check if all workers are available.
On controller, check if the nodes are enrolled in k8s master node and check their status
kubectl get nodes -o wide kubectl describe nodes worker-01Check if all pods are running healthy:
kubectl get pods --all-namespaces -o wide --field-selector spec.nodeName=worker-01Check the logs of a pod (container) on a node:
kubectl logs -n <namespace> <pod-name>On controller, check the network attachment definitions for this worker:
(venv) expeca@controller-01:/opt/chi-in-a-box$ kubectl get network-attachment-definitions NAME AGE worker-02.ens1f0 3h11m worker-02.ens1f1 3h11m(venv) expeca@controller-01:/opt/chi-in-a-box$ kubectl describe network-attachment-definition worker-02.ens1f1
Renewing k3s certificate
If troubleshooting shows that the k3s certificate for a worker node has expired, then it needs to be renewed. This can be done by restarting the k3s service on that worker node.
Command that can show if a certificate has expired (run on the worker node)
systemctl status k3s-agent.serviceRestarting the k3s service (run on the worker node)
systemctl restart k3s-agent.service
Wait a few minutes to allow the k3s service to restart, then check that the worker node is operating well again
Reserve a worker
On Horizon (Openstack's web interface)
Go to "Reservations"->"Leases"->"Create Lease".
Choose a name for your lease.
Choose the starting date, end data, and/or lease duration.
Go to "Devices" tab, select "Reserve Devices".
From "Resource Properties" drop down lists, choose "machine_name".
Wait on the "Reservations"->"Leases" page, until the lease status turns "ACTIVE" from "PENDING".
Open the new lease page by clicking on the lease name on the "Reservations"->"Leases" page, write down "id" in "Reservation" section.
On Openstack's CLI
TBD
On OpenStack's Python client
TBD
Standalone tests
When you have a reserved a worker, you can run any container on it using zun service.
On Horizon (Openstack's web interface)
Hello world test
Go to "Container"->"Containers"->"Create Container".
Choose an arbitrary name and the image for your container. e.g. "praqma/network-multitool" from "Docker Hub".
Go to "Scheduler Hints" tab, create a custom "Scheduler Hint" with the name "reservation".
On the right side, select it and set its value to the reservation id of your worker lease.
Press "Create" and wait on "Container"->"Containers" page for the created container to turn into "Running" state.
Check the logs of the container.
Check on kubectl at the controller, if the containers are running on the corresponding server:
kubectl get pods --all-namespaces -o wide --field-selector spec.nodeName=worker-01Check the deployment running on the server. Remember to use namesapce -n and pod's name.
kubectl get deployments --all-namespaces -o wide
kubectl describe deployment -n b7ba79d309794d84a7b34c7ee8981e31 zun-6ec0824d-004c-44f6-8f93-865ed1ac84ebCheck the pod running on the server, with logs and status using the following commands. Remember to use namesapce -n and pod's name.
kubectl logs -n b7ba79d309794d84a7b34c7ee8981e31 zun-e6b7c8bc-023d-444f-8e5e-d5145360a532-865dc648c4-zqkdx
kubectl describe pod -n b7ba79d309794d84a7b34c7ee8981e31 zun-e6b7c8bc-023d-444f-8e5e-d5145360a532-865dc648c4-zqkdxCheck zun container logs, there should be no errors:
docker exec -it zun_compute_k8s tail -100 /var/log/kolla/zun/zun-compute.logAttaching Network Interfaces
Simple interface attachment:
Create a VLAN network:
Go to "Network"->"Networks"->"Create Network" to create a network.
Enter a name in the "Network" tab, then in the "Subnet" tab enter the subnet name and address e.g. "192.168.100.0/24".
Create network.
Write down "Segmentation ID" from the network's page. This is the VLAN id on the tenant switch. Check that a new VLAN is created on the tenant switch.
ssh expeca@10.10.1.3
switchf789d1#show vlanCreate a container and attach a network to it:
Create a container simillar to Hello world test, with "praqma/network-multitool".
When creating a container, use "Networks" page to select your desired network.
In the labels page, specify the server's interface and its ip:
networks.1.interface=ens1f1,networks.1.ip=192.168.100.200/24Go to "Scheduler Hints" tab, create a custom "Scheduler Hint" with the name "reservation" and set its value to the reservation id of your worker lease.
Press "Create" and wait on "Container"->"Containers" page for the created container to turn into "Running" state.
When the container is started, use ip a command in the console. You must see 2 interfaces, one from Calico, one from Multus:
4: eth0@if33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UP group default
link/ether 0e:bd:fd:41:46:4e brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 192.168.106.195/32 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::cbd:fdff:fe41:464e/64 scope link
valid_lft forever preferred_lft forever
5: net1@if9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default
link/ether 02:00:00:1a:9e:fc brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 192.168.100.221/24 brd 192.168.100.255 scope global net1
valid_lft forever preferred_lft forever
inet6 fe80::ff:fe1a:9efc/64 scope link
valid_lft forever preferred_lft forever On the tenant switch, check that the chosen interface has become an UNTAGGED member of the VLAN.
ssh expeca@10.10.1.3
switchf789d1#show vlanPing the Virtual Router:
Create a Router and connect it to the new network:
"Network"->"Routers"->"Create Router" to create a router.
Choose the name and select the external network if you want it to be a public network gateway.
Go to "Network"->"Routers"->YOUR-ROUTER->"Interfaces" and add an interface.
Select the new network subnet and enter an ip address for this interface.
After the interface is created, go to the container's console, you should be able to ping the new router's interface.
Setting CPU and memory
Run in privileged mode
Client server tests
One-to-one
One-to-many
Last updated