Worker Nodes

Check the workers' status in Openstack

  • Check enrolled workers in Doni, from Openstack's CLI

    openstack hardware list

  • Check if their states on both blazer and k8s workers are STEADY:

    openstack hardware show <UUID>

  • Check which workers are registered in Blazar (Openstack's reservation tool):

    (venv) expeca@controller-01:/opt/chi-in-a-box$ openstack reservation device list
+--------------------------------------+---------------+-----------+-------------+
| id                                   | device_driver | name      | device_type |
+--------------------------------------+---------------+-----------+-------------+
| 95b195a2-ec63-45ee-8302-c4116de5d1b2 | k8s           | worker-01 | container   |
| 56ffcf3e-516e-49ab-8494-7b60745350de | k8s           | worker-02 | container   |
| b94c30b0-378a-4ced-867d-464f8c9d1562 | k8s           | worker-03 | container   |
| 9326ba04-bbda-4f08-a407-2f8cbe1eb0a1 | k8s           | worker-04 | container   |
+--------------------------------------+---------------+-----------+-------------+

  • Check a worker's information in Blazar:

    (venv) expeca@controller-01:/opt/chi-in-a-box$ openstack reservation device show 95b195a2-ec63-45ee-8302-c4116de5d1b2
+------------------+--------------------------------------+
| Field            | Value                                |
+------------------+--------------------------------------+
| created_at       | 2023-05-04 13:30:01                  |
| device_driver    | k8s                                  |
| device_name      | Dell PowerEdge R650xs                |
| device_type      | container                            |
| id               | 95b195a2-ec63-45ee-8302-c4116de5d1b2 |
| machine_name     | worker-01                            |
| model            | PowerEdge                            |
| name             | worker-01                            |
| platform_version | 2                                    |
| reservable       | True                                 |
| uid              | 51997f0d-1f5e-4993-a508-2a7f3affdf20 |
| updated_at       |                                      |
| vendor           | Dell                                 |
+------------------+--------------------------------------+

  • On Horizon (Openstack's web interface), go to "Reservations"->"Leases"->"Device Calendar", check if all workers are available.

  • On controller, check if the nodes are enrolled in k8s master node and check their status

    kubectl get nodes -o wide
kubectl describe nodes worker-01

    Check if all pods are running healthy:

    kubectl get pods --all-namespaces -o wide --field-selector spec.nodeName=worker-01

    Check the logs of a pod (container) on a node:

    kubectl logs -n <namespace> <pod-name>

  • On controller, check the network attachment definitions for this worker:

    (venv) expeca@controller-01:/opt/chi-in-a-box$ kubectl get network-attachment-definitions
NAME                    AGE
worker-02.ens1f0        3h11m
worker-02.ens1f1        3h11m
    (venv) expeca@controller-01:/opt/chi-in-a-box$ kubectl describe network-attachment-definition worker-02.ens1f1

Renewing k3s certificate

If troubleshooting shows that the k3s certificate for a worker node has expired, then it needs to be renewed. This can be done by restarting the k3s service on that worker node.

  • Command that can show if a certificate has expired (run on the worker node)

    systemctl status k3s-agent.service

  • Restarting the k3s service (run on the worker node)

    systemctl restart k3s-agent.service

Wait a few minutes to allow the k3s service to restart, then check that the worker node is operating well again

Reserve a worker

On Horizon (Openstack's web interface)

  • Go to "Reservations"->"Leases"->"Create Lease".

  • Choose a name for your lease.

  • Choose the starting date, end data, and/or lease duration.

  • Go to "Devices" tab, select "Reserve Devices".

  • From "Resource Properties" drop down lists, choose "machine_name".

  • Wait on the "Reservations"->"Leases" page, until the lease status turns "ACTIVE" from "PENDING".

  • Open the new lease page by clicking on the lease name on the "Reservations"->"Leases" page, write down "id" in "Reservation" section.

On Openstack's CLI

TBD

On OpenStack's Python client

TBD

Standalone tests

When you have a reserved a worker, you can run any container on it using zun service.

On Horizon (Openstack's web interface)

Hello world test

  • Go to "Container"->"Containers"->"Create Container".

  • Choose an arbitrary name and the image for your container. e.g. "praqma/network-multitool" from "Docker Hub".

  • Go to "Scheduler Hints" tab, create a custom "Scheduler Hint" with the name "reservation".

  • On the right side, select it and set its value to the reservation id of your worker lease.

  • Press "Create" and wait on "Container"->"Containers" page for the created container to turn into "Running" state.

  • Check the logs of the container.

Check on kubectl at the controller, if the containers are running on the corresponding server:

kubectl get pods --all-namespaces -o wide --field-selector spec.nodeName=worker-01

Check the deployment running on the server. Remember to use namesapce -n and pod's name.

kubectl get deployments --all-namespaces -o wide
kubectl describe deployment -n b7ba79d309794d84a7b34c7ee8981e31 zun-6ec0824d-004c-44f6-8f93-865ed1ac84eb

Check the pod running on the server, with logs and status using the following commands. Remember to use namesapce -n and pod's name.

kubectl logs -n b7ba79d309794d84a7b34c7ee8981e31 zun-e6b7c8bc-023d-444f-8e5e-d5145360a532-865dc648c4-zqkdx
kubectl describe pod -n b7ba79d309794d84a7b34c7ee8981e31 zun-e6b7c8bc-023d-444f-8e5e-d5145360a532-865dc648c4-zqkdx

Check zun container logs, there should be no errors:

docker exec -it zun_compute_k8s tail -100 /var/log/kolla/zun/zun-compute.log

Attaching Network Interfaces

Simple interface attachment:

Create a VLAN network:

  • Go to "Network"->"Networks"->"Create Network" to create a network.

  • Enter a name in the "Network" tab, then in the "Subnet" tab enter the subnet name and address e.g. "192.168.100.0/24".

  • Create network.

Write down "Segmentation ID" from the network's page. This is the VLAN id on the tenant switch. Check that a new VLAN is created on the tenant switch.

ssh expeca@10.10.1.3
switchf789d1#show vlan

Create a container and attach a network to it:

  • Create a container simillar to Hello world test, with "praqma/network-multitool".

  • When creating a container, use "Networks" page to select your desired network.

  • In the labels page, specify the server's interface and its ip:

    networks.1.interface=ens1f1,networks.1.ip=192.168.100.200/24

  • Go to "Scheduler Hints" tab, create a custom "Scheduler Hint" with the name "reservation" and set its value to the reservation id of your worker lease.

  • Press "Create" and wait on "Container"->"Containers" page for the created container to turn into "Running" state.

When the container is started, use ip a command in the console. You must see 2 interfaces, one from Calico, one from Multus:

4: eth0@if33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UP group default                                   
    link/ether 0e:bd:fd:41:46:4e brd ff:ff:ff:ff:ff:ff link-netnsid 0                                                           
    inet 192.168.106.195/32 scope global eth0                                                                                   
       valid_lft forever preferred_lft forever                                                                                  
    inet6 fe80::cbd:fdff:fe41:464e/64 scope link                                                                                
       valid_lft forever preferred_lft forever                                                                                  
5: net1@if9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default                                    
    link/ether 02:00:00:1a:9e:fc brd ff:ff:ff:ff:ff:ff link-netnsid 0                                                           
    inet 192.168.100.221/24 brd 192.168.100.255 scope global net1                                                               
       valid_lft forever preferred_lft forever                                                                                  
    inet6 fe80::ff:fe1a:9efc/64 scope link                                                                                      
       valid_lft forever preferred_lft forever

On the tenant switch, check that the chosen interface has become an UNTAGGED member of the VLAN.

ssh expeca@10.10.1.3
switchf789d1#show vlan

Ping the Virtual Router:

Create a Router and connect it to the new network:

  • "Network"->"Routers"->"Create Router" to create a router.

  • Choose the name and select the external network if you want it to be a public network gateway.

  • Go to "Network"->"Routers"->YOUR-ROUTER->"Interfaces" and add an interface.

  • Select the new network subnet and enter an ip address for this interface.

After the interface is created, go to the container's console, you should be able to ping the new router's interface.

Setting CPU and memory

Run in privileged mode

Client server tests

One-to-one

One-to-many

PreviousServicesNextSoftware-Defined Radios

Last updated