Edit

Worker Nodes

Check the workers' status in Openstack

  • Check enrolled workers in Doni, from Openstack's CLI

    openstack hardware list

  • Check if their states on both blazer and k8s workers are STEADY:

    openstack hardware show <UUID>

  • Check which workers are registered in Blazar (Openstack's reservation tool):

    (venv) expeca@controller-01:/opt/chi-in-a-box$ openstack reservation device list
+--------------------------------------+---------------+-----------+-------------+
| id                                   | device_driver | name      | device_type |
+--------------------------------------+---------------+-----------+-------------+
| 95b195a2-ec63-45ee-8302-c4116de5d1b2 | k8s           | worker-01 | container   |
| 56ffcf3e-516e-49ab-8494-7b60745350de | k8s           | worker-02 | container   |
| b94c30b0-378a-4ced-867d-464f8c9d1562 | k8s           | worker-03 | container   |
| 9326ba04-bbda-4f08-a407-2f8cbe1eb0a1 | k8s           | worker-04 | container   |
+--------------------------------------+---------------+-----------+-------------+

  • Check a worker's information in Blazar:

    (venv) expeca@controller-01:/opt/chi-in-a-box$ openstack reservation device show 95b195a2-ec63-45ee-8302-c4116de5d1b2
+------------------+--------------------------------------+
| Field            | Value                                |
+------------------+--------------------------------------+
| created_at       | 2023-05-04 13:30:01                  |
| device_driver    | k8s                                  |
| device_name      | Dell PowerEdge R650xs                |
| device_type      | container                            |
| id               | 95b195a2-ec63-45ee-8302-c4116de5d1b2 |
| machine_name     | worker-01                            |
| model            | PowerEdge                            |
| name             | worker-01                            |
| platform_version | 2                                    |
| reservable       | True                                 |
| uid              | 51997f0d-1f5e-4993-a508-2a7f3affdf20 |
| updated_at       |                                      |
| vendor           | Dell                                 |
+------------------+--------------------------------------+

  • On Horizon (Openstack's web interface), go to "Reservations"->"Leases"->"Device Calendar", check if all workers are available.

  • On controller, check if the nodes are enrolled in k8s master node and check their status

    kubectl get nodes -o wide
kubectl describe nodes worker-01

    Check if all pods are running healthy:

    kubectl get pods --all-namespaces -o wide --field-selector spec.nodeName=worker-01

    Check the logs of a pod (container) on a node:

    kubectl logs -n <namespace> <pod-name>

  • On controller, check the network attachment definitions for this worker:

    (venv) expeca@controller-01:/opt/chi-in-a-box$ kubectl get network-attachment-definitions
NAME                    AGE
worker-02.ens1f0        3h11m
worker-02.ens1f1        3h11m
    (venv) expeca@controller-01:/opt/chi-in-a-box$ kubectl describe network-attachment-definition worker-02.ens1f1

Renewing k3s certificate

If troubleshooting shows that the k3s certificate for a worker node has expired, then it needs to be renewed. This can be done by restarting the k3s service on that worker node.

  • Command that can show if a certificate has expired (run on the worker node)

  • Restarting the k3s service (run on the worker node)

Wait a few minutes to allow the k3s service to restart, then check that the worker node is operating well again

Reserve a worker

On Horizon (Openstack's web interface)

  • Go to "Reservations"->"Leases"->"Create Lease".

  • Choose a name for your lease.

  • Choose the starting date, end data, and/or lease duration.

  • Go to "Devices" tab, select "Reserve Devices".

  • From "Resource Properties" drop down lists, choose "machine_name".

  • Wait on the "Reservations"->"Leases" page, until the lease status turns "ACTIVE" from "PENDING".

  • Open the new lease page by clicking on the lease name on the "Reservations"->"Leases" page, write down "id" in "Reservation" section.

On Openstack's CLI

TBD

On OpenStack's Python client

TBD

Standalone tests

When you have a reserved a worker, you can run any container on it using zun service.

On Horizon (Openstack's web interface)

Hello world test

  • Go to "Container"->"Containers"->"Create Container".

  • Choose an arbitrary name and the image for your container. e.g. "praqma/network-multitool" from "Docker Hub".

  • Go to "Scheduler Hints" tab, create a custom "Scheduler Hint" with the name "reservation".

  • On the right side, select it and set its value to the reservation id of your worker lease.

  • Press "Create" and wait on "Container"->"Containers" page for the created container to turn into "Running" state.

  • Check the logs of the container.

Check on kubectl at the controller, if the containers are running on the corresponding server:

Check the deployment running on the server. Remember to use namesapce -n and pod's name.

Check the pod running on the server, with logs and status using the following commands. Remember to use namesapce -n and pod's name.

Check zun container logs, there should be no errors:

Attaching Network Interfaces

Simple interface attachment:

Create a VLAN network:

  • Go to "Network"->"Networks"->"Create Network" to create a network.

  • Enter a name in the "Network" tab, then in the "Subnet" tab enter the subnet name and address e.g. "192.168.100.0/24".

  • Create network.

Write down "Segmentation ID" from the network's page. This is the VLAN id on the tenant switch. Check that a new VLAN is created on the tenant switch.

Create a container and attach a network to it:

  • Create a container simillar to Hello world test, with "praqma/network-multitool".

  • When creating a container, use "Networks" page to select your desired network.

  • In the labels page, specify the server's interface and its ip:

  • Go to "Scheduler Hints" tab, create a custom "Scheduler Hint" with the name "reservation" and set its value to the reservation id of your worker lease.

  • Press "Create" and wait on "Container"->"Containers" page for the created container to turn into "Running" state.

When the container is started, use ip a command in the console. You must see 2 interfaces, one from Calico, one from Multus:

On the tenant switch, check that the chosen interface has become an UNTAGGED member of the VLAN.

Ping the Virtual Router:

Create a Router and connect it to the new network:

  • "Network"->"Routers"->"Create Router" to create a router.

  • Choose the name and select the external network if you want it to be a public network gateway.

  • Go to "Network"->"Routers"->YOUR-ROUTER->"Interfaces" and add an interface.

  • Select the new network subnet and enter an ip address for this interface.

After the interface is created, go to the container's console, you should be able to ping the new router's interface.

Setting CPU and memory

Run in privileged mode

Client server tests

One-to-one

One-to-many

PreviousServicesNextSoftware-Defined Radios

Last updated