> For the complete documentation index, see [llms.txt](https://kth-expeca.gitbook.io/testbedconfig/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://kth-expeca.gitbook.io/testbedconfig/deploy/monitoring.md).

# Deploy Monitoring Services

## Update Prometheus

Prometheus is the Time Series DataBase handler that is at the core of our monitoring solution. It executes as a docker container service and is included with the chi-in-a-box system in the controller node. It periodically fetches metrics from exporter applications via HTTP scraping. Grafana is used then to query metrics from Prometheus for display (dashboards).

1. Upload Prometheus config files to controller

   ```
   ssh controller
   mkdir exporter
   cd exporter
   ```

   Upload following files to the exporter directory:

   * config.json
   * prometheus.yml

   The files can be found at <https://github.com/KTH-EXPECA/TestbedConfig/tree/master/chi-site-config/monitoring>
2. Copy the config files to correct Prometheus directory:

   ```
   sudo cp config.json /etc/kolla/prometheus-server
   sudo cp prometheus.yml /etc/kolla/prometheus-server
   sudo ls -l /etc/kolla/prometheus-server
   ```

   Verify that owner of the target files are root:root
3. Restart the Prometheus server

   ```
   docker ps | grep prometheus_server
   docker stop prometheus_server
   docker start prometheus_server
   docker ps | grep prometheus_server
   docker logs --tail 1000 prometheus_server
   ```

   Verify that the process started without errors

## Start ExPECA exporter + collectors

The ExPECA exporter is a custom built exporter that provides metrics for Prometheus. Alongside the exporter, there are "collectors" that are called from the exporter to provide specific metrics.

1. Upload ExPECA exporter + collector files to controller

   ```
   ssh controller
   cd exporter
   ```

   Upload following files to the exporter directory:

   * expeca-exporter.py
   * expeca-exporter.yml
   * expeca-ptp-collector.py
   * expeca-scan-collector.py
   * expeca-serverscan-collector.py

   The files can be found at <https://github.com/KTH-EXPECA/TestbedConfig/tree/master/chi-site-config/monitoring>
2. Install Python modules in controller

   ```
   pip install prometheus-client
   pip install PyYAML
   ```
3. Run ExPECA exporter as background process in controller

   ```
   python3 ./expeca-exporter.py &
   ```

## Start node exporter in worker nodes (only needed if worker node is recreated)

Each worker node has a node exporter (open source) running, that provides important node metrics to Prometheus. Examples are disk and memory usage. Note that the controller node already has the node exporter included as a container service, so it needs no node exporter action.

1. Upload node exporter file to worker node

   ```
   ssh worker-xx
   mkdir exporter
   cd exporter
   ```

   Upload following file to the exporter directory:

   * node\_exporter-1.5.0.linux-amd64.tar.gz

   The file can be found at <https://github.com/KTH-EXPECA/TestbedConfig/tree/master/chi-site-config/monitoring>
2. Extract node exporter

   ```
   tar xvfz node_exporter-1.5.0.linux-amd64.tar.gz
   ```
3. Run node exporter as background process

   ```
   cd node_exporter-1.5.0.linux-amd64
   ./node_exporter &
   ```