Deploy Monitoring Services

Update Prometheus

Prometheus is the Time Series DataBase handler that is at the core of our monitoring solution. It executes as a docker container service and is included with the chi-in-a-box system in the controller node. It periodically fetches metrics from exporter applications via HTTP scraping. Grafana is used then to query metrics from Prometheus for display (dashboards).

1. Upload Prometheus config files to controller

   ssh controller
   mkdir exporter
   cd exporter

   Upload following files to the exporter directory:

   • config.json

   • prometheus.yml

   The files can be found at https://github.com/KTH-EXPECA/TestbedConfig/tree/master/chi-site-config/monitoring

2. Copy the config files to correct Prometheus directory:

   sudo cp config.json /etc/kolla/prometheus-server
   sudo cp prometheus.yml /etc/kolla/prometheus-server
   sudo ls -l /etc/kolla/prometheus-server

   Verify that owner of the target files are root:root

3. Restart the Prometheus server

   docker ps | grep prometheus_server
   docker stop prometheus_server
   docker start prometheus_server
   docker ps | grep prometheus_server
   docker logs --tail 1000 prometheus_server

   Verify that the process started without errors

Start ExPECA exporter + collectors

The ExPECA exporter is a custom built exporter that provides metrics for Prometheus. Alongside the exporter, there are "collectors" that are called from the exporter to provide specific metrics.

1. Upload ExPECA exporter + collector files to controller

   ssh controller
   cd exporter

   Upload following files to the exporter directory:

   • expeca-exporter.py

   • expeca-exporter.yml

   • expeca-ptp-collector.py

   • expeca-scan-collector.py

   • expeca-serverscan-collector.py

   The files can be found at https://github.com/KTH-EXPECA/TestbedConfig/tree/master/chi-site-config/monitoring

2. Install Python modules in controller

   pip install prometheus-client
   pip install PyYAML

3. Run ExPECA exporter as background process in controller

   python3 ./expeca-exporter.py &

Start node exporter in worker nodes (only needed if worker node is recreated)

Each worker node has a node exporter (open source) running, that provides important node metrics to Prometheus. Examples are disk and memory usage. Note that the controller node already has the node exporter included as a container service, so it needs no node exporter action.

1. Upload node exporter file to worker node

   ssh worker-xx
   mkdir exporter
   cd exporter

   Upload following file to the exporter directory:

   • node_exporter-1.5.0.linux-amd64.tar.gz

   The file can be found at https://github.com/KTH-EXPECA/TestbedConfig/tree/master/chi-site-config/monitoring

2. Extract node exporter

   tar xvfz node_exporter-1.5.0.linux-amd64.tar.gz

3. Run node exporter as background process

   cd node_exporter-1.5.0.linux-amd64
   ./node_exporter &