# Deploy Monitoring Services

## Update Prometheus

Prometheus is the Time Series DataBase handler that is at the core of our monitoring solution. It executes as a docker container service and is included with the chi-in-a-box system in the controller node. It periodically fetches metrics from exporter applications via HTTP scraping. Grafana is used then to query metrics from Prometheus for display (dashboards).

1.  Upload Prometheus config files to controller

    ```
    ssh controller
    mkdir exporter
    cd exporter
    ```

    Upload following files to the exporter directory:

    * config.json
    * prometheus.yml

    The files can be found at https://github.com/KTH-EXPECA/TestbedConfig/tree/master/chi-site-config/monitoring
2.  Copy the config files to correct Prometheus directory:

    ```
    sudo cp config.json /etc/kolla/prometheus-server
    sudo cp prometheus.yml /etc/kolla/prometheus-server
    sudo ls -l /etc/kolla/prometheus-server
    ```

    Verify that owner of the target files are root:root
3.  Restart the Prometheus server

    ```
    docker ps | grep prometheus_server
    docker stop prometheus_server
    docker start prometheus_server
    docker ps | grep prometheus_server
    docker logs --tail 1000 prometheus_server
    ```

    Verify that the process started without errors

## Start ExPECA exporter + collectors

The ExPECA exporter is a custom built exporter that provides metrics for Prometheus. Alongside the exporter, there are "collectors" that are called from the exporter to provide specific metrics.

1.  Upload ExPECA exporter + collector files to controller

    ```
    ssh controller
    cd exporter
    ```

    Upload following files to the exporter directory:

    * expeca-exporter.py
    * expeca-exporter.yml
    * expeca-ptp-collector.py
    * expeca-scan-collector.py
    * expeca-serverscan-collector.py

    The files can be found at https://github.com/KTH-EXPECA/TestbedConfig/tree/master/chi-site-config/monitoring
2.  Install Python modules in controller

    ```
    pip install prometheus-client
    pip install PyYAML
    ```
3.  Run ExPECA exporter as background process in controller

    ```
    python3 ./expeca-exporter.py &
    ```

## Start node exporter in worker nodes (only needed if worker node is recreated)

Each worker node has a node exporter (open source) running, that provides important node metrics to Prometheus. Examples are disk and memory usage. Note that the controller node already has the node exporter included as a container service, so it needs no node exporter action.

1.  Upload node exporter file to worker node

    ```
    ssh worker-xx
    mkdir exporter
    cd exporter
    ```

    Upload following file to the exporter directory:

    * node\_exporter-1.5.0.linux-amd64.tar.gz

    The file can be found at https://github.com/KTH-EXPECA/TestbedConfig/tree/master/chi-site-config/monitoring
2.  Extract node exporter

    ```
    tar xvfz node_exporter-1.5.0.linux-amd64.tar.gz
    ```
3.  Run node exporter as background process

    ```
    cd node_exporter-1.5.0.linux-amd64
    ./node_exporter &
    ```
