Worker Nodes

Set up Linux users and permissions

Make the expeca user password a sudoer. Add the following it to the end of the file:

$ sudo visudo
...
expeca    ALL = (ALL) NOPASSWD: ALL

Add controller's public key to ssh authorized keys

Install Dependencies

• Install Docker (Diginal Ocean Docker installation guide for Ubuntu 20.04)

  We install version 20.10.22 of Docker. Use the following commands during the installation instead of sudo apt install docker-ce:

  sudo apt update
  sudo apt install apt-transport-https ca-certificates curl software-properties-common
  curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
  sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu focal stable"
  sudo apt install docker-ce-cli=5:20.10.22~3-0~ubuntu-focal
  sudo apt install docker-ce-rootless-extras=5:20.10.22~3-0~ubuntu-focal
  sudo apt install docker-ce=5:20.10.22~3-0~ubuntu-focal

• Check whether docker is installed properly or not

  sudo docker version

• Install other dependencies:

  sudo apt update && sudo apt install -f -y python3 python3-venv python3-pip jq git

• Install Docker-py:

  First we have to downgrade the requests and urllib3 python packages:

  sudo pip install requests==2.28.1
  sudo pip install urllib3==1.26.13

  Then install docker-py:

  sudo pip install docker

• Install GO

  wget https://dl.google.com/go/go1.19.2.linux-amd64.tar.gz
  sudo tar -xvf go1.19.2.linux-amd64.tar.gz
  sudo mv go /usr/local 

  if you get “directory not empty”, run the go directory with following commands cd /usr/local and `rm -R go/”.

  Add the following lines to the .bashrc file using vim ~/.bashrc:

  GOROOT=/usr/local/go
  GOPATH=$HOME/Projects/Proj1
  PATH=$GOPATH/bin:$GOROOT/bin:$PATH

  To apply the new env vars, log out of the server and back in, or type the following:

  su - ${USER}

  Check whether go is installed properly or not

  go version

Configure performance

• Enable jumbo frames with mtu set to 9000 on capable interfaces

• Disable hyperthreading on BIOS settings or only temporary:

  sudo -i
  echo off > /sys/devices/system/cpu/smt/control
  exit

• Disable power management states of CPU

  sudo apt install linux-tools-common linux-tools-5.4.0-155-generic
  sudo cpupower idle-set -D 2
  sudo cpupower idle-set --disable-by-latency 0

• Set CPU Governor to performance

  sudo apt install cpufrequtils
  for cpu in $(seq 0 $(($(nproc) -1))) ; do sudo cpufreq-set -c $cpu -g performance ; done

• Check:

  sudo cpufreq-info
  ...
  analyzing CPU 31:
  driver: intel_pstate
  CPUs which run at the same hardware frequency: 31
  CPUs which need to have their frequency coordinated by software: 31
  maximum transition latency: 4294.55 ms.
  hardware limits: 800 MHz - 3.40 GHz
  available cpufreq governors: performance, powersave
  current policy: frequency should be within 800 MHz and 3.40 GHz.
                  The governor "performance" may decide which speed to use
                  within this range.
  current CPU frequency is 2.90 GHz.

Stop Ubuntu iscsi services

iscsd process uses configfs which is normally mounted at /sys/kernel/config to store discovered targets information, on centos/rhel type of systems this special file system gets mounted automatically, which is not the case on debian/ubuntu. Since iscsid container runs on every nova compute node, the following steps must be completed on every Ubuntu server targeted for nova compute role.

• Add configfs module to /etc/modules

• Rebuild initramfs using: update-initramfs -u command

• Stop open-iscsi system service due to its conflicts with iscsid container.

  $ sudo systemctl stop open-iscsi
  $ sudo systemctl stop iscsid
  $ sudo yum remove iscsi-initiator-utils
  $ sudo systemctl stop iscsid.socket iscsiuio.socket iscsid.service 
  $ sudo systemctl disable iscsid.socket iscsiuio.socket iscsid.service  

• Make sure configfs gets mounted during a server boot up process. There are multiple ways to accomplish it, one example:

  $ mount -t configfs /etc/rc.local /sys/kernel/config

Start PTP Daemons

Linux PTP is an implementation of Precision Time Protocol (PTP). On Ubuntu, LinuxPTP can be downloaded as installed as follows:

   wget https://deac-fra.dl.sourceforge.net/project/linuxptp/v3.1/linuxptp-3.1.1.tgz
   tar xvf linuxptp-3.1.1.tgz
   cd linuxptp-3.1.1/
   make
   sudo make install
   sudo cp ptp4l phc2sys /usr/sbin/
 

This will install two tools-- ptp4l and phc2sys.

• To install ptp4l as a system service, edit '/usr/lib/systemd/system/ptp4l.service' as shown below:

  [Unit]
  Description=Precision Time Protocol (PTP) service
  Documentation=man:ptp4l
  
  [Service]
  Type=simple
  EnvironmentFile=-/etc/sysconfig/ptp4l
  ExecStart=/usr/sbin/ptp4l $OPTIONS
  
  [Install]
  WantedBy=multi-user.target

  $OPTIONS is located at the following path at '/etc/sysconfig/ptp4l' with the following:

  OPTIONS=-2 -i eth0 -m

  The above method can be used to configure linuxptp as PTP slaves with multicast. In order to enable unicast for PTP slaves the following procedure should be followed.

  The steps to install ptp4l as a system service are the same as above.

  $OPTIONS is located at the following path at '/etc/sysconfig/ptp4l' with the following:

  OPTIONS=-f /etc/ptp4l.cfg -m

  The content for /etc/ptp4l.cfg are

  [global]
  
  unicast_req_duration 60
  
  [unicast_master_table]
  table_id 1
  logQueryInterval 2
  UDPv4 10.10.80.1
  
  [eno8303]
  unicast_master_table 1

  To start and stop the ptp4l service use:

  systemctl start ptp4l 
  systemctl stop ptp4l 

  To enable pt4l to run after every reboot enable it using systemctl:

  systemctl enable ptp4l

• Similarly, to install phc2sys as a system service, edit '/usr/lib/systemd/system/phc2sys.service' as shown below:

  [Unit]
  Description=Synchronize system clock or PTP hardware clock (PHC)
  Documentation=man:phc2sys
  After=ntpdate.service
  Requires=ptp4l.service
  After=ptp4l.service
  
  [Service]
  Type=simple
  EnvironmentFile=-/etc/sysconfig/phc2sys
  ExecStart=/usr/sbin/phc2sys $OPTIONS
  
  [Install]
  WantedBy=multi-user.target
  

  $OPTIONS is located at the following path at '/etc/sysconfig/phc2sys' with the following:

  OPTIONS=-a -r -m

  To start and stop the phc2sys service:

  systemctl start phc2sys 
  systemctl stop phc2sys 

  To enable phc2sys to run after every reboot enable it using systemctl:

  systemctl enable phc2sys

  Finally, make sure NTP is not active on the machine.

  sudo timedatectl set-ntp 0