> For the complete documentation index, see [llms.txt](https://kth-expeca.gitbook.io/testbedconfig/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://kth-expeca.gitbook.io/testbedconfig/prepare/networks.md).

# Networks

![ExPECA Openstack Architecture](/files/kOKhsIDnSf9TtweCjdVw)

## Management Networks

Operators of the testbed can access all equipments using this network. This out-of-band network should always be up regardless of the openstack. The gateway to the management network is the management (mgmt) router. Mgmt router has a dedicated public ip which is accessible via ssh. From mgmt router you have access to the management network. There is static host mapping setting on mgmt router too. For example, you can access mgmt router and ping the controller as shown below.

The testbed can be accessed from the internet at [`mgmt.expeca.proj.kth.se`](ssh://mgmt.expeca.proj.kth.se:22) or [`130.237.11.98`](ssh://130.237.11.98:22), on SSH port `22`. The management entry point is the management router (`mgmt.expeca`). On this device you can access 3 networks.

The management network consists of 3 networks: `10.10.0.0/16` network for non-sdr equipments, `10.30.1.0/24` network for sdr equipments, and `192.168.0.0/16` network for ipmi access. We use sub-ranges under these blocks to semantically separate devices on the network:

* `10.10.1.[1-255]/16`: reserved for networking devices.
* `10.10.2.[1-255]/16`: reserved for servers and VMs.
* `10.10.254.[1-254]/16`: reserved for external devices (e.g. personal laptops) connected to the network. *NOTE: This range is assigned dynamically by the router, using DHCP.*
* `10.30.1.[1-255]/24`: reserved for sdr equipments.
  * `10.30.1.[ODD numbers]/24`: reserved for Mango design.
  * `10.30.1.[EVEN numbers]/24`: reserved for NI design.
* `192.168.[1-255].[1-255]/16`: reserved for out-of-band IPMI network e.g. iDRAC.

You can run `python scan.py` Python script in order to check all the nodes in every management network.

### SDR Equipments Network

<figure><img src="/files/fASGB1GqO85w1Zv8E6oy" alt=""><figcaption></figcaption></figure>

SDR nodes come with 2 interfaces: a 1G rj45 and a 10G sfp. Originally we had decided to assign rj45 to the management network and sfp to the tenants network. However, we realized that Mangocomm WiFi stack only uses the rj45 interface for data. Hence, rj45 port must be connected to tenants switch too. To make SDR nodes accessible to the operators:

* The rj45 interface is assumed to be on tenants network.
* We make sure that the mgmt router has access to all SDR nodes' rj45 interfaces, hence the operator.

In order to achieve that, we prepared a new interface on the router with an ip in SDR eth `10.30.1.252`. This port is connected to a trunk port on the tenant switch `te1/0/1`. Therefore SDR nodes' rj45 interfaces are accessible from mgmt router.

NOTE: because of the trunk setting, if the SDR node is in an active VLAN on the tenant switch, router's interface must be aware of the VLAN, and send tagged packets targeted towards the desired SDR's VLAN. Otherwise, SDR nodes will be on VLAN#1 by default and port `te1/0/1` is `untagged` on VLAN#1.

There are poe switches used to provide power to the SDR nodes. On these switches, per SDR node we have created a layer 2 VLAN. This VLAN includes an SDR facing port and a tenant switch facing port. Therefore these switches act as poe injectors. Make sure you set PVID for each port similar to VLAN id otherwise it does not work.

## Internal Network

Openstack services such as k8s, cinder, and swift communicate over this network. It is recommended to enable 10Gb/s speeds with mtu set to 9000 for it.

* Create a VLAN with 10-12 ports on the tenant switch to form the internal network.

  There is VLAN#2 on the tenant switch which is labeled as `Internal Network`. Ports `te1/0/[17-24]`, `te1/0/[43-46]` (10 ports) are members of this VLAN.
* Make the controller's internal interface is a trunk port on the tenants switch. Make sure VLAN#2 should be `untagged` and VLAN#1 `tagged`.

  Port `te1/0/50` on the tenant switch is reserved for this purpose and set to be a trunk port:

  `XG50:Trunk ---- 1T, 2U, 3-4094I ---- 1U, 2T`

  VLAN#1 contains all ports other than internals including SDR ports.
* Configure the following IPs on the controller internal interface are set:
  * `10.20.111.100/24`: to access the internal network
  * `10.30.1.100/24`: to access SDRs rj45 ports
  * `10.30.10.100/24`: to access SDRs sfp ports
  * `192.168.1.100/16`: to access all IPMI ports
* Bring up the nodes' internal interfaces.
* Add `controller` node's public key to all other nodes on this network and test `ssh`.

NOTE: the communication over the internal ip `10.20.111.100/24` on the controller should be tagged with the corresponding VLAN on the switch which is 1 in current setup.

The controller netplan config file:

```
...
    brinternal:
      mtu: 9000
      interfaces:
      - enp3s0
      - veth-internala
      addresses:
      - 10.20.111.100/24
      - 192.168.1.100/16
  vlans:
    vlan.1:
      id: 1
      link: brinternal
      addresses:
      - 10.30.1.100/24
      - 10.30.10.100/24
```

## Public Network

The public network is used for several things. Each purpose can use a separate interface, or reuse the same one (with caveats)

You'll need a minimum of 3 public IP addresses: 1 for the public facing interface, 1 for the haproxy VIP, and one for the default neutron router. However, each instance will need 1 for a floating IP, as will each private tenant network. 20 IPs would be a good starting place, allowing, for example, 15 instances and 2 private networks.

HAProxy creates a /32 address, and binds it to an interface in the public subnet. This address is used for all the public facing API endpoints, as well as the web interface. DNS should resolve for this IP, as well as a valid SSL certificate.

Neutron is the Openstack Networking service. It provides external connectivity for instances by creating virtual routers, and managing public IPs and NAT to internal addresses. It will bind at least one public IP address as the external address for the default router, as well as all assigned floating IPs for instances. The interface defined for this network is attached to an openvswitch bridge.

We have allocated this subnet from KTH: `130.237.11.96/27` with the gateway `130.237.11.96/27`.\
It is connected via 3 ports to the testbed:

1. The management router, to enable internet and SSH access. This port has `130.237.11.98` static IP assigned.
2. The controller server, on the `publicbr` virtual interfaces created for Openstack haproxy. Static IPs allocated for them are `130.237.11.99` and `130.237.11.99`. Also in Openstack, through this interface, Neutron service of Openstack can assign floating IP addresses. We set `130.237.11.[101-112]` range allocatable for `public` network in Neutron.
3. Tenant switch port `te3/0/4`, to enable containers allocate public addresses. We set `130.237.11.[113-126]` range allocatable for `serverpublic` network in Neutron.

## IPMI Network

The controller node (or node running ironic conductor service), must have L3 access to the out of band controller for all baremetal nodes. This can be routed access from one of the other interfaces, or a dedicated connection.

## IP address bindings

<table><thead><tr><th>FQDN</th><th>Management IP</th><th>SDR Equipment IP</th><th>IPMI IP</th><th>Internal IP</th><th>Public IPs</th><th>VM</th><th>Function(s)</th><th data-hidden></th></tr></thead><tbody><tr><td><code>mgmt.expeca</code></td><td><code>10.10.1.1</code></td><td><code>10.30.1.252</code></td><td><code>192.168.1.1</code></td><td><code>-</code></td><td><code>130.237.11.98</code></td><td>no</td><td>Gateway, NAT, firewall</td><td></td></tr><tr><td><code>mgmt-switch-01.expeca</code></td><td><code>10.10.1.8</code></td><td><code>-</code></td><td><code>-</code></td><td><code>-</code></td><td><code>-</code></td><td>no</td><td>Management network switch (advertising ptp)</td><td></td></tr><tr><td><code>tenant-switch-01.expeca</code></td><td><code>10.10.1.3</code></td><td><code>-</code></td><td><code>-</code></td><td><code>-</code></td><td><code>-</code></td><td>no</td><td>Tenant facing managed switch</td><td></td></tr><tr><td><code>tenant-switch-02.expeca</code></td><td><code>10.10.1.4</code></td><td><code>-</code></td><td><code>-</code></td><td><code>-</code></td><td><code>-</code></td><td>no</td><td>Tenant facing managed switch</td><td></td></tr><tr><td><code>tenant-switch-03.expeca</code></td><td><code>10.10.1.5</code></td><td><code>-</code></td><td><code>-</code></td><td><code>-</code></td><td><code>-</code></td><td>no</td><td>Tenant facing managed switch</td><td></td></tr><tr><td><code>tenant-switch-04.expeca</code></td><td><code>10.10.1.6</code></td><td><code>-</code></td><td><code>-</code></td><td><code>-</code></td><td><code>-</code></td><td>no</td><td>Tenant facing managed switch</td><td></td></tr><tr><td><code>ptp-clock.expeca</code></td><td><code>10.10.1.7</code></td><td><code>-</code></td><td><code>-</code></td><td><code>-</code></td><td><code>-</code></td><td>no</td><td>PTP grandmaster clock mgmt</td><td></td></tr><tr><td><code>ptp-clock-src.expeca</code></td><td><code>10.10.4.1</code></td><td><code>-</code></td><td><code>-</code></td><td><code>-</code></td><td><code>-</code></td><td>no</td><td>PTP source</td><td></td></tr><tr><td><code>storage-01.expeca</code></td><td><code>10.10.2.1</code></td><td><code>-</code></td><td><code>192.168.0.1</code></td><td><code>10.20.111.1</code></td><td><code>-</code></td><td>no</td><td>Openstack storage node</td><td></td></tr><tr><td><code>controller-01.expeca</code></td><td><code>10.10.2.101</code></td><td><p><code>10.30.1.100</code></p><p><code>10.30.10.100</code></p></td><td><code>-</code></td><td><code>10.20.111.[99,100]</code></td><td><code>130.237.11.[99,100]</code></td><td>yes</td><td>Openstack controller node</td><td></td></tr><tr><td><code>worker-01.expeca</code></td><td><code>10.10.2.2</code></td><td><code>-</code></td><td><code>192.168.0.2</code></td><td><code>10.20.111.2</code></td><td><code>-</code></td><td>no</td><td>worker server</td><td></td></tr><tr><td><code>worker-02.expeca</code></td><td><code>10.10.2.5</code></td><td><code>-</code></td><td><code>192.168.0.5</code></td><td><code>10.20.111.5</code></td><td><code>-</code></td><td>no</td><td>worker server</td><td></td></tr><tr><td><code>worker-03.expeca</code></td><td><code>10.10.2.6</code></td><td><code>-</code></td><td><code>192.168.0.6</code></td><td><code>10.20.111.6</code></td><td><code>-</code></td><td>no</td><td>worker server</td><td></td></tr><tr><td><code>worker-04.expeca</code></td><td><code>10.10.2.7</code></td><td><code>-</code></td><td><code>192.168.0.7</code></td><td><code>10.20.111.7</code></td><td><code>-</code></td><td>no</td><td>worker server</td><td></td></tr><tr><td><code>worker-05.expeca</code></td><td><code>10.10.2.3</code></td><td><code>-</code></td><td><code>192.168.0.3</code></td><td><code>10.20.111.3</code></td><td><code>-</code></td><td>no</td><td>worker server</td><td></td></tr><tr><td><code>worker-06.expeca</code></td><td><code>10.10.2.4</code></td><td><code>-</code></td><td><code>192.168.0.4</code></td><td><code>10.20.111.4</code></td><td><code>-</code></td><td>no</td><td>worker server</td><td></td></tr><tr><td><code>worker-07.expeca</code></td><td><code>10.10.2.8</code></td><td><code>-</code></td><td><code>192.168.0.8</code></td><td><code>10.20.111.8</code></td><td><code>-</code></td><td>no</td><td>worker server</td><td></td></tr><tr><td><code>worker-08.expeca</code></td><td><code>10.10.2.9</code></td><td><code>-</code></td><td><code>192.168.0.9</code></td><td><code>10.20.111.9</code></td><td><code>-</code></td><td>no</td><td>worker server</td><td></td></tr><tr><td><code>worker-09.expeca</code></td><td><code>10.10.2.10</code></td><td><code>-</code></td><td><code>192.168.0.10</code></td><td><code>10.20.111.10</code></td><td><code>-</code></td><td>no</td><td>worker server</td><td></td></tr><tr><td><code>worker-10.expeca</code></td><td><code>10.10.2.11</code></td><td><code>-</code></td><td><code>192.168.0.11</code></td><td><code>10.20.111.11</code></td><td><code>-</code></td><td>no</td><td>worker server</td><td></td></tr><tr><td><code>ipmi-switch-01.expeca</code></td><td><code>192.168.1.2</code></td><td><code>-</code></td><td><code>-</code></td><td><code>-</code></td><td><code>-</code></td><td>no</td><td>IPMI network switch</td><td></td></tr><tr><td><code>poe-switch-01.expeca</code></td><td><code>-</code></td><td><code>10.30.1.253</code></td><td><code>-</code></td><td><code>-</code></td><td><code>-</code></td><td>no</td><td>SDR POE Switch</td><td></td></tr><tr><td><code>poe-switch-02.expeca</code></td><td><code>-</code></td><td><code>10.30.1.254</code></td><td><code>-</code></td><td><code>-</code></td><td><code>-</code></td><td>no</td><td>SDR POE Switch</td><td></td></tr><tr><td><code>sdr-0x-ni.expeca</code></td><td><code>-</code></td><td><code>10.30.1.x</code></td><td><code>-</code></td><td><code>-</code></td><td><code>-</code></td><td>no</td><td>Software defined radio</td><td></td></tr><tr><td><code>sdr-0x-mango.expeca</code></td><td><code>-</code></td><td><code>10.30.1.x</code></td><td><code>-</code></td><td><code>-</code></td><td><code>-</code></td><td>no</td><td>Software defined radio</td><td></td></tr></tbody></table>