Networks

ExPECA Openstack Architecture

Management Networks

Operators of the testbed can access all equipments using this network. This out-of-band network should always be up regardless of the openstack. The gateway to the management network is the management (mgmt) router. Mgmt router has a dedicated public ip which is accessible via ssh. From mgmt router you have access to the management network. There is static host mapping setting on mgmt router too. For example, you can access mgmt router and ping the controller as shown below.

The testbed can be accessed from the internet at mgmt.expeca.proj.kth.se or 130.237.11.98, on SSH port 22. The management entry point is the management router (mgmt.expeca). On this device you can access 3 networks.

The management network consists of 3 networks: 10.10.0.0/16 network for non-sdr equipments, 10.30.1.0/24 network for sdr equipments, and 192.168.0.0/16 network for ipmi access. We use sub-ranges under these blocks to semantically separate devices on the network:

• 10.10.1.[1-255]/16: reserved for networking devices.

• 10.10.2.[1-255]/16: reserved for servers and VMs.

• 10.10.254.[1-254]/16: reserved for external devices (e.g. personal laptops) connected to the network. NOTE: This range is assigned dynamically by the router, using DHCP.

• 10.30.1.[1-255]/24: reserved for sdr equipments.

  • 10.30.1.[ODD numbers]/24: reserved for Mango design.

  • 10.30.1.[EVEN numbers]/24: reserved for NI design.

• 192.168.[1-255].[1-255]/16: reserved for out-of-band IPMI network e.g. iDRAC.

You can run python scan.py Python script in order to check all the nodes in every management network.

SDR Equipments Network

SDR nodes come with 2 interfaces: a 1G rj45 and a 10G sfp. Originally we had decided to assign rj45 to the management network and sfp to the tenants network. However, we realized that Mangocomm WiFi stack only uses the rj45 interface for data. Hence, rj45 port must be connected to tenants switch too. To make SDR nodes accessible to the operators:

• The rj45 interface is assumed to be on tenants network.

• We make sure that the mgmt router has access to all SDR nodes' rj45 interfaces, hence the operator.

In order to achieve that, we prepared a new interface on the router with an ip in SDR eth 10.30.1.252. This port is connected to a trunk port on the tenant switch te1/0/1. Therefore SDR nodes' rj45 interfaces are accessible from mgmt router.

NOTE: because of the trunk setting, if the SDR node is in an active VLAN on the tenant switch, router's interface must be aware of the VLAN, and send tagged packets targeted towards the desired SDR's VLAN. Otherwise, SDR nodes will be on VLAN#1 by default and port te1/0/1 is untagged on VLAN#1.

There are poe switches used to provide power to the SDR nodes. On these switches, per SDR node we have created a layer 2 VLAN. This VLAN includes an SDR facing port and a tenant switch facing port. Therefore these switches act as poe injectors. Make sure you set PVID for each port similar to VLAN id otherwise it does not work.

Internal Network

Openstack services such as k8s, cinder, and swift communicate over this network. It is recommended to enable 10Gb/s speeds with mtu set to 9000 for it.

• Create a VLAN with 10-12 ports on the tenant switch to form the internal network.

  There is VLAN#2 on the tenant switch which is labeled as Internal Network. Ports te1/0/[19-24], te1/0/[43-46] (10 ports) are members of this VLAN.

• Make the controller's internal interface is a trunk port on the tenants switch. Make sure VLAN#2 should be untagged and VLAN#1 tagged.

  Port te1/0/50 on the tenant switch is reserved for this purpose and set to be a trunk port:

  XG50:Trunk ---- 1T, 2U, 3-4094I ---- 1U, 2T

  VLAN#1 contains all ports other than internals including SDR ports.

• Configure the following IPs on the controller internal interface are set:

  • 10.20.111.100/24: to access the internal network

  • 10.30.1.100/24: to access SDRs rj45 ports

  • 10.30.10.100/24: to access SDRs sfp ports

  • 192.168.1.100/16: to access all IPMI ports

• Bring up the nodes' internal interfaces.

• Add controller node's public key to all other nodes on this network and test ssh.

NOTE: the communication over the internal ip 10.20.111.100/24 on the controller should be tagged with the corresponding VLAN on the switch which is 1 in current setup.

The controller netplan config file:

...
    brinternal:
      mtu: 9000
      interfaces:
      - enp3s0
      - veth-internala
      addresses:
      - 10.20.111.100/24
      - 192.168.1.100/16
  vlans:
    vlan.1:
      id: 1
      link: brinternal
      addresses:
      - 10.30.1.100/24
      - 10.30.10.100/24

Public Network

The public network is used for several things. Each purpose can use a separate interface, or reuse the same one (with caveats)

You'll need a minimum of 3 public IP addresses: 1 for the public facing interface, 1 for the haproxy VIP, and one for the default neutron router. However, each instance will need 1 for a floating IP, as will each private tenant network. 20 IPs would be a good starting place, allowing, for example, 15 instances and 2 private networks.

HAProxy creates a /32 address, and binds it to an interface in the public subnet. This address is used for all the public facing API endpoints, as well as the web interface. DNS should resolve for this IP, as well as a valid SSL certificate.

Neutron is the Openstack Networking service. It provides external connectivity for instances by creating virtual routers, and managing public IPs and NAT to internal addresses. It will bind at least one public IP address as the external address for the default router, as well as all assigned floating IPs for instances. The interface defined for this network is attached to an openvswitch bridge.

We have allocated this subnet from KTH: 130.237.11.96/27 with the gateway 130.237.11.96/27. It is connected via 3 ports to the testbed:

1. The management router, to enable internet and SSH access. This port has 130.237.11.98 static IP assigned.

2. The controller server, on the publicbr virtual interfaces created for Openstack haproxy. Static IPs allocated for them are 130.237.11.99 and 130.237.11.99. Also in Openstack, through this interface, Neutron service of Openstack can assign floating IP addresses. We set 130.237.11.[101-112] range allocatable for public network in Neutron.

3. Tenant switch port te3/0/4, to enable containers allocate public addresses. We set 130.237.11.[113-126] range allocatable for serverpublic network in Neutron.

IPMI Network

The controller node (or node running ironic conductor service), must have L3 access to the out of band controller for all baremetal nodes. This can be routed access from one of the other interfaces, or a dedicated connection.

IP address bindings

FQDN	Management IP	SDR Equipment IP	IPMI IP	Internal IP	Public IPs	VM	Function(s)
mgmt.expeca	10.10.1.1	10.30.1.252	192.168.1.1	-	130.237.11.98	no	Gateway, NAT, firewall
mgmt-switch-01.expeca	10.10.1.8	-	-	-	-	no	Management network switch (advertising ptp)
tenant-switch-01.expeca	10.10.1.3	-	-	-	-	no	Tenant facing managed switch
tenant-switch-02.expeca	10.10.1.4	-	-	-	-	no	Tenant facing managed switch
tenant-switch-03.expeca	10.10.1.5	-	-	-	-	no	Tenant facing managed switch
tenant-switch-04.expeca	10.10.1.6	-	-	-	-	no	Tenant facing managed switch
ptp-clock.expeca	10.10.1.7	-	-	-	-	no	PTP grandmaster clock mgmt
ptp-clock-src.expeca	10.10.4.1	-	-	-	-	no	PTP source
storage-01.expeca	10.10.2.1	-	192.168.0.1	10.20.111.1	-	no	Openstack storage node
controller-01.expeca	10.10.2.101	10.30.1.100 10.30.10.100	-	10.20.111.[99,100]	130.237.11.[99,100]	yes	Openstack controller node
worker-01.expeca	10.10.2.2	-	192.168.0.2	10.20.111.2	-	no	worker server
worker-02.expeca	10.10.2.5	-	192.168.0.5	10.20.111.5	-	no	worker server
worker-03.expeca	10.10.2.6	-	192.168.0.6	10.20.111.6	-	no	worker server
worker-04.expeca	10.10.2.7	-	192.168.0.7	10.20.111.7	-	no	worker server
worker-05.expeca	10.10.2.3	-	192.168.0.3	10.20.111.3	-	no	worker server
worker-06.expeca	10.10.2.4	-	192.168.0.4	10.20.111.4	-	no	worker server
worker-07.expeca	10.10.2.8	-	192.168.0.8	10.20.111.8	-	no	worker server
worker-08.expeca	10.10.2.9	-	192.168.0.9	10.20.111.9	-	no	worker server
worker-09.expeca	10.10.2.10	-	192.168.0.10	10.20.111.10	-	no	worker server
worker-10.expeca	10.10.2.11	-	192.168.0.11	10.20.111.11	-	no	worker server
ipmi-switch-01.expeca	192.168.1.2	-	-	-	-	no	IPMI network switch
poe-switch-01.expeca	-	10.30.1.253	-	-	-	no	SDR POE Switch
poe-switch-02.expeca	-	10.30.1.254	-	-	-	no	SDR POE Switch
sdr-0x-ni.expeca	-	10.30.1.x	-	-	-	no	Software defined radio
sdr-0x-mango.expeca	-	10.30.1.x	-	-	-	no	Software defined radio