Storage Node

At ExPECA testbed, we configure Cinder and Swift storage services of Openstack and a seperate storage node is used. We configure Openstack to share the Cinder's block storage (e.g. /dev/sdb on the storage node) among all worker nodes using iSCSI.

Make sure the following are set in site-config/defaults.yaml:

enable_cinder: yes
enable_cinder_backend_lvm: yes
enable_swift: yes

Also make sure that a storage node is set in site-config/inventory/hosts as mentioned in the deployment instructions.

After the deployment, tgtd service will be running on the storage node. All the volumes created using Cinder will be added as targets in tgt so K8S pods can connect to them to get access to the Cinder's block storage.

Prepare the Storage Node

The storage node on Openstack is a machine with sufficient block storage capacity. Preferably multiple hdds configured in raid format. For instance, the following block storage setup could be used:

$ lsblk
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda      8:0    0 447.1G  0 disk 
|-sda1   8:1    0   1.1G  0 part /boot/efi
|-sda2   8:2    0   512M  0 part 
|-sda3   8:3    0  31.3G  0 part [SWAP]
`-sda4   8:4    0 414.3G  0 part /
sdb      8:16   0   2.7T  0 disk 
sdc      8:32   0   3.7T  0 disk

The sda block is the SSD storage that the machine uses for the operating system. Additionally, there is sdb and sdc that can be used for Openstack storage node.

ExPECA testbed offers both block storage (Cinder) and object store services (Swift). A simple plan is to use each block for each service. For example sdb for Cinder and sdc for Swift.

Set up Linux users and permissions

Make the expeca user password a sudoer. Add the following it to the end of the file:

$ sudo visudo
...
expeca    ALL = (ALL) NOPASSWD: ALL

Add controller's public key to ssh authorized keys

Swift preparations

Create labeled partitions

Swift configuration works best if the disk is partitioned. In order to do that according to your needs, you can use online tutorials such as this for the disks smaller than 2TB. However, for disks larger than 2TB, parted must be used. The following commands will create one primary partition that takes over all the space on sdc and adds KOLLA_SWIFT_DATA label to it.

$ sudo parted /dev/sdc -s -- mklabel gpt mkpart KOLLA_SWIFT_DATA 1 -1

Then we format the partition to xfs file system with label d0

$ sudo mkfs.xfs -f -L d0 /dev/sdc1
meta-data=/dev/sdc1              isize=512    agcount=32, agsize=30515136 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=1, rmapbt=0
         =                       reflink=1
data     =                       bsize=4096   blocks=976484352, imaxpct=5
         =                       sunit=64     swidth=128 blks
naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
log      =internal log           bsize=4096   blocks=476800, version=2
         =                       sectsz=512   sunit=64 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

Verify that the partition with file system xfs and label KOLLA_SWIFT_DATA

$ sudo parted /dev/sdc print
Model: DELL PERC H755 Front (scsi)
Disk /dev/sdc: 4000GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags: 

Number  Start   End     Size    File system  Name              Flags
 1      1049kB  4000GB  4000GB  xfs          KOLLA_SWIFT_DATA

For evaluation pureposes, loop devices could be created as shown in the upstream references.

Cinder preparations

In this section we describe the commands to prepare a storage device sdb to serve as an LVM backend for Cinder. Unlike Swift, Cinder needs the disk to be without any partition table. The goal is to create cinder-volumes LVM volume group.

Make sure there is an unused storage device on the host.

$ lsblk
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda      8:0    0 447.1G  0 disk 
|-sda1   8:1    0   1.1G  0 part /boot/efi
|-sda2   8:2    0   512M  0 part 
|-sda3   8:3    0  31.3G  0 part [SWAP]
`-sda4   8:4    0 414.3G  0 part /
sdb      8:16   0   2.7T  0 disk 
sdc      8:32   0   3.7T  0 disk 
`-sdc1   8:33   0   3.7T  0 part

In this case sda is the OS device and we consider sdb for the LVM volume group.

Create the LVM physical volume /dev/vdb:

$ sudo pvcreate /dev/sdb
WARNING: dos signature detected on /dev/sdb at offset 510. Wipe it? [y/n]: y
  Wiping dos signature on /dev/sdb.
  Physical volume "/dev/sdb" successfully created.

If you encountered the following error

$ sudo pvcreate /dev/sdb
  Device /dev/sdb excluded by a filter.

The reason is that there is partition info on the disk that must be wiped:

$ sudo wipefs -a /dev/sdb
/dev/sdb: 2 bytes were erased at offset 0x000001fe (dos): 55 aa
/dev/sdb: calling ioctl to re-read partition table: Success

$ sudo pvcreate /dev/sdb
  Physical volume "/dev/sdb" successfully created.

Create the LVM volume group cinder-volumes:

$ sudo vgcreate cinder-volumes /dev/sdb
  Volume group "cinder-volumes" successfully created

Verify the volumes are ready:

$ sudo vgdisplay
  --- Volume group ---
  VG Name               cinder-volumes
  System ID             
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  1
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                0
  Open LV               0
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               <2.73 TiB
  PE Size               4.00 MiB
  Total PE              715007
  Alloc PE / Size       0 / 0   
  Free  PE / Size       715007 / <2.73 TiB
  VG UUID               7o4Kg1-VxDY-4sBn-FUd4-VzKy-Abzh-Fpu9b0

References

• docs.openstack.org/cinder/xena/install
• docs.openstack.org/cinder/xena/install/cinder-storage-install-obs.html
• docs.openstack.org/kolla-ansible/xena/reference/storage/swift-guide.html
• docs.openstack.org/swift/xena/deployment_guide.html