Edit

SDR Nodes

An SDR node at ExPECA is a USRP E320 which is powered over ethernet.

Make sure the following configurations are done before deplyment. You need to have connect to the SDR with a micro USB cable. Prepare a SD card reader as well.

Download NI Reference Design version 4.3.0.0

NOTE: make sure you disable conda environment conda deactivate if there is any before the following commands.

In order to change the UHD version on the radio, you must first install the desired UHD version on your host.

sudo apt install -y libboost-all-dev libusb-1.0-0-dev doxygen python3-docutils python3-mako python3-numpy python3-requests python3-ruamel.yaml python3-setuptools cmake build-essential

git clone https://github.com/EttusResearch/uhd.git ~/uhd
cd ~/uhd
git checkout v4.3.0.0
cd host
mkdir build
cd build
cmake ../
make -j $(nproc)
make test # This step is optional
sudo make install
sudo ldconfig

Use uhd_images_downloader to pull the relevant sd image and fpga bitstream.

expeca@forlong:~$ sudo uhd_images_downloader -t sdimg -t e320
[INFO] Using base URL: https://files.ettus.com/binaries/cache/
[INFO] Images destination: /usr/local/share/uhd/images
The file size for this target (565.1 MiB) exceeds the download limit (100.0 MiB). Continue downloading? [y/N]y
233759 kB / 592569 kB (039%) e3xx_e320_sdimg_default-v4.0.0.0.zip

expeca@forlong:~$ uhd_images_downloader -t e320 -t fpga
[INFO] Using base URL: https://files.ettus.com/binaries/cache/
[INFO] Images destination: /usr/local/share/uhd/images
[INFO] Target e3xx_e320_fpga_default is up to date.

Clean Copy NI Reference Design to SD Card

Insert E320's SD card and unmount all partitions. Running the command lsblk again will show these partitions have been unmounted. If they are not (MOUNTPOINT is specified) unmount them.

expeca@forlong:~$ lsblk
NAME           MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sda      8:16   1 14.9G  0 disk
├─sda1   8:17   1   16M  0 part
├─sda2   8:18   1  1.9G  0 part
├─sda3   8:19   1  1.9G  0 part
└─sda4   8:20   1   11G  0 part

We write the SD card image using dd to write the disk image, if the sdimg is located on the PC, run

sudo dd if=/usr/local/share/uhd/images/usrp_e320_fs.sdimg | pv | sudo dd of=/dev/sda bs=1M

  • there is no indication of progress. So be patient.

If the sdimg is located on a USB stick, run

sudo dd if=/media/wlab/cf416480-4366-44bc-8f82-14269c3cde9c/usrp_e320_fs.sdimg | pv | sudo dd of=/dev/sdb bs=10M

To ensure the disk is synchronized, run the sync command:

sync

Copy all the files inside boot partition to the host. Download micro-controller's new firmware.

sudo cp /media/wlab/boot/* ~/e320_ni_design_4.3.0.0/
curl https://files.ettus.com/binaries/misc/upgrade_mcu_neon_v1.1.7358-a190641-musl-glibc-rev3-7.tar.gz -o ~/e320_ni_design_4.3.0.0/upgrade_mcu_neon_v1.1.7358-a190641-musl-glibc-rev3-7.tar.gz

Re-partition the SD card

In order to make an E320 dual boot, we need to copy Mangocomm's boot files into the boot partition alongside the reference boot files. However, boot partition is too small and it needs to be resized. The approach is to append some storage from data partition which is almost free to boot partition.

We use Linux's GParted tool. Install it using the package manager and open the SD card using it. NI E320 SD card by default comes with 4 partitions:

expeca@forlong:~$ sudo fdisk -l /dev/sda
Disk /dev/sda: 14,9 GiB, 16022241280 bytes, 31293440 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x2aacd715

Device     Boot   Start      End  Sectors  Size Id Type
/dev/sda1  *      49152    81919    32768   16M  c W95 FAT32 (LBA)
/dev/sda2         81920  4079615  3997696  1,9G 83 Linux
/dev/sda3       4079616  8077311  3997696  1,9G 83 Linux
/dev/sda4       8077312 31293439 23216128 11,1G 83 Linux

expeca@forlong:~$ sudo parted -l
Model: Generic MassStorageClass (scsi)
Disk /dev/sda: 16,0GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags: 

Number  Start   End     Size    Type     File system  Flags
 1      25,2MB  41,9MB  16,8MB  primary  fat16        boot, lba
 2      41,9MB  2089MB  2047MB  primary  ext4
 3      2089MB  4136MB  2047MB  primary  ext4
 4      4136MB  16,0GB  11,9GB  primary  ext4

Note that partition 1 is named boot, its type is FAT16 and has boot and lba flags.

Make a backup from both boot and data partitions that we want to resize. For the boot partition you can copy the files with cp command. But for the data partition use dd to create an image. We do that to avoid modifing the files permissions on this partition.

expeca@forlong:~$ sudo dd if=/dev/sda4 of=~/Desktop/NI-e320-sdcard/data.img
22323200+0 records in
22323200+0 records out
11429478400 bytes (11 GB, 11 GiB) copied, 132,54 s, 86,2 MB/s

expeca@forlong:~$ mkdir -p ~/Desktop/NI-e320-sdcard/boot
expeca@forlong:~$ cp /media/expeca/boot/* ~/Desktop/NI-e320-sdcard/boot/

Run GParted, unmount all partitions.

Apply the following changes using GParted

  1. Shrink /dev/sda4 to 10.7G (400MB will be freed)

  2. Delete /dev/sda1.

  3. Create a new FAT16 partition with the label boot located after /dev/sda4 using the freed space.

Apply the operations, then don't forget to add the flags: 4. Add flags boot and lba to /dev/sda1

The result should look like this

expeca@forlong:~$ sudo fdisk -l /dev/sda
Disk /dev/sda: 14,9 GiB, 16022241280 bytes, 31293440 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x7d6a2a38

Device     Boot    Start      End  Sectors  Size Id Type
/dev/sda1  *    30400512 31293439   892928  436M  e W95 FAT16 (LBA)
/dev/sda2          81920  4079615  3997696  1,9G 83 Linux
/dev/sda3        4079616  8077311  3997696  1,9G 83 Linux
/dev/sda4        8077312 30400511 22323200 10,7G 83 Linux

Partition table entries are not in disk order.

Mount the new BOOT partition

sudo mkdir /media/BOOT
sudo mount /dev/sdb1 /media/BOOT

Copy from the host, boot partition files that you saved before to the new boot partition. Also create mango_bootbin and ni_bootbin folders on this partition. Copy NI's boot.bin file to its folder.

sudo cp ~/e320_ni_design_4.3.0.0/* /media/BOOT/
sudo mkdir /media/BOOT/mango_bootbin
sudo mkdir /media/BOOT/ni_bootbin
sudo cp ~/e320_ni_design_4.3.0.0/boot.bin /media/BOOT/ni_bootbin/

Create and Copy Mangocomm boot files

Create FPGA image

TODO

Create petalinux image

For each device, a new petalinux image must be built with a new MAC address for the rj45 port. Follow the instructions here to create the petalinux project.

wlab@forlong:~/new_mango_e320$ source ~/petalinux/2019.1/settings.sh
wlab@forlong:~/new_mango_e320$ petalinux-create -t project -s mango_wlan_E320_petalinux_v2.0.2.bsp
INFO: Create project: 
INFO: Projects: 
INFO: 	* mango_wlan_E320_petalinux
INFO: has been successfully installed to /home/wlab/new_mango_e320/
INFO: New project successfully created in /home/wlab/new_mango_e320/

Open the project folder

wlab@forlong:~/new_mango_e320$ cd ~/new_mango_e320/mango_wlan_E320_petalinux

  1. Modify /etc/network/interfaces file for the static ip of eth0

vim components/meta-mango-wlan/meta-mango/recipes-core/init-ifupdown/files/interfaces
...
auto eth0
iface eth0 inet static
address 10.30.10.1
netmask 255.255.255.0

Note that auto eth0 is important, otherwise the interface won't come up.

  1. Build ifupdown module

petalinux-build -c init-ifupdown

  1. Change ethernet MAC address

Open the petalinux-config

wlab@forlong:~/new_mango_e320/mango_wlan_E320_petalinux$ petalinux-config

Then proceed to "Subsystem AUTO Hardware Settings" -> "Ethernet Settings" for setting the ethernet MAC address and static ip. Set the MAC to randomized address generation and static ip without gateway. Then, save the settings and exit the configuration. Return to the config's "Ethernet Settings" and check the generated MAC address. Write it down.

NOTE: the project creates u-boot with Xilinx's automatic configurations. However, the project does not compile successfully in Ubuntu 18.04. To bypass that, open petalinux-config and save it again without any change :D. Then build the project again.

  1. Build the project

petalinux-build

Copy Mango boot files

From petalinux project

sudo cp ~/new_mango_e320/mango_wlan_E320_petalinux/images/linux/image.ub /media/BOOT/

From Xilinx SDK

sudo cp ~/new_mango_e320/Mango_802.11_MAC_Software_E320_v2.0.2/bootimg/boot_wlan_lin_dcf.bit /media/BOOT/wlan.bit
sudo cp ~/new_mango_e320/Mango_802.11_MAC_Software_E320_v2.0.2/wlan-sdk/bootimg/BOOT.bin /media/BOOT/mango_bootbin/BOOT.bin
sudo cp ~/new_mango_e320/Mango_802.11_MAC_Software_E320_v2.0.2/wlan-sdk/bootimg/ec-firmware-no-watchdog.bin /media/BOOT/
sudo cp ~/new_mango_e320/Mango_802.11_MAC_Software_E320_v2.0.2/wlan-sdk/bootimg/uEnv.txt /media/BOOT/

Finally boot should be like:

expeca@forlong:/media/BOOT$ tree
.
├── boot.bin
├── ec-firmware-no-watchdog.bin
├── image.ub
├── mango_bootbin
│   └── BOOT.bin
├── ni_bootbin
│   └── boot.bin
├── u-boot.img
├── uEnv.txt
└── wlan.bit

2 directories, 8 files

Note that we keep boot.bin files in seperate folders for later use.

Unmount the BOOT partition

sync
sudo umount /media/BOOT
sudo rm -rf /media/BOOT

Bring up and Configure NI design

Now insert back the SD card and turn on the device with micro usb connected to the host. Establish UART connection using the screen tool.

expeca@forlong:~$ ls /dev/serial/by-id
usb-Digilent_Digilent_USB_Device_251635236A3B-if00-port0  usb-Silicon_Labs_CP2105_Dual_USB_to_UART_Bridge_Controller_011C582C-if00-port0
usb-Digilent_Digilent_USB_Device_251635236A3B-if01-port0  usb-Silicon_Labs_CP2105_Dual_USB_to_UART_Bridge_Controller_011C582C-if01-port0
expeca@forlong:~$ sudo screen /dev/serial/by-id/usb-Silicon_Labs_CP2105_Dual_USB_to_UART_Bridge_Controller_011C1095-if01-port0 115200
ni-e320-3238B97 login: root
root@ni-e320-3238B97:~#

Setup NI design's static ips by editting these files and restarting the networking deamon:

  1. Set the eth0 and sfp0 ip addresses using these commands:

vim /lib/systemd/network/40-eth0.network
vim /lib/systemd/network/40-sfp0.network

For example eth0 file should be like:

[Match]
Name=eth0
KernelCommandLine=!nfsroot

[Network]
Address=10.30.10.6/24
IPForward=ipv4

sfp0 should be like:

[Match]
Name=sfp0

[Network]
Address=10.30.1.6/24

[Link]
MTUBytes=9000
TxFlowControl=1

  1. Check the eeprom by using eeprom-dump command on the USRP.

-- PID/REV: e320 0005
-- MCU_FLAGS[0]: 00000009
-- MCU_FLAGS[1]: 00000000
-- MCU_FLAGS[2]: 00000000
-- MCU_FLAGS[3]: 00000000
-- Serial: 3238B94
-- eth_addr0: 00:80:2f:33:af:97
-- eth_addr1: 00:80:2f:33:af:98
-- eth_addr2: 00:00:00:00:00:00
-- DT-Compat/MCU-Compat: 0000 0002
-- Max-REV: 0004
-- CRC: a3e865db (matches)

Write down eth_addr0 address to set in the next step for eth0.

  1. Create a systemd link file for the eth0 to change the MAC address policy to none and set the mac address, speed, and duplex modes.

vim /lib/systemd/network/10-eth0.link

Replace the mac address with the one from eeprom:

[Match]
OriginalName=eth0

[Link]
MACAddressPolicy=none
MACAddress=00:80:2f:33:af:97
AutoNegotiation=yes
BitsPerSecond=1000M
Duplex=full

You can check link file and test it via

udevadm -d test-builtin net_setup_link /sys/class/net/eth0

  1. Remove all tmp backup files created by vim or anythin else in the folder.

rm /lib/systemd/network/.10-eth0.link.un~
rm /lib/systemd/network/.40-eth0.network.un~
rm /lib/systemd/network/.40-sfp0.network.un~
rm /lib/systemd/network/10-eth0.link~
rm /lib/systemd/network/40-eth0.network~
rm /lib/systemd/network/40-sfp0.network~

  1. Restart systemd-networkd and reboot

systemctl restart systemd-networkd

Network troubleshoot and checking instructions

Check all interfaces status

root@ni-e320-3238BA5:~# networkctl status     
●   State: routable           
  Address: 10.30.10.16 on eth0 
           10.30.1.16 on sfp0
           169.254.0.1 on int0

Check eth0 network config. Verify that it is using 10-eth0.link and 40-eth0.network network file.

root@ni-e320-3238B97:~# networkctl status eth0
* 2: eth0                                                             
             Link File: /lib/systemd/network/99-default.link          
          Network File: /lib/systemd/network/40-eth0.network          
                  Type: ether                                         
                 State: carrier (configuring)      
                  Path: platform-e000b000.ethernet                    
                Driver: macb                                          
            HW Address: 00:80:2f:33:af:9d (NATIONAL INSTRUMENTS CORP.)
                   MTU: 1500 (min: 68, max: 1500)                     
  Queue Length (Tx/Rx): 1/1                                           
      Auto negotiation: yes                                           
                 Speed: 1Gbps                                         
                Duplex: full                                          
                  Port: mii                                           
          Connected To: poe-switch-01 on port mg2

Check advertised link speeds and connection staatus via:

root@ni-e320-3238BA5:~# ethtool eth0
Settings for eth0:
	Supported ports: [ TP MII ]
	Supported link modes:   10baseT/Half 10baseT/Full 
	                        100baseT/Half 100baseT/Full 
	                        1000baseT/Full 
	Supported pause frame use: Symmetric
	Supports auto-negotiation: Yes
	Supported FEC modes: Not reported
	Advertised link modes:  10baseT/Half 10baseT/Full 
	                        100baseT/Half 100baseT/Full 
	                        1000baseT/Full 
	Advertised pause frame use: No
	Advertised auto-negotiation: Yes
	Advertised FEC modes: Not reported
	Link partner advertised link modes:  100baseT/Half 100baseT/Full 
	                                     1000baseT/Full 
	Link partner advertised pause frame use: No
	Link partner advertised auto-negotiation: Yes
	Link partner advertised FEC modes: Not reported
	Speed: 1000Mb/s
	Duplex: Full
	Port: MII
	PHYAD: 7
	Transceiver: internal
	Auto-negotiation: on
	Link detected: yes

Update E320 FPGA on NI design

In this section we update the FPGA image to version 4.3.0.0 and the XG version to work with the SFP port and 9000 mtu. Put back the sd card into E320, load NI design if you haven't and make sure the host is connected to it via network. Load fpga image by running the following command on the host

sudo uhd_image_loader --args "type=e3xx,mgmt_addr=192.168.2.4,fpga=XG"

Probe the Device from host if you needed

uhd_usrp_probe --args "mgmt_addr=192.168.10.3,addr=192.168.20.3"
uhd_usrp_probe --args "type=e3xx"
uhd_config_info --print-all

Enable autoboot

This should be done on NI design. Enable autoboot on the SDR as show here: kb.ettus.com/E320_Getting_Started_Guide#Autoboot.

root@ni-e320-3238B97:~# eeprom-set-flags $((0x00000008 | 0x1))
-- PID/REV: e320 0005
-- MCU_FLAGS[0]: 00000008
-- MCU_FLAGS[1]: 00000000
-- MCU_FLAGS[2]: 00000000
-- MCU_FLAGS[3]: 00000000
-- Serial: 3238B97
-- eth_addr0: 00:80:2f:33:af:9d
-- eth_addr1: 00:80:2f:33:af:9e
-- eth_addr2: 00:00:00:00:00:00
-- DT-Compat/MCU-Compat: 0000 0002
-- Max-REV: 0004
-- CRC: 4b477607 (matches)
-- Reading back 
-- PID/REV: e320 0005
-- MCU_FLAGS[0]: 00000009
-- MCU_FLAGS[1]: 00000000
-- MCU_FLAGS[2]: 00000000
-- MCU_FLAGS[3]: 00000000
-- Serial: 3238B97
-- eth_addr0: 00:80:2f:33:af:9d
-- eth_addr1: 00:80:2f:33:af:9e
-- eth_addr2: 00:00:00:00:00:00
-- DT-Compat/MCU-Compat: 0000 0002
-- Max-REV: 0004
-- CRC: c41f5914 (matches)

You need to upgrade the embedded controller's firmware so it honors the autoboot setting.

wlab@forlong:~$ curl https://files.ettus.com/binaries/misc/upgrade_mcu_neon_v1.1.7358-a190641-musl-glibc-rev3-7.tar.gz -o upgrade_mcu_neon_v1.1.7358-a190641-musl-glibc-rev3-7.tar.gz
wlab@forlong:~$ scp upgrade_mcu_neon_v1.1.7358-a190641-musl-glibc-rev3-7.tar.gz root@10.30.10.10:/home/root/
root@ni-e320-3238B97:~# tar zxf upgrade_mcu_neon_v1.1.7358-a190641-musl-glibc-rev3-7.tar.gz
root@ni-e320-3238B97:~# cd upgrade_mcu_neon_v1.1.7358-a190641-musl-glibc-rev3-7
root@ni-e320-3238B97:~/upgrade_mcu_neon_v1.1.7358-a190641-musl-glibc-rev3-7# ./flash-firmware.sh

Type y on both questions. Then reboot shutdown -r now.

Verify that autoboot is enabled by unplugging and replugging the PoE cable. The SDR must boot up without the need to press the power button.

Switch to Mango Deign and check it

Open UART screen or use SSH to run the following commands.

Switch to Mango WLAN from NI:

root@ni-e320-3238B97:~# cp /uboot/mango_bootbin/BOOT.bin /uboot/

Switch back to NI Reference design from Mango:

root@mango_wlan_linux_E320:~$ cp /media/card/ni_bootbin/boot.bin /media/card/

Then, run a reboot.

Check the eth static ip:

wlab@forlong:~$ ping 10.30.10.5
PING 10.30.10.5 (10.30.10.5) 56(84) bytes of data.
64 bytes from 10.30.10.5: icmp_seq=1 ttl=64 time=0.515 ms
64 bytes from 10.30.10.5: icmp_seq=2 ttl=64 time=0.275 ms
64 bytes from 10.30.10.5: icmp_seq=3 ttl=64 time=0.272 ms

Run WiFi on Mango Design

Every time mango design boots up, modprobe should be run to bring up the wlan driver, then we change the mac address from default to another one in order to resolve mac address conflicts.

modprobe mango_wlan
iw wlan0 del
iw phy mango-wlan-phy interface add wlan0 type managed addr 40:d8:55:04:20:19
ip link set wlan0 up
./run_ap.sh

Fix UART preventing boot in Mango design

Mango design does not boot without a micro USB cable (UART) connected. In order to resolve that, we use the following hack:

  • Prepare a microUSB to USB cable, where the +5V wire and GND wire are untouched but data wires are grounded (solder them to the GND wire).

  • Connect the USB socket to E320's USB port, connect the microUSB socket to E320's microUSB port.

  • This way the E320's UART transciver which should be powered externally comes up, but since data pins are grounded, it finds nothing and continues.

Read more here: https://support.xilinx.com/s/question/0D52E00006iHma8SAC/zc702-booting-stops-without-uart?language=en_US

Use sdr-tools package to control the SDR

The sdr-tools Python package is developed to control the ExPECA SDRs from any device with network connection to the management port. We show how to do that on the controller node. It is containerized:

For example to change the design on SDR 3 to mango, run the following on the controller:

docker run -it --rm -e SERVICE='change_design' -e DESIGN='mango' -e SDR='sdr-03' -e JSON_PATH='sdrs.json' --network host samiemostafavi/sdr-tools

Tenant Network Configuration

When the device is ready to be attached to the network, there will be 2 connections to the tenants switch: one rj45 and one sfp. Write down which ports they are connected to e.g. "te1/0/28" and "te2/0/15". Later, we register these two ports as one network segment in Openstack after deployment. Therefore, you must make sure that on the tenant switch, they are cleared of any default/native access VLANs. This is necessary to avoid access from other open ports on the switch.

PreviousWorker NodesNextAdvantech Routers

Last updated