# Demikernel in Cloudlab # Requesting resources on Cloudlab - Look at the node IDs which have which have Mellanox ConnectX-4 or above in the hardware allocation in the hardware list (https://docs.cloudlab.us/hardware.html). - Some of the ones that work: Utah (c6525-100g, c6525-25g, xl170, d6515), Clemson (r7525) - Starting a new experiment - Choose Profile: small-lan - Parameterize - No of nodes: 2+ - OS Image: Ubuntu 20.04 - Optional Physical Node Type [REALLY IMPORTANT]: Put one of the IDSs mentioned above. Check availability here: https://www.cloudlab.us/resinfo.php?embedded=true - Finalize - Schedule # Pre Requisites Run the following scripts on all the nodes. ## Installing Mellanox OFED (mlx5) wget https://content.mellanox.com/ofed/MLNX_OFED-5.5-1.0.3.2/MLNX_OFED_LINUX-5.5-1.0.3.2-ubuntu20.04-x86_64.tgz --no-check-certificate tar -xzvf MLNX_OFED_LINUX-5.5-1.0.3.2-ubuntu20.04-x86_64.tgz cd MLNX_OFED_LINUX-5.5-1.0.3.2-ubuntu20.04-x86_64 sudo ./mlnxofedinstall --upstream-libs --dpdk ## Installing Rust Cargo curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh # Setup for Demikernel ## Pull the code from repository Fetch the code from the original repository and use `dev` branch for latest updates git clone --recursive https://github.com/demikernel/demikernel.git cd demikernel git checkout dev ## Running setup scripts Execute these scripts within the demikernel repository to setup pre-requisite for Ubuntu, Rust DPDK, and Hugepages ./scripts/setup/debian.sh ./scripts/setup/dpdk.sh ./scripts/setup/hugepages.sh # Running Demikernel Run `make all` to make sure everything compiles. ## Configuration files Edit the config files in `scripts/config/default.yaml` using the following steps. **Node 0 → Server** * Get information about the Mellanox NICs using `lshw` ``` user@node0:~/demikernel/scripts/config$ sudo lshw | grep -i Mellanox -A 10 vendor: Mellanox Technologies physical id: 0.1 bus info: pci@0000:03:00.1 logical name: ens1f1 version: 00 serial: 9c:dc:71:5e:2f:71 capacity: 25Gbit/s width: 64 bits clock: 33MHz capabilities: pciexpress vpd msix pm bus_master cap_list ethernet physical 1000bt-fd 10000bt-fd 25000bt-fd autonegotiation configuration: autonegotiation=off broadcast=yes driver=mlx5_core driverversion=5.5-1.0.3 duplex=full firmware=14.18.2030 (HP_2420110034) ip=10.10.1.1 latency=0 link=yes multicast=yes ``` From the above output extract the following entities required for config file: 1. `my_interface_name` → `logical name` (`ens1f1`) 2. `my_link_address` → serial (`9c:dc:71:5e:2f:71`) If there are multiple interfaces pick one of the interface. For this example, our interface name is `ens1f1`. * Finding out the IP of the system using `ifconfig` ``` user@node0:~/demikernel/scripts/config$ ifconfig ens1f1 ens1f1: flags=4163 mtu 1500 inet 10.10.1.1 netmask 255.255.255.0 broadcast 10.10.1.255 inet6 fe80::9edc:71ff:fe5e:2f71 prefixlen 64 scopeid 0x20 ether 9c:dc:71:5e:2f:71 txqueuelen 1000 (Ethernet) RX packets 8 bytes 670 (670.0 B) RX errors 0 dropped 1 overruns 0 frame 0 TX packets 19 bytes 1895 (1.8 KB) TX errors 1 dropped 0 overruns 0 carrier 1 collisions 0 ``` IP: `10.10.1.1` * Finding the PCI address for the `eal_init` section using `lshw` ``` user@node0:~/demikernel/scripts/config$ sudo lshw -c network -businfo Bus info Device Class Description ======================================================= pci@0000:03:00.1 ens1f1 network MT27710 Family [ConnectX-4 Lx] ``` From the above example, the PCI address will be `03:00.1` **Node 1 → Client** Apply the same commands to get the values for Node 1 which will be our client. * Get information about the NIC using `lshw` ``` user@node1:~/demikernel/scripts/config$ sudo lshw | grep -i Mellanox -A 10 vendor: Mellanox Technologies physical id: 0.1 bus info: pci@0000:03:00.1 logical name: ens1f1 version: 00 serial: 9c:dc:71:5d:41:31 capacity: 25Gbit/s width: 64 bits clock: 33MHz capabilities: pciexpress vpd msix pm bus_master cap_list ethernet physical 1000bt-fd 10000bt-fd 25000bt-fd autonegotiation configuration: autonegotiation=off broadcast=yes driver=mlx5_core driverversion=5.5-1.0.3 duplex=full firmware=14.18.2030 (HP_2420110034) ip=10.10.1.2 latency=0 link=yes multicast=yes ``` From the above output extract the following entities required for config file 1. `my_interface_name` → `logical name` (`ens1f1`) 2. `my_link_address` → serial (`9c:dc:71:5d:41:31`) If there are multiple interfaces pick one of the interface. For this example, our interface name is `ens1f1`. * Get information about IP using `ifconfig` ``` user@node1:~/demikernel$ ifconfig ens1f1 ens1f1: flags=4163 mtu 1500 inet 10.10.1.2 netmask 255.255.255.0 broadcast 10.10.1.255 inet6 fe80::9edc:71ff:fe5d:4131 prefixlen 64 scopeid 0x20 ether 9c:dc:71:5d:41:31 txqueuelen 1000 (Ethernet) RX packets 14 bytes 1293 (1.2 KB) RX errors 0 dropped 1 overruns 0 frame 0 TX packets 19 bytes 1895 (1.8 KB) TX errors 1 dropped 0 overruns 0 carrier 1 collisions 0 ``` IP → `10.10.1.2` * Finding the PCI address for the `eal_init` section using `lshw` ``` user@node1:~/demikernel$ sudo lshw -c network -businfo Bus info Device Class Description ======================================================= pci@0000:03:00.1 ens1f1 network MT27710 Family [ConnectX-4 Lx] ``` From the above example, the PCI address will be `03:00.1` **Putting together config file for Node 0** ``` # Copyright (c) Microsoft Corporation. # Licensed under the MIT license. server: bind: host: 10.10.1.1 (IP of Node 0) port: 12345 client: host: 10.10.1.2 (IP of Node 1) port: 12345 catnip: my_ipv4_addr: 10.10.1.1 (IP of Node 0) my_link_addr: "9c:dc:71:5e:2f:71" (Serial of ens1f1 in Node 0) my_interface_name: "ens1f1" arp_table: : "9c:dc:71:5e:2f:71": 10.10.1.1 "9c:dc:71:5d:41:31": 10.10.1.2 dpdk: eal_init: ["-c", "0xff", "-n", "4", "-w", "03:00.1","--proc-type=auto"] || \/ PCI address of ens1f1 in Node 0 # vim: set tabstop=2 shiftwidth=2 ``` **Putting together config file for Node 1** ``` user@node1:~/demikernel$ cat scripts/config/cl_node_1.yaml # Copyright (c) Microsoft Corporation. # Licensed under the MIT license. client: connect_to: host: 10.10.1.1 (IP of Node 0) port: 12345 client: host: 10.10.1.2 (IP of Node 1) port: 12345 catnip: my_ipv4_addr: 10.10.1.2 (IP of Node 1) my_link_addr: "9c:dc:71:5d:41:31" (Serial of ens1f1 in Node 1) my_interface_name: "ens1f1" arp_disable: true dpdk: eal_init: ["-c", "0xff", "-n", "4", "-w", "03:00.1","--proc-type=auto"] || \/ PCI address of ens1f1 in Node 1 # vim: set tabstop=2 shiftwidth=2 ``` # Running the code Running test files: Edit the `CONFIG_PATH` in `Makefile` with the absolute path of your config file: Node 0 → ``` export CONFIG_PATH ?= /users/user/demikernel/scripts/config/cl_node_0.yaml ``` Node 1 → ``` export CONFIG_PATH ?= /users/user/demikernel/scripts/config/cl_node_1.yaml ``` Run the following test scripts to make sure everything works Node 0 (Running Server) → ``` user@node0:~/demikernel$ PEER=server TIMEOUT=300 TEST=udp_ping_pong sudo -E make test-catnip ``` Node 1 (Running Client) → ``` user@node1:~/demikernel$ PEER=client TIMEOUT=300 TEST=udp_ping_pong sudo -E make test-catnip ```