How to install, build and use XDP related packages

PAGE UNDER CONSTRUCTION

There are multiple software packages that need to be downloaded, compiled, and installed for XPD-related code to run

Installing libbpf

1st: install the libelf library

sudo apt-get install -y libelf-dev

2nd: get and install the libbpf related files

export DESTDIR=<installation dir>
export PREFIX=""
git clone https://github.com/libbpf/libbpf.git
cd libbpf
cd src
make install
make install_uapi_headers

Get and install the libxpd related files

export DESTDIR=<installation dir>
export PREFIX=""
git clone https://github.com/xdp-project/xdp-tools.git
cd xdp-tools

Follow the instructions in xdp-tools/README.md file for compilation.

There is no need to do a git submodule on libbpf since we just installed it above.

Instructions are at https://github.com/xdp-project/xdp-tutorial , go to the setup_dependencies.org link at

https://github.com/xdp-project/xdp-tutorial/blob/master/setup_dependencies.org

This tells us to:

// (to get bpftool)
sudo apt install linux-tools-common linux-tools-generic
sudo apt install clang llvm libpcap-dev build-essential
sudo apt install linux-headers-$(uname -r)

// xdl-tools needs emacs
sudo apt install emacs

// you will need to use clang 11 for this to work so install and set commands to this version
sudo apt install clang-11 clang-format-11
sudo update-alternatives --install /usr/bin/clang clang /usr/bin/clang-11 100
sudo update-alternatives --install /usr/bin/clang++ clang++ /usr/bin/clang++-11 100
sudo update-alternatives --install /usr/bin/clang-format clang-format /usr/bin/clang-format-11 100
sudo update-alternatives --install /usr/bin/llc llc /usr/bin/llc-11 100

// check to see if this worked by doing
ls -al /usr/bin/clang*
ls -al /etc/alternatives/clang*
ls -al /usr/bin/llc*
ls -al /etc/alternatives/llc*

// now one can do
./configure
make

// to install
export DESTDIR=<install dir>
export LIBDIR=lib
export HDRDIR=include
export MANDIR=share
export SBINDIR=bin
export SCRIPTSDIR=scripts
make install

Get and install ejfat's XDP related files

git clone https://github.com/JeffersonLab/ejfat-xdp.git
cd ejfat-xdp
mkdir build
cd build
cmake ..
make install

Getting ready to use XDP sockets

• Each ejfat node has a Mellanox ConnectX-6 Dx NIC which can handle 2x100Gbps or 1x200Gbps.
• Avoid running XDP code in the skb (generic) mode in which the linux stack is NOT bypassed.
• Use the XDP native mode in which the linux network stack is bypassed by placing special code in the kernel's NIC driver.

To do this, the NIC's MTU must not be larger than 1 linux page minus some headers.

On the ejfat nodes the max MTU which still allows native mode is 3498.

sudo ifconfig enp193s0f1np1 mtu 3498

Loading our special code into the NIC driver can be done in a number of different ways. This is one way which works. The code was compiled in the ejfat-xdp repo and stored in

/daqfs/ersap/ejfat-xdp/build/bin/af_ejfat_kern.o

// send all UDP IPv4 packets to queue 0
sudo ethtool -N enp193s0f1np1 flow-type udp4 action 0

// Put all incoming packets into 1 queue (perhaps this can be changed later)
sudo ethtool -L enp193s0f1np1 combined 1

Now run a program that receives packets: /daqfs/ersap/ejfat-xdp/build/bin/af_xdp_ejfat_user