How to install, build and use XDP related packages
Jump to navigation
Jump to search
PAGE UNDER CONSTRUCTION
Getting Started
- XDP stands for eXpress Data Path, and eBPF or BPF stands for extended Berkeley Data Filter
- There are 2 main libraries that are needed to use XDP sockets: the libxdp library and libbpf library upon which it depends. Although one can load the 2 from separate packages, that is not recommended as this software is changing so quickly that you'll need versions of the 2 which are compatible. I believe the best option is to use the xdp-tools repository which has compatible versions of both. The difficulty is that the xdp-tools repo's makefiles are not setup to install libbpf so some custom changes (quite minimal) are needed to be able to do this. For stability's sake I have forked the repo and made all the necessary modifications. Future advancements in xdp or bpf will mean that this will need to be redone at some point, so I make note of what was done.
Installing libbpf
- 1st: install the libelf library
sudo apt-get install -y libelf-dev
- 2nd: get and install the libbpf related files
export DESTDIR=<installation dir> export PREFIX="" git clone https://github.com/libbpf/libbpf.git cd libbpf cd src make install make install_uapi_headers
export DESTDIR=<installation dir> export PREFIX="" git clone https://github.com/xdp-project/xdp-tools.git cd xdp-tools
- Follow the instructions in xdp-tools/README.md file for compilation.
- There is no need to do a git submodule on libbpf since we just installed it above.
- Instructions are at https://github.com/xdp-project/xdp-tutorial , go to the setup_dependencies.org link at
- https://github.com/xdp-project/xdp-tutorial/blob/master/setup_dependencies.org
- This tells us to:
// (to get bpftool) sudo apt install linux-tools-common linux-tools-generic // to get this to build sudo apt install linux-tools-5.15.0-87-generic sudo apt install clang llvm libpcap-dev build-essential sudo apt install linux-headers-$(uname -r) // xdl-tools needs emacs sudo apt install emacs // you will need to use clang 11 for this to work so install and set commands to this version sudo apt install clang-11 clang-format-11 sudo update-alternatives --install /usr/bin/clang clang /usr/bin/clang-11 100 sudo update-alternatives --install /usr/bin/clang++ clang++ /usr/bin/clang++-11 100 sudo update-alternatives --install /usr/bin/clang-format clang-format /usr/bin/clang-format-11 100 sudo update-alternatives --install /usr/bin/llc llc /usr/bin/llc-11 100 // check to see if this worked by doing ls -al /usr/bin/clang* ls -al /etc/alternatives/clang* ls -al /usr/bin/llc* ls -al /etc/alternatives/llc* // now one can do ./configure make // to install export DESTDIR=<install dir> export LIBDIR=lib export HDRDIR=include export MANDIR=share export SBINDIR=bin export SCRIPTSDIR=scripts make install
git clone https://github.com/JeffersonLab/ejfat-xdp.git cd ejfat-xdp mkdir build cd build cmake .. make install
Getting ready to use XDP sockets
- Each ejfat node has a Mellanox ConnectX-6 Dx NIC which can handle 2x100Gbps or 1x200Gbps.
- Avoid running XDP code in the skb (generic) mode in which the linux stack is NOT bypassed.
- Use the XDP native mode in which the linux network stack is bypassed by placing special code in the kernel's NIC driver.
- To do this, the NIC's MTU must not be larger than 1 linux page minus some headers.
- On the ejfat nodes the max MTU which still allows native mode is 3498.
sudo ifconfig enp193s0f1np1 mtu 3498
Loading our special code into the NIC driver can be done in a number of different ways. This is one way which works. The code was compiled in the ejfat-xdp repo and stored in
/daqfs/ersap/ejfat-xdp/build/bin/af_ejfat_kern.o
// send all UDP IPv4 packets to queue 0 sudo ethtool -N enp193s0f1np1 flow-type udp4 action 0 // Show all flow rules sudo ethtool -u enp193s0f1np1 // Delete rule (rule numbers seen with above command) sudo ethtool -N enp193s0f1np1 delete <rule #> // NOT NEEDED. Put all incoming packets into 1 queue (perhaps this can be changed later) sudo ethtool -L enp193s0f1np1 combined 1 // To undo the above: sudo ethtool -L enp193s0f1np1 combined 63 // Load the kernel NIC driver code sudo <xdp_install_dir>/bin/xdp-loader load -m native enp193s0f1np1 xdp_carl_kern.o // Check the NIC to see if code really loaded and in what mode sudo /daqfs/xdp/xdp-tools/xdp-loader/xdp-loader status // Remove everything just loaded sudo /daqfs/xdp/xdp-tools/xdp-loader/xdp-loader unload enp193s0f1np1 --all //Now run a program that receives packets: /daqfs/ersap/ejfat-xdp/build/bin/af_xdp_ejfat_user