Difference between revisions of "How to install, build and use XDP related packages"

Revision as of 22:01, 15 November 2023

PAGE UNDER CONSTRUCTION

Getting Started

XDP stands for eXpress Data Path, and eBPF or BPF stands for extended Berkeley Data Filter

There are 2 main libraries that are needed to use XDP sockets: the libxdp library and libbpf library upon which it depends. Although one can load the 2 from separate packages, that is not recommended as this software is changing so quickly that you'll need versions of the 2 which are compatible. I believe the best option is to use the xdp-tools repository which has compatible versions of both. The difficulty is that the xdp-tools repo's makefiles are not setup to install libbpf so some custom changes (quite minimal) are needed to be able to do this. For stability's sake I have forked the repo and made all the necessary modifications.

Future advancements/versions in XDP/BPF will mean that this will need to be redone at some point, so I make note of what was done. Be warned that changes in this code are often checked in which break the build (due to, Ahem!, things like putting headers in different directories and not changing the makefile, Ahem!).

Following are links to a few good places to start learning:

The best place to learn to program is the tutorial:

https://github.com/xdp-project/xdp-tutorial

Forked GitHub repo of XDP/BPF code (with changes to makefiles for libbpf installation):

https://github.com/JeffersonLab/xdp-tools

Original GitHub repo of XDP/BPF code:

https://github.com/xdp-project/xdp-tools

Very short beginner's guide:

https://dev.to/satrobit/absolute-beginner-s-guide-to-bcc-xdp-and-ebpf-47oi

Get and install the XPD/BPF related files

export PREFIX=""
git clone --recurse-submodules https://github.com/JeffersonLab/xdp-tools.git
cd xdp-tools

Before this code can be compiled, you must follow the proper setup procedure to address its dependencies.

Setup instructions are at given in the tutorial, https://github.com/xdp-project/xdp-tutorial.

Go to the setup_dependencies.org link at https://github.com/xdp-project/xdp-tutorial/blob/master/setup_dependencies.org

However, if you want to avoid wading through that, it boils down to:

// (to get bpftool)
sudo apt install linux-tools-common linux-tools-generic
// to get this to build
sudo apt install linux-tools-5.15.0-87-generic
sudo apt install clang llvm libpcap-dev build-essential
sudo apt install linux-headers-$(uname -r)

// xdp-tools needs emacs
sudo apt install emacs

// you will need to use clang 11 for this to work so install and set commands to this version
sudo apt install clang-11 clang-format-11
sudo update-alternatives --install /usr/bin/clang clang /usr/bin/clang-11 100
sudo update-alternatives --install /usr/bin/clang++ clang++ /usr/bin/clang++-11 100
sudo update-alternatives --install /usr/bin/clang-format clang-format /usr/bin/clang-format-11 100
sudo update-alternatives --install /usr/bin/llc llc /usr/bin/llc-11 100

// check to see if this worked by doing
ls -al /usr/bin/clang*
ls -al /etc/alternatives/clang*
ls -al /usr/bin/llc*
ls -al /etc/alternatives/llc*

// now one can do
./configure
make

// for installation

// make sure there is an ending slash "/" on your install dir !!
export DESTDIR=<install dir>/
export LIBDIR=lib
export HDRDIR=include
export MANDIR=share
export SBINDIR=bin
export SCRIPTSDIR=scripts
make install

The above installation will make and install the xdp-loader program into <install dir>/bin.

It can be used (see below) to both load/unload programs and query what programs have been loaded.

Get and install EJFAT's XDP related files

Go through the XDP tutorial & how to change it into EJFAT's XDP repo

git clone https://github.com/JeffersonLab/ejfat-xdp.git
cd ejfat-xdp
mkdir build
cd build
cmake ..
make install

Getting ready to use XDP sockets

Each ejfat node has a Mellanox ConnectX-6 Dx NIC which can handle 2x100Gbps or 1x200Gbps.
The interface name corresponding to this card is enp193s0f1np1. If yours is different, substitute it.
Avoid running XDP code in the skb (generic) mode in which the linux stack is NOT bypassed.
Use the XDP native mode in which the linux network stack is bypassed by placing special code in the kernel's NIC driver.

To do this, the NIC's MTU must not be larger than 1 linux page minus some headers.

On the ejfat nodes the max MTU which still allows native mode is 3498.

sudo ifconfig enp193s0f1np1 mtu 3498

Each linux NIC can have up to 64 XDP sockets associated with it, each associated with a unique queue number of 0-63.

Normally incoming packets are distributed evenly over all the queues. So for example, if we have 1 socket, all network traffic must be then be corralled into one queue, say queue #0, in order for the socket to receive everything. In this case the following must be done for the socket to receive all packets:

// send all UDP IPv4 packets to queue 0
sudo ethtool -N enp193s0f1np1 flow-type udp4 action 0

Here is an alternative way to put everything onto queue 0:

sudo ethtool -L enp193s0f1np1 combined 1

// Undo this with
sudo ethtool -L enp193s0f1np1 combined 63

With multiple data sources, each destined for a separate socket, multiple rules can be setup.

If we have 2 sockets for example, with packets destined for ports 17750 and 18000, then following could be done to send port 17750 traffic to queue 0, and the 18000 traffic to queue 1:

// send port 17750 UDP IPv4 packets to queue 0
sudo ethtool -N enp193s0f1np1 flow-type udp4 dst-port 17750 action 0
// send port 18000 UDP IPv4 packets to queue 1
sudo ethtool -N enp193s0f1np1 flow-type udp4 dst-port 18000 action 1

Here are a couple of commands to administer such rules:

// Show all flow rules
sudo ethtool -u enp193s0f1np1

// Delete rule (rule numbers seen with above command)
sudo ethtool -N enp193s0f1np1 delete <rule #>

Loading our special code into the NIC driver can be done in a number of different ways.

This is one way which works. The code was compiled in the ejfat-xdp repo and stored in

/daqfs/ersap/ejfat-xdp/build/bin/af_ejfat_kern.o

// send all UDP IPv4 packets to queue 0
sudo ethtool -N enp193s0f1np1 flow-type udp4 action 0

// Show all flow rules
sudo ethtool -u enp193s0f1np1

// Delete rule (rule numbers seen with above command)
sudo ethtool -N enp193s0f1np1 delete <rule #>

// NOT NEEDED. Put all incoming packets into 1 queue (perhaps this can be changed later)
sudo ethtool -L enp193s0f1np1 combined 1

// To undo the above:
sudo ethtool -L enp193s0f1np1 combined 63

// Load the kernel NIC driver code
sudo <xdp_install_dir>/bin/xdp-loader load -m native enp193s0f1np1 xdp_carl_kern.o

// Check the NIC to see if code really loaded and in what mode
sudo /daqfs/xdp/xdp-tools/xdp-loader/xdp-loader status

// Remove everything just loaded
sudo /daqfs/xdp/xdp-tools/xdp-loader/xdp-loader unload enp193s0f1np1 --all

//Now run a program that receives packets:
/daqfs/ersap/ejfat-xdp/build/bin/af_xdp_ejfat_user

/---------------------------------------------------------------------------------------------------------------

Notes: modifications made to the xdp-tools repo in order to get everything to build and install properly

modified configure file to change include directory

from
     LIBBPF_INCLUDE_DIR='$(LIB_DIR)/libbpf/src/root/usr/include'
to 
     LIBBPF_INCLUDE_DIR='$(LIB_DIR)/libbpf/src/root/include'

modified lib/Makefile to enable installation of libbpf (get all indents right!)

all: $(SUBDIRS) libxdp   ------>  all: $(SUBDIRS) libxdp libbpf
 
install: libxdp_install  ------>  install: libxdp_install libbpf_install

libbpf: $(OBJECT_LIBBPF) ------>

.PHONY: libbpf_install
libbpf_install: libbpf
       install -m 0755 -d $(DESTDIR)$(HDRDIR)
       $(MAKE) -C libbpf/src install

.PHONY: libbpf
 libbpf: $(OBJECT_LIBBPF)
       @echo; echo "  $@"; $(MAKE) -C $@/src

modified lib/libxdp/Makefile by adding these lines to the "install" target to enable installation of header files

$(Q)install -d -m 0755 $(DESTDIR)$(HDRDIR)/xdp
$(Q)install -m 0644 $(LIB_HEADERS) $(DESTDIR)$(HDRDIR)/xdp