Difference between revisions of "How to install, build and use XDP related packages"
Jump to navigation
Jump to search
Line 20: | Line 20: | ||
: There are 2 main libraries that are needed to use XDP sockets: the '''libxdp''' library and '''libbpf''' library upon which it depends. Although one can load the 2 from separate packages, that is not recommended as this software is changing so quickly that you'll need versions of the 2 which are compatible. I believe the best option is to use the '''xdp-tools''' GitHub repository which has compatible versions of both. The difficulty is that the xdp-tools makefiles are not setup to install libbpf so some custom changes (quite minimal) are needed to be able to do this. For stability's sake I have forked the repo and made all the necessary modifications. | : There are 2 main libraries that are needed to use XDP sockets: the '''libxdp''' library and '''libbpf''' library upon which it depends. Although one can load the 2 from separate packages, that is not recommended as this software is changing so quickly that you'll need versions of the 2 which are compatible. I believe the best option is to use the '''xdp-tools''' GitHub repository which has compatible versions of both. The difficulty is that the xdp-tools makefiles are not setup to install libbpf so some custom changes (quite minimal) are needed to be able to do this. For stability's sake I have forked the repo and made all the necessary modifications. | ||
+ | |||
+ | |||
+ | ==== Links ==== | ||
Line 32: | Line 35: | ||
− | + | ==== Get the GitHub repo ==== | |
+ | |||
<blockquote> | <blockquote> | ||
<pre> | <pre> | ||
Line 40: | Line 44: | ||
</pre> | </pre> | ||
</blockquote> | </blockquote> | ||
+ | |||
+ | ==== Host Dependencies ==== | ||
:Before this code can be compiled, you must follow the proper setup procedure to address its dependencies. | :Before this code can be compiled, you must follow the proper setup procedure to address its dependencies. |
Revision as of 19:47, 20 November 2023
PAGE UNDER CONSTRUCTION
Getting Started
- XDP stands for eXpress Data Path, and eBPF or BPF stands for extended Berkeley Data Filter
- Following are links to a few good places to start learning to program with XDP sockets:
- The best place to learn to program is the tutorial:
- There are 2 main libraries that are needed to use XDP sockets: the libxdp library and libbpf library upon which it depends. Although one can load the 2 from separate packages, that is not recommended as this software is changing so quickly that you'll need versions of the 2 which are compatible. I believe the best option is to use the xdp-tools GitHub repository which has compatible versions of both. The difficulty is that the xdp-tools makefiles are not setup to install libbpf so some custom changes (quite minimal) are needed to be able to do this. For stability's sake I have forked the repo and made all the necessary modifications.
Links
- Future advancements/versions in XDP/BPF will mean that this will need to be redone at some point, so here is a note of what was done to make things compile and install:
- Following are links to the xdp-tools repos:
Get the GitHub repo
export PREFIX="" git clone --recurse-submodules https://github.com/JeffersonLab/xdp-tools.git cd xdp-tools
Host Dependencies
- Before this code can be compiled, you must follow the proper setup procedure to address its dependencies.
- Setup instructions are at given in the tutorial, XDP tutorial.
- Go to the setup_dependencies.org link at Setup Dependencies
- However, if you want to avoid wading through that, it boils down to:
// (to get bpftool) sudo apt install linux-tools-common linux-tools-generic // to get this to build sudo apt install linux-tools-5.15.0-87-generic sudo apt install clang llvm libpcap-dev build-essential sudo apt install linux-headers-$(uname -r) // xdp-tools needs emacs sudo apt install emacs // you will need to use clang 11 for this to work so install and set commands to this version sudo apt install clang-11 clang-format-11 sudo update-alternatives --install /usr/bin/clang clang /usr/bin/clang-11 100 sudo update-alternatives --install /usr/bin/clang++ clang++ /usr/bin/clang++-11 100 sudo update-alternatives --install /usr/bin/clang-format clang-format /usr/bin/clang-format-11 100 sudo update-alternatives --install /usr/bin/llc llc /usr/bin/llc-11 100 // check to see if this worked by doing ls -al /usr/bin/clang* ls -al /etc/alternatives/clang* ls -al /usr/bin/llc* ls -al /etc/alternatives/llc* // now one can do ./configure make // for installation // make sure there is an ending slash "/" on your install dir !! export DESTDIR=<install dir>/ export LIBDIR=lib export HDRDIR=include export MANDIR=share export SBINDIR=bin export SCRIPTSDIR=scripts make install
- The above installation will make and install the xdp-loader program into <install dir>/bin.
- It can be used (see below) to both load/unload programs and query what programs have been loaded.
Getting ready to use XDP sockets
- Each ejfat node has a Mellanox ConnectX-6 Dx NIC which can handle 2x100Gbps or 1x200Gbps.
- The interface name corresponding to this card is enp193s0f1np1. If yours is different, substitute it.
- Avoid running XDP code in the skb (generic) mode in which the linux stack is NOT bypassed.
- Use the XDP native mode in which the linux network stack is bypassed by placing special code in the kernel's NIC driver.
- To do this, the NIC's MTU must not be larger than 1 linux page minus some headers.
- On the ejfat nodes the max MTU which still allows native mode is 3498.
sudo ifconfig enp193s0f1np1 mtu 3498
- Each linux NIC can have up to 64 XDP sockets associated with it, each associated with a unique queue number of 0-63.
- Normally incoming packets are distributed evenly over all the queues. So for example, if we have 1 socket, all network traffic must be then be corralled into one queue, say queue #0, in order for the socket to receive everything. In this case the following must be done for the socket to receive all packets:
// send all UDP IPv4 packets to queue 0 sudo ethtool -N enp193s0f1np1 flow-type udp4 action 0
- Here is an alternative way to put everything onto queue 0:
sudo ethtool -L enp193s0f1np1 combined 1 // Check status of combining queues sudo ethtool -L enp193s0f1np1 // Undo this with sudo ethtool -L enp193s0f1np1 combined 63
- With multiple data sources, each destined for a separate socket, multiple rules can be setup.
- If we have 2 sockets for example, with packets destined for ports 17750 and 18000, then following could be done to send port 17750 traffic to queue 0, and the 18000 traffic to queue 1:
// send port 17750 UDP IPv4 packets to queue 0 sudo ethtool -N enp193s0f1np1 flow-type udp4 dst-port 17750 action 0 // send port 18000 UDP IPv4 packets to queue 1 sudo ethtool -N enp193s0f1np1 flow-type udp4 dst-port 18000 action 1
- Here are a couple of commands to administer such rules:
// Show all flow rules sudo ethtool -u enp193s0f1np1 // Delete rule (rule numbers seen with above command) sudo ethtool -N enp193s0f1np1 delete <rule #>
git clone https://github.com/JeffersonLab/ejfat-xdp.git cd ejfat-xdp mkdir build cd build cmake .. make install
- Loading our special code into the NIC driver can be done in a number of different ways.
- This is one way which works. The code was compiled in the ejfat-xdp repo and stored in
- /daqfs/ersap/ejfat-xdp/build/bin/af_ejfat_kern.o
// Load the kernel NIC driver code sudo <xdp_install_dir>/bin/xdp-loader load -m native enp193s0f1np1 xdp_carl_kern.o // Check the NIC to see if code really loaded and in what mode sudo /daqfs/xdp/xdp-tools/xdp-loader/xdp-loader status // Remove everything just loaded sudo /daqfs/xdp/xdp-tools/xdp-loader/xdp-loader unload enp193s0f1np1 --all //Now run a program that receives packets: /daqfs/ersap/ejfat-xdp/build/bin/af_xdp_ejfat_user