EJFAT UDP General Performance Considerations

Here are a few things to ponder. I'll go over some things I've done to try to speed up performance so that those who follow won't waste their time. Here are some interesting links:

Scaling in the Linux Networking Stack

NIC packet reception

How to receive 1M pkts/sec

NIC queues on multi-cpu nodes

Contemporary NICs support multiple receive and transmit descriptor queues (Receive Side Scaling or RSS). On reception a NIC distributes packets by applying a filter to each that assigns it to one of a number of logical flows. Packets for each flow are steered to a separate receive queue, which in turn can be processed by a separate CPU. The goal of this is to increase performance. Find out how many NIC queues there are on your node by looking at the combined property:

// See how many queues there are 
sudo ethtool -l enp193s0f1np1

// See how big the queue sizes are
sudo ethtool -g enp193s0f1np1

// Make the receiving queues the max size
sudo ethtool -G enp193s0f1np1 rx 8192

The filter used is typically a hash function over the network and/or transport layer headers. Typically and for ejfat nodes this is a 4-tuple hash over IP addresses and ports of a packet. The most common implementation uses an indirection table (256 entries for ejfat nodes) where each entry stores a queue number. The receive queue for a packet is determined by masking out the low order seven bits of the computed hash for the packet (usually a Toeplitz hash), taking this number as a key into the indirection table and reading the corresponding value.

// See if hashing is enabled
sudo ethtool -k enp193s0f1np1 | grep hashing

// Print out the indirection table to see how packets are distributed to Qs
sudo ethtool -x enp193s0f1np1

It's also possible to steer packets by modifying the hash being used:

// See the details of the hash algorithm
sudo ethtool -n enp193s0f1np1 rx-flow-hash udp4

// Change hashing to only destination port (slows things down if using 63 queues)
sudo ethtool -N enp193s0f1np1 rx-flow-hash n

// Change hashing to back to 4-tuple
sudo ethtool -N enp193s0f1np1 rx-flow-hash sdfn

There are other filters that can be specified, rules on which packets go to which queues. For example packets destined for a specific port can be sent to a fixed queue:

// send port 17750 UDP IPv4 packets to queue #7
sudo ethtool -N enp193s0f1np1 flow-type udp4 dst-port 17750 queue 7

Effect of NIC queues on UDP transmission

Change # of queues

Changing the number of queues has a major effect on performance. To start with, let's look at one packet sender and one receiver.

Changing hash and queue size

Changing the hash algorithm and queue sizes made little difference in performance.

Effect of packet receiving functions

Using recvmsg (found only in linux) in order to receive arrays of messages in one system call appears to be able to possibly speeding things up; however, in practice it only slows things down. : Here's tip for programming with recvmmsg, set the timeout to nullptr or the function only returns 1 packet at a time! Set flag = MSG_WAITALL;