Difference between revisions of "EJFAT UDP Single Thread Packet Sending and Receiving"
(Created page with "= Transmission between ejfat-2 and U280 on ejfat-1 (Sep 2022) = <font size="+1">We can find the NUMA node number of ejfat-2's NIC by looking at the output of:</font> <pre...") |
|||
(4 intermediate revisions by the same user not shown) | |||
Line 1: | Line 1: | ||
− | = Transmission between ejfat-2 and | + | = Transmission between sender on ejfat-2 and the LB on ejfat-1 and from there to receiver on ejfat-1(Sep 2022) = |
− | <font size="+1"> | + | <font size="+1">Here we test the data rate between a single threaded UDP sender and a single threaded receiver. Data was sent from ejfat-2 to LB on ejfat-1 (172.19.22.241), using</font> |
− | |||
− | |||
<pre> | <pre> | ||
− | + | ./packetBlaster -p 19522 -host 172.19.22.241 -mtu 9000 -s 25000000 -b 100000 -cores 80 | |
</pre> | </pre> | ||
− | + | <font size="+1">in which the UDP Send buffer = 50MB and the app sent buffers of 100kB. The receiver was run as:</font> | |
− | |||
− | <font size="+1"> | ||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
<pre> | <pre> | ||
− | ./ | + | ./packetBlastee -p 17750 -b 400000 -r 25000000 -cores 80 |
</pre> | </pre> | ||
− | |||
+ | <font size="+1">The sending and receiving threads were pinned to core #80. This is because cores 80-87 are on the same NUMA node as the NIC and perform by far the best when transferring data. When the same test was run pinning the threads to core #1 (on worst performing node) the max transfer rate was 1000 MB/s. After that packets were constantly being dropped.</font> | ||
− | + | <font size="+1">The following graph shows the CPU usage of both sender and receiver as a function of the data rate.</font> | |
− | [[File: | + | [[File: UdpSingleThdSendAndReceive.png|1000px|link=|"Data Source Stream Processing"]] |
− | <font size="+2"> | + | <font size="+2">Observations</font><br> |
− | <font size="+1">Notice | + | <font size="+1">Notice the strange retrograde behavior. I have no idea what is going on behind the scenes to produce this effect. Note that there were some packets dropped. At 4500 MB/s 700 were dropped and at 4800 MB/s, 360 were dropped</font> |
Latest revision as of 17:12, 12 September 2022
Transmission between sender on ejfat-2 and the LB on ejfat-1 and from there to receiver on ejfat-1(Sep 2022)
Here we test the data rate between a single threaded UDP sender and a single threaded receiver. Data was sent from ejfat-2 to LB on ejfat-1 (172.19.22.241), using
./packetBlaster -p 19522 -host 172.19.22.241 -mtu 9000 -s 25000000 -b 100000 -cores 80
in which the UDP Send buffer = 50MB and the app sent buffers of 100kB. The receiver was run as:
./packetBlastee -p 17750 -b 400000 -r 25000000 -cores 80
The sending and receiving threads were pinned to core #80. This is because cores 80-87 are on the same NUMA node as the NIC and perform by far the best when transferring data. When the same test was run pinning the threads to core #1 (on worst performing node) the max transfer rate was 1000 MB/s. After that packets were constantly being dropped.
The following graph shows the CPU usage of both sender and receiver as a function of the data rate.
Observations
Notice the strange retrograde behavior. I have no idea what is going on behind the scenes to produce this effect. Note that there were some packets dropped. At 4500 MB/s 700 were dropped and at 4800 MB/s, 360 were dropped