Hall-B to EJFat transfer speeds
Jump to navigation
Jump to search
2023-12-14
Beam down for a couple of hours so take advantage to run some bandwidth tests between Hall-B and EJFat in CC.
Some details are captured below but here is a summary:
- iperf3 does not report rates as high as iperf(2)
- (I copied /usr/bin/iperf from efat-fs-daq to a temporary directory on clonfarm21sro to use it)
- Can achieve 99Gbps between ejfat-2 and ejfat-fs-daq
- Can achieve 75Gbps between clonfarm21sro and ejfat-fs-daq
- n.b. 75Gbps is a reasonably reliable average, but it fluctuates. The highest I saw was just over 80Gbps, but not sustained.
- UDP is very limited (~1Mbps/thread). Possibly due to maximum allowed packet size on the system.
- Routing from ejfat-fs-daq to clonfarm21sro is broken so only connections initiated from clonfarm21sro can be made at the moment.
TCP
clonfarm21:~>traceroute ejfat-fs-daq traceroute to ejfat-fs-daq (129.57.177.7), 30 hops max, 60 byte packets 1 gw-69.jlab.org (129.57.69.1) 0.169 ms 0.202 ms 0.226 ms 2 * * * 3 ejfat-fs-daq.jlab.org (129.57.177.7) 0.174 ms 0.157 ms *
n.b. this fails to find a route to clonfarm21sro from ejfat-fs-daq
davidl@ejfat-fs:~$ traceroute clonfarm21sro traceroute to clonfarm21sro (129.57.69.11), 30 hops max, 60 byte packets 1 _gateway (129.57.177.129) 0.154 ms 0.156 ms 0.137 ms 2 hallb-ejfat-rtr-2023-172-31-28-29.jlab.org (172.31.28.29) 0.179 ms 0.211 ms 0.243 ms 3 * * * 4 * * * 5 * * *
Sending data via iperf3 to ejfat-fs-daq cannot seem to go faster than ~4Gbps. Same results for 1 thread and 8 threads:
Connecting to host ejfat-fs-daq, port 5201 [ 5] local 129.57.69.11 port 41206 connected to 129.57.177.7 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-2.00 sec 924 MBytes 3.88 Gbits/sec 0 683 KBytes [ 5] 2.00-4.00 sec 895 MBytes 3.75 Gbits/sec 0 683 KBytes [ 5] 4.00-6.00 sec 889 MBytes 3.73 Gbits/sec 7 543 KBytes [ 5] 6.00-7.05 sec 464 MBytes 3.70 Gbits/sec 0 543 KBytes clonfarm21:~>iperf3 -c ejfat-fs-daq -t3600 -i2 -P8 Connecting to host ejfat-fs-daq, port 5201 [ 5] local 129.57.69.11 port 52242 connected to 129.57.177.7 port 5201 [ 7] local 129.57.69.11 port 52252 connected to 129.57.177.7 port 5201 [ 9] local 129.57.69.11 port 52258 connected to 129.57.177.7 port 5201 [ 11] local 129.57.69.11 port 52268 connected to 129.57.177.7 port 5201 [ 13] local 129.57.69.11 port 52276 connected to 129.57.177.7 port 5201 [ 15] local 129.57.69.11 port 52286 connected to 129.57.177.7 port 5201 [ 17] local 129.57.69.11 port 52300 connected to 129.57.177.7 port 5201 [ 19] local 129.57.69.11 port 52308 connected to 129.57.177.7 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-2.00 sec 121 MBytes 507 Mbits/sec 0 375 KBytes [ 7] 0.00-2.00 sec 122 MBytes 514 Mbits/sec 0 546 KBytes [ 9] 0.00-2.00 sec 120 MBytes 503 Mbits/sec 0 335 KBytes [ 11] 0.00-2.00 sec 120 MBytes 504 Mbits/sec 0 324 KBytes [ 13] 0.00-2.00 sec 120 MBytes 502 Mbits/sec 0 294 KBytes [ 15] 0.00-2.00 sec 120 MBytes 503 Mbits/sec 0 313 KBytes [ 17] 0.00-2.00 sec 120 MBytes 502 Mbits/sec 0 308 KBytes [ 19] 0.00-2.00 sec 120 MBytes 502 Mbits/sec 0 314 KBytes [SUM] 0.00-2.00 sec 963 MBytes 4.04 Gbits/sec 0 - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 2.00-4.00 sec 107 MBytes 448 Mbits/sec 0 375 KBytes [ 7] 2.00-4.00 sec 108 MBytes 451 Mbits/sec 0 546 KBytes [ 9] 2.00-4.00 sec 108 MBytes 451 Mbits/sec 0 335 KBytes [ 11] 2.00-4.00 sec 108 MBytes 451 Mbits/sec 0 324 KBytes [ 13] 2.00-4.00 sec 108 MBytes 451 Mbits/sec 0 294 KBytes [ 15] 2.00-4.00 sec 107 MBytes 450 Mbits/sec 0 313 KBytes [ 17] 2.00-4.00 sec 108 MBytes 453 Mbits/sec 0 308 KBytes [ 19] 2.00-4.00 sec 107 MBytes 450 Mbits/sec 0 314 KBytes [SUM] 2.00-4.00 sec 860 MBytes 3.61 Gbits/sec 0 - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 4.00-6.00 sec 110 MBytes 461 Mbits/sec 0 375 KBytes [ 7] 4.00-6.00 sec 110 MBytes 461 Mbits/sec 0 546 KBytes [ 9] 4.00-6.00 sec 109 MBytes 459 Mbits/sec 0 335 KBytes [ 11] 4.00-6.00 sec 109 MBytes 459 Mbits/sec 0 324 KBytes [ 13] 4.00-6.00 sec 110 MBytes 460 Mbits/sec 0 294 KBytes [ 15] 4.00-6.00 sec 110 MBytes 461 Mbits/sec 0 313 KBytes [ 17] 4.00-6.00 sec 109 MBytes 457 Mbits/sec 0 308 KBytes [ 19] 4.00-6.00 sec 109 MBytes 458 Mbits/sec 0 314 KBytes [SUM] 4.00-6.00 sec 876 MBytes 3.68 Gbits/sec 0
Setting the window size higher does get faster rate. Note that if I set it higher than 416K it will return an error that the window size is not set correctly.
clonfarm21:~>iperf3 -c ejfat-fs-daq -t3600 -i2 -w 416K Connecting to host ejfat-fs-daq, port 5201 [ 5] local 129.57.69.11 port 47358 connected to 129.57.177.7 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-2.00 sec 4.15 GBytes 17.8 Gbits/sec 0 436 KBytes [ 5] 2.00-4.00 sec 4.17 GBytes 17.9 Gbits/sec 1 436 KBytes [ 5] 4.00-6.00 sec 4.16 GBytes 17.9 Gbits/sec 0 436 KBytes
The default buffer length is 128k and the maximum size it allows you to set to is 1024k. This with the maximum window width gives the fastest rate of ~20Gbps. Note that THIS IS THAT SAME WHEN RUNNING THE CLIENT ON EJFAT-2!.
clonfarm21:~>iperf3 -c ejfat-fs-daq -t3600 -i2 -w 416K -l 1024K Connecting to host ejfat-fs-daq, port 5201 [ 5] local 129.57.69.11 port 36066 connected to 129.57.177.7 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-2.00 sec 4.68 GBytes 20.1 Gbits/sec 0 430 KBytes [ 5] 2.00-4.00 sec 4.66 GBytes 20.0 Gbits/sec 0 430 KBytes [ 5] 4.00-6.00 sec 4.70 GBytes 20.2 Gbits/sec 0 430 KBytes
I am able to get faster transfer speeds using iperf (=iperf2) than I can with iperf3. Running this command on ejfat-2 gives full bandwidth (up to 99Gbps). Running on clonfarm21 though only gets up to around 50Gbps. Note that I edited out individual thread lines from the following for brevity.
clonfarm21:~/tmp>./iperf -c ejfat-fs-daq -t 3600 -i4 -P8 -w 500k ------------------------------------------------------------ Client connecting to ejfat-fs-daq, TCP port 5001 TCP window size: 416 KByte (WARNING: requested 488 KByte) ------------------------------------------------------------ [ ID] Interval Transfer Bandwidth [SUM] 0.0- 4.0 sec 18.4 GBytes 39.6 Gbits/sec [SUM] 4.0- 8.0 sec 20.1 GBytes 43.1 Gbits/sec [SUM] 8.0-12.0 sec 22.3 GBytes 47.9 Gbits/sec [SUM] 12.0-16.0 sec 25.3 GBytes 54.4 Gbits/sec [SUM] 16.0-20.0 sec 25.1 GBytes 53.9 Gbits/sec [SUM] 20.0-24.0 sec 25.0 GBytes 53.8 Gbits/sec [SUM] 24.0-28.0 sec 25.3 GBytes 54.4 Gbits/sec [SUM] 28.0-32.0 sec 25.1 GBytes 54.0 Gbits/sec [SUM] 0.0-32.8 sec 192 GBytes 50.2 Gbits/sec
Increasing the number of threads and playing with the window and buffer sizes gets up to around 75Gbps (avg)
clonfarm21:~/tmp>./iperf -c ejfat-fs-daq -t 3600 -i4 -P24 -w 416K -l512K ------------------------------------------------------------ Client connecting to ejfat-fs-daq, TCP port 5001 TCP window size: 416 KByte ------------------------------------------------------------ [ ID] Interval Transfer Bandwidth [SUM] 0.0- 4.0 sec 32.9 GBytes 70.7 Gbits/sec [SUM] 4.0- 8.0 sec 36.5 GBytes 78.4 Gbits/sec [SUM] 8.0-12.0 sec 35.5 GBytes 76.2 Gbits/sec [SUM] 12.0-16.0 sec 36.8 GBytes 79.1 Gbits/sec [SUM] 16.0-20.0 sec 35.1 GBytes 75.4 Gbits/sec [SUM] 20.0-24.0 sec 36.2 GBytes 77.7 Gbits/sec [SUM] 24.0-28.0 sec 33.9 GBytes 72.8 Gbits/sec [SUM] 0.0-29.4 sec 259 GBytes 75.7 Gbits/sec
UDP
Sending via UDP is severely limited. Buffer lengths must be kept below the maximum allowed (<64KB) or they will not be sent. It also seems to be limited to ~1Mbps/thread (excluding the first set of packets which always register ~1.25Mbps). Running with 64 threads leads to a steady state of only ~65Mbps aggregate.
./iperf -c ejfat-fs-daq -t 3600 -i4 -l62K -u -P64 ------------------------------------------------------------ Client connecting to ejfat-fs-daq, UDP port 5001 Sending 63488 byte datagrams, IPG target: 484375.00 us (kalman adjust) UDP buffer size: 208 KByte (default) ------------------------------------------------------------ [SUM] 0.0- 4.0 sec 38.8 MBytes 81.3 Mbits/sec [SUM] 4.0- 8.0 sec 31.0 MBytes 65.0 Mbits/sec