Difference between revisions of "Hall-B to EJFat transfer speeds"
Jump to navigation
Jump to search
| Line 79: | Line 79: | ||
[ 5] 2.00-4.00 sec 4.66 GBytes 20.0 Gbits/sec 0 430 KBytes | [ 5] 2.00-4.00 sec 4.66 GBytes 20.0 Gbits/sec 0 430 KBytes | ||
[ 5] 4.00-6.00 sec 4.70 GBytes 20.2 Gbits/sec 0 430 KBytes | [ 5] 4.00-6.00 sec 4.70 GBytes 20.2 Gbits/sec 0 430 KBytes | ||
| + | |||
| + | I am able to get faster transfer speeds using iperf (=iperf2) than I can with iperf3. Running this command on ejfat-2 gives full bandwidth (up to 99Gbps). Running on clonfarm21 though only gets up to around 50Gbps. Note that I edited out individual thread lines from the following for brevity. | ||
| + | |||
| + | clonfarm21:~/tmp>./iperf -c ejfat-fs-daq -t 3600 -i4 -P8 -w 500k | ||
| + | ------------------------------------------------------------ | ||
| + | Client connecting to ejfat-fs-daq, TCP port 5001 | ||
| + | TCP window size: 416 KByte (WARNING: requested 488 KByte) | ||
| + | ------------------------------------------------------------ | ||
| + | [ ID] Interval Transfer Bandwidth | ||
| + | [SUM] 0.0- 4.0 sec 18.4 GBytes 39.6 Gbits/sec | ||
| + | [SUM] 4.0- 8.0 sec 20.1 GBytes 43.1 Gbits/sec | ||
| + | [SUM] 8.0-12.0 sec 22.3 GBytes 47.9 Gbits/sec | ||
| + | [SUM] 12.0-16.0 sec 25.3 GBytes 54.4 Gbits/sec | ||
| + | [SUM] 16.0-20.0 sec 25.1 GBytes 53.9 Gbits/sec | ||
| + | [SUM] 20.0-24.0 sec 25.0 GBytes 53.8 Gbits/sec | ||
| + | [SUM] 24.0-28.0 sec 25.3 GBytes 54.4 Gbits/sec | ||
| + | [SUM] 28.0-32.0 sec 25.1 GBytes 54.0 Gbits/sec | ||
| + | [SUM] 0.0-32.8 sec 192 GBytes 50.2 Gbits/sec | ||
Revision as of 12:40, 14 December 2023
2023-12-14
- Beam down for a couple of hours so take advantage to run some bandwidth tests between Hall-B and EJFat in CC
clonfarm21:~>traceroute ejfat-fs-daq traceroute to ejfat-fs-daq (129.57.177.7), 30 hops max, 60 byte packets 1 gw-69.jlab.org (129.57.69.1) 0.169 ms 0.202 ms 0.226 ms 2 * * * 3 ejfat-fs-daq.jlab.org (129.57.177.7) 0.174 ms 0.157 ms *
Sending data via iperf3 to ejfat-fs-daq cannot seem to go faster than ~4Gbps. Same results for 1 thread and 8 threads:
Connecting to host ejfat-fs-daq, port 5201 [ 5] local 129.57.69.11 port 41206 connected to 129.57.177.7 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-2.00 sec 924 MBytes 3.88 Gbits/sec 0 683 KBytes [ 5] 2.00-4.00 sec 895 MBytes 3.75 Gbits/sec 0 683 KBytes [ 5] 4.00-6.00 sec 889 MBytes 3.73 Gbits/sec 7 543 KBytes [ 5] 6.00-7.05 sec 464 MBytes 3.70 Gbits/sec 0 543 KBytes clonfarm21:~>iperf3 -c ejfat-fs-daq -t3600 -i2 -P8 Connecting to host ejfat-fs-daq, port 5201 [ 5] local 129.57.69.11 port 52242 connected to 129.57.177.7 port 5201 [ 7] local 129.57.69.11 port 52252 connected to 129.57.177.7 port 5201 [ 9] local 129.57.69.11 port 52258 connected to 129.57.177.7 port 5201 [ 11] local 129.57.69.11 port 52268 connected to 129.57.177.7 port 5201 [ 13] local 129.57.69.11 port 52276 connected to 129.57.177.7 port 5201 [ 15] local 129.57.69.11 port 52286 connected to 129.57.177.7 port 5201 [ 17] local 129.57.69.11 port 52300 connected to 129.57.177.7 port 5201 [ 19] local 129.57.69.11 port 52308 connected to 129.57.177.7 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-2.00 sec 121 MBytes 507 Mbits/sec 0 375 KBytes [ 7] 0.00-2.00 sec 122 MBytes 514 Mbits/sec 0 546 KBytes [ 9] 0.00-2.00 sec 120 MBytes 503 Mbits/sec 0 335 KBytes [ 11] 0.00-2.00 sec 120 MBytes 504 Mbits/sec 0 324 KBytes [ 13] 0.00-2.00 sec 120 MBytes 502 Mbits/sec 0 294 KBytes [ 15] 0.00-2.00 sec 120 MBytes 503 Mbits/sec 0 313 KBytes [ 17] 0.00-2.00 sec 120 MBytes 502 Mbits/sec 0 308 KBytes [ 19] 0.00-2.00 sec 120 MBytes 502 Mbits/sec 0 314 KBytes [SUM] 0.00-2.00 sec 963 MBytes 4.04 Gbits/sec 0 - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 2.00-4.00 sec 107 MBytes 448 Mbits/sec 0 375 KBytes [ 7] 2.00-4.00 sec 108 MBytes 451 Mbits/sec 0 546 KBytes [ 9] 2.00-4.00 sec 108 MBytes 451 Mbits/sec 0 335 KBytes [ 11] 2.00-4.00 sec 108 MBytes 451 Mbits/sec 0 324 KBytes [ 13] 2.00-4.00 sec 108 MBytes 451 Mbits/sec 0 294 KBytes [ 15] 2.00-4.00 sec 107 MBytes 450 Mbits/sec 0 313 KBytes [ 17] 2.00-4.00 sec 108 MBytes 453 Mbits/sec 0 308 KBytes [ 19] 2.00-4.00 sec 107 MBytes 450 Mbits/sec 0 314 KBytes [SUM] 2.00-4.00 sec 860 MBytes 3.61 Gbits/sec 0 - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 4.00-6.00 sec 110 MBytes 461 Mbits/sec 0 375 KBytes [ 7] 4.00-6.00 sec 110 MBytes 461 Mbits/sec 0 546 KBytes [ 9] 4.00-6.00 sec 109 MBytes 459 Mbits/sec 0 335 KBytes [ 11] 4.00-6.00 sec 109 MBytes 459 Mbits/sec 0 324 KBytes [ 13] 4.00-6.00 sec 110 MBytes 460 Mbits/sec 0 294 KBytes [ 15] 4.00-6.00 sec 110 MBytes 461 Mbits/sec 0 313 KBytes [ 17] 4.00-6.00 sec 109 MBytes 457 Mbits/sec 0 308 KBytes [ 19] 4.00-6.00 sec 109 MBytes 458 Mbits/sec 0 314 KBytes [SUM] 4.00-6.00 sec 876 MBytes 3.68 Gbits/sec 0
Setting the window size higher does get faster rate. Note that if I set it higher than 416K it will return an error that the window size is not set correctly.
clonfarm21:~>iperf3 -c ejfat-fs-daq -t3600 -i2 -w 416K Connecting to host ejfat-fs-daq, port 5201 [ 5] local 129.57.69.11 port 47358 connected to 129.57.177.7 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-2.00 sec 4.15 GBytes 17.8 Gbits/sec 0 436 KBytes [ 5] 2.00-4.00 sec 4.17 GBytes 17.9 Gbits/sec 1 436 KBytes [ 5] 4.00-6.00 sec 4.16 GBytes 17.9 Gbits/sec 0 436 KBytes
The default buffer length is 128k and the maximum size it allows you to set to is 1024k. This with the maximum window width gives the fastest rate of ~20Gbps. Note that THIS IS THAT SAME WHEN RUNNING THE CLIENT ON EJFAT-2!.
clonfarm21:~>iperf3 -c ejfat-fs-daq -t3600 -i2 -w 416K -l 1024K Connecting to host ejfat-fs-daq, port 5201 [ 5] local 129.57.69.11 port 36066 connected to 129.57.177.7 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-2.00 sec 4.68 GBytes 20.1 Gbits/sec 0 430 KBytes [ 5] 2.00-4.00 sec 4.66 GBytes 20.0 Gbits/sec 0 430 KBytes [ 5] 4.00-6.00 sec 4.70 GBytes 20.2 Gbits/sec 0 430 KBytes
I am able to get faster transfer speeds using iperf (=iperf2) than I can with iperf3. Running this command on ejfat-2 gives full bandwidth (up to 99Gbps). Running on clonfarm21 though only gets up to around 50Gbps. Note that I edited out individual thread lines from the following for brevity.
clonfarm21:~/tmp>./iperf -c ejfat-fs-daq -t 3600 -i4 -P8 -w 500k ------------------------------------------------------------ Client connecting to ejfat-fs-daq, TCP port 5001 TCP window size: 416 KByte (WARNING: requested 488 KByte) ------------------------------------------------------------ [ ID] Interval Transfer Bandwidth [SUM] 0.0- 4.0 sec 18.4 GBytes 39.6 Gbits/sec [SUM] 4.0- 8.0 sec 20.1 GBytes 43.1 Gbits/sec [SUM] 8.0-12.0 sec 22.3 GBytes 47.9 Gbits/sec [SUM] 12.0-16.0 sec 25.3 GBytes 54.4 Gbits/sec [SUM] 16.0-20.0 sec 25.1 GBytes 53.9 Gbits/sec [SUM] 20.0-24.0 sec 25.0 GBytes 53.8 Gbits/sec [SUM] 24.0-28.0 sec 25.3 GBytes 54.4 Gbits/sec [SUM] 28.0-32.0 sec 25.1 GBytes 54.0 Gbits/sec [SUM] 0.0-32.8 sec 192 GBytes 50.2 Gbits/sec