EJFAT
(ESnet / JLaB FPGA Accelerated Transport)
EJFAT is a collaboration between Energy Sciences Network (ESnet) and Thomas Jefferson National Laboratory (JLab) for proof of concept engineering for accelerated load balancer (LB) using dynamic IP4/6 address forwarding. Dynamic because the forwarding address is chosen dynamically from a collection of destination endpoints based on near real-time destination workload conditions, and accelerated because the forwarding is accomplished with low fixed latency at line rates of up to 200Gbps per FPGA, where in general a functioning LB may consist of up to four FPGAs acting as one logical DP for a total bandwidth capacity of over 1 Tbps. The low, fixed latency is achieved by utilization of an appropriately programmed Field Programmable Gate Array (FPGA) to effect the Data Plane (DP) functions of the LB.
EJFAT System Status
ejfat-1
- 100Gbps NIC: ejfat-1-daq 129.57.177.8
- 10Gbps NIC: ejfat-1 129.57.177.131
- U280 FPGA: ejfat-1-dp 129.57.177.{9-12} - LAG'd for 200Gbps
- LB CP: ejfat-1 129.57.177.131, latest Stable branch
- LB: DP latest Stable FW
ejfat-2
- 100Gbps NIC: ejfat-2-daq 129.57.177.2
- 10Gbps NIC: ejfat-2 129.57.177.132
- 100Gbps U280 FPGA: ejfat-2-dp 129.57.177.{17-20}
- LB CP: ejfat-2 129.57.177.132, latest Stable branch
- LB: DP latest Stable FW
ejfat-3
- 200Gbps NIC: ejfat-3-daq 129.57.177.3
- 10Gbps NIC: ejfat-3 129.57.177.133
- Two U280s installed - LAG'd for 400Gbps
- FW Containers built by Stacey
ejfat-4
- 100Gbps NIC: ejfat-4-daq 129.57.177.4
- 10Gbps NIC: ejfat-4 129.57.177.134
- XDP experiments
- 100Gbps U280 FPGA: ejfat-4-dp 129.57.177.{41-44}
- LB CP: ejfat-4 129.57.177.134, latest Stable branch
- LB: DP latest Stable FW
ejfat-5
- 200Gbps NIC: ejfat-5-daq 129.57.177.5
- 10Gbps NIC: ejfat-5 129.57.177.135
- LB CP: ejfat-5 129.57.177.135, latest Stable branch
- 100Gbps U280 FPGA: ejfat-5-dp 129.57.177.{49-52}
- LB: DP latest Stable FW
- Optical Taps Installed
ejfat-6
- 200Gbps NIC: ejfat-6-daq 129.57.177.6
- 10Gbps NIC: ejfat-6 129.57.177.136
- DAOS experiments
- Using Ubuntu 24.04 LTS
- FW containers built
- Waiting for podman compose installation
ejfat-fs
- 100Gbps NIC: ejfat-fs-daq 129.57.177.7
- 10Gbps NIC: ejfat-fs 129.57.177.130
- Hosts NVME memory/disk
- 100Gbps U280 FPGA: ejfat-fs-dp 129.57.177.{65-68}
- LB CP: ejfat-fs 129.57.177.130, latest Stable branch
- LB: DP latest Stable FW
Presentations/Papers
| date | presenter | Event | links |
|---|---|---|---|
| 2021-03-01 | G. Heyes | EJFAT Proposal | Word |
| 2021-10-21 | M. S. Goodrich | Div Brief | |
| 2021-11-05 | M. S. Goodrich | Canisius College | |
| 2021-12-03 | S. Sheldon | ESnet LB Tutorial | MP4 |
| 2021-12-10 | Y. Kumar | SRO iX Presentation | PPTX |
| 2022-08-05 | M. S. Goodrich | RT-2022 Presentation | PPTX |
| 2022-08-05 | M. S. Goodrich, et al. | RT-2022 Proceedings | |
| 2022-10-20 | S. Sheldon, et al. | INDIS-2022 | |
| 2022-10-24 | M. S. Goodrich | ACAT-2022 Presentation | PPTX |
| 2023-03-17 | M. S. Goodrich, et al. | ACAT-2022 Proceedings | |
| 2023-05-11 | M. S. Goodrich, et al. | CHEP-2023 Presentation | PPTX |
| 2023-10-12 | D. Howard, et al. | CHEP-2023 Conference Publication | |
| 2024-03-11 | M. S. Goodrich, et al. | ACAT-2024 Presentation | PPTX |
| 2024-04-10 | M. S. Goodrich, et al. | RT-2024 Presentation | PPTX |
| 2024-07-31 | M. S. Goodrich, et al. | ACAT-2024 Proceedings | |
| 2024-10-02 | S. Veseli, APS/SDM | APS/ALS - EJFAT | PPTX |
EJFAT Weekly EPSCI Meetings
EJFAT Weekly Collaboration Meetings
Technical Design Overview
EJFAT Technical Design Overview
UDP Transmission Performance
EJFAT UDP General Performance Considerations
EJFAT UDP Packet Receiving and Core Switching
EJFAT UDP Packet Sending and NUMA Nodes
EJFAT UDP Single Thread Packet Sending and Receiving
Testing Load Balancer Bandwidth
HOW-TOs
How to install, build and use gRPC
How to install, build and use XDP related packages
How to Compute Schedule Density from PID Signals
Network Path MTU Discovery support in the Linux Kernel:
file: /proc/sys/net/ipv4/tcp_mtu_probing variable: net.ipv4.tcp_mtu_probing (integer; default: 0; since Linux 2.6.17): tcp_mtu_probing - INTEGER Controls TCP Packetization-Layer Path MTU Discovery. Takes three values: 0 - Disabled 1 - Disabled by default, enabled when an ICMP black hole detected 2 - Always enabled, use initial MSS of tcp_base_mss.
REFERENCEs
A horizontally scalable online processing system for trigger-less data acquisition
The-triggerless-data-acquisition-system-of-the-XENONnT-experiment
Real-time data analysis in particle physics
Edge to Core Test Equipment:
- Price Estimate Spreadsheet
- Networking Diagram, Updated (PDF) (from Brent 2024-02-09)
- PR408549 : Requisition 1 of 2 :
- Statement of Work for Servers
- 1/13/2022: EJFAT team decided to solicit two bid responses, one with MLX NIC and one without. Response from Procurement is "I can ask for the two separate quotes. If you are going to purchase both option (with & without add-in cards), once I receive the quotes back, you will have submit a new PR to cover the option (without add-in cards)."
- 1/18/2022: Question from KOI Computers: "please clarify what the part number for the NVIDIA Dual Port ConnectX-6". Replied with part # MCX623106AN-CDAT.
- 1/24/2022: Requisition currently open for bid responses from vendors. Due date is COB 1/24/2022.
- 1/27/2022: PO awarded to Atipa for 6 servers and 1 file-server with FPGA and MLX SmartNIC. Expected delivery date from vendor is 5/31/2022.
- PR408870 PR408938 Requisition 2 of 2: Statement of Work for Switches & Cables
- 1/14/2022: PRs for the switches, transceivers and fiber have been submitted. I added (4) 2km 100G transceivers to support dual 100G connections between the switches. We can always upgrade to 400G in the future, if needed.
- PR409850 NVIDIA ARM HPC Developer Kit
- Hardware Specifications for dev kit
- Model GIGABYTE G242-P32, 2U server
- CPU 1x Ampere Altra Q80-30 (Arm processor)
- Memory 512G DDR4 memory
- Storage 6TB SAS/ SATA 3.5″
- GPU 2x NVIDIA A100 GPU
- Network 2x NVIDIA® BlueField®-2 E-Series DPU, 200GbE/HDR single-port QSFP56, PCIe Gen4 x16, secure boot enabled, crypto disabled, 16GB on-board DDR, 1GbE OOB management
- Hardware Specifications for dev kit