EJFAT

Welcome to the EJFAT Wiki

(ESnet / JLaB FPGA Accelerated Transport)

---

System Overview:

EJFAT is a collaboration between Energy Sciences Network (ESnet) and Thomas Jefferson National Laboratory (JLab) for proof of concept engineering to program a Field Programmable Gate Array (FPGA) for network data routing of commonly tagged UDP packets from any data source to individual and configurable destination endpoints in an end-point compute work load balanced manner, including some additional tagging for stream reassembly at the endpoint. The primary purpose of this FPGA based acceleration is to load balance work to destination compute farm endpoints with low latency and full line rate bandwidth of 100 Gbs with feedback from the destination compute farm.

EJFAT System Status

ejfat-1

1. 200Gbps NIC: ejfat-1-daq 129.57.177.8
2. 10Gbps NIC: ejfat-1 129.57.177.131
3. U280 FPGA: ejfat-1-dp 129.57.177.11
4. LB CP: ejfat-5 129.57.177.131
5. LB: DP latest stable FW (44059/9d16d8e), CP latest Development branch (7ebf1ec7)
6. IT Account Setup Sandbox

ejfat-2

1. 200Gbps NIC: ejfat-2-daq 129.57.177.2
2. 10Gbps NIC: ejfat-2 129.57.177.132
3. U280 FPGA: ejfat-2-dp 129.57.177.12
4. LB CP: ejfat-2 129.57.177.132
5. LB: DP latest stable FW (44059/96cc489), CP latest stable branch (b7952f4)

ejfat-3

1. 200Gbps NIC: ejfat-3-daq 129.57.177.3
2. 10Gbps NIC: ejfat-3 129.57.177.133
3. Two U280s installed

ejfat-4

1. 200Gbps NIC: ejfat-4-daq 129.57.177.4
2. 10Gbps NIC: ejfat-4 129.57.177.134
3. XDP experiments
4. U280 FPGA: ejfat-4-dp 129.57.177.14
5. DNA 0x40020000013ae32304208185
6. LB CP: ejfat-4 129.57.177.134
7. LB: DP latest stable FW (44059/e3b5bc91), CP latest stable branch (b7952f4)

ejfat-5

1. 200Gbps NIC: ejfat-5-daq 129.57.177.5
2. 10Gbps NIC: ejfat-5 129.57.177.135
3. U280 FPGA: ejfat-5-dp 129.57.177.15
4. DNA 0x40020000013adea04c9044c5
5. LB CP: ejfat-5 129.57.177.135
6. LB: DP latest stable FW (44059/e3b5bc91), CP latest stable branch (b7952f4)

ejfat-6

1. 200Gbps NIC: ejfat-6-daq 129.57.177.6
2. 10Gbps NIC: ejfat-6 129.57.177.136
3. U280 FPGA: ejfat-6-dp 129.57.177.16
4. DNA 0x40020000013b83c12c108485
5. LB CP: ejfat-6 129.57.177.136
6. LB: DP latest stable FW (44059/e3b5bc91), CP latest stable branch (b7952f4)
7. DAOS experiments

ejfat-fs

1. 200Gbps NIC: ejfat-fs-daq 129.57.177.7
2. 10Gbps NIC: ejfat-fs 129.57.177.130
3. Hosts NVME memory/disk
4. U280 FPGA: ejfat-fs-dp 129.57.177.10
5. DNA
6. LB CP: ejfat-6 129.57.177.130
7. LB: DP latest stable FW (44059/e3b5bc91), CP latest stable branch (b7952f4)

Presentations/Papers

date	presenter	Event	links
2021-03-01	G. Heyes	EJFAT Proposal	Word
2021-10-21	M. S. Goodrich	Div Brief	PDF
2021-11-05	M. S. Goodrich	Canisius College	PDF
2021-12-03	S. Sheldon	ESnet LB Tutorial	MP4
2021-12-10	Y. Kumar	SRO iX Presentation	PPTX
2022-08-05	M. S. Goodrich	RT-2022 Presentation	PPTX
2022-08-05	M. S. Goodrich, et al.	RT-2022 Proceedings	PDF
2022-10-20	S. Sheldon, et al.	INDIS-2022	PDF
2022-10-24	M. S. Goodrich	ACAT-2022 Presentation	PPTX
2023-03-17	M. S. Goodrich, et al.	ACAT-2022 Proceedings	PDF
2023-05-11	M. S. Goodrich, et al.	CHEP-2023 Presentation	PPTX
2023-10-12	D. Howard, et al.	CHEP-2023 Conference Publication	PDF
2024-03-11	M. S. Goodrich, et al.	ACAT-2024 Presentation	PPTX
2024-04-10	M. S. Goodrich, et al.	RT-2024 Presentation	PPTX

EJFAT Weekly EPSCI Meetings

EJFAT Weekly EPSCI Meetings

EJFAT Weekly Collaboration Meetings

EJFAT Weekly Meetings

Technical Design Overview

EJFAT Technical Design Overview

UDP Packet Header Formats

IRIAD/EJFAT Testbed

UDP Transmission Performance

EJFAT UDP General Information

EJFAT UDP General Performance Considerations

EJFAT UDP Packet Receiving and Core Switching

EJFAT UDP Packet Sending and NUMA Nodes

EJFAT UDP Single Thread Packet Sending and Receiving

Testing Load Balancer Bandwidth

HOW-TOs

How to setup ejfat nodes

How to install, build and use gRPC

How to install, build and use XDP related packages

How to Compute Schedule Density from PID Signals

EJFAT API

Edge to Core Test Equipment:

1. Price Estimate Spreadsheet
2. Networking Diagram, Updated (PDF) (from Brent 2024-02-09)
3. PR408549 : Requisition 1 of 2 :
   1. Statement of Work for Servers
   2. 1/13/2022: EJFAT team decided to solicit two bid responses, one with MLX NIC and one without. Response from Procurement is "I can ask for the two separate quotes. If you are going to purchase both option (with & without add-in cards), once I receive the quotes back, you will have submit a new PR to cover the option (without add-in cards)."
   3. 1/18/2022: Question from KOI Computers: "please clarify what the part number for the NVIDIA Dual Port ConnectX-6". Replied with part # MCX623106AN-CDAT.
   4. 1/24/2022: Requisition currently open for bid responses from vendors. Due date is COB 1/24/2022.
   5. 1/27/2022: PO awarded to Atipa for 6 servers and 1 file-server with FPGA and MLX SmartNIC. Expected delivery date from vendor is 5/31/2022.
4. PR408870 PR408938 Requisition 2 of 2: Statement of Work for Switches & Cables
   1. 1/14/2022: PRs for the switches, transceivers and fiber have been submitted. I added (4) 2km 100G transceivers to support dual 100G connections between the switches. We can always upgrade to 400G in the future, if needed.
5. PR409850 NVIDIA ARM HPC Developer Kit
   1. Hardware Specifications for dev kit

      Model GIGABYTE G242-P32, 2U server

      CPU 1x Ampere Altra Q80-30 (Arm processor)

      Memory 512G DDR4 memory

      Storage 6TB SAS/ SATA 3.5″

      GPU 2x NVIDIA A100 GPU

      Network 2x NVIDIA® BlueField®-2 E-Series DPU, 200GbE/HDR single-port QSFP56, PCIe Gen4 x16, secure boot enabled, crypto disabled, 16GB on-board DDR, 1GbE OOB management

Resources

• First FPGA PO
• TBD