Difference between revisions of "EJFAT"

From epsciwiki
Jump to navigation Jump to search
 
(210 intermediate revisions by 4 users not shown)
Line 1: Line 1:
−
= Abstract =
+
<div class="orbitron"><font size="+3">Welcome to the EJFAT Wiki</font><br></div>('''E'''Snet / '''J'''LaB '''F'''PGA '''A'''ccelerated '''T'''ransport)
  
−
We describe a collaboration between Energy Sciences Network (ESnet) and Thomas Jefferson National Laboratory (JLab) for proof of concept engineering to program a Field Programmable Gate Array (FPGA) for network data routing of commonly tagged UDP packets to individual and configurable destination endpoints in a follow-on compute work load balanced manner, including some additional tagging for stream reassembly at the endpoint. The primary purpose of this FPGA based acceleration is to load balance work to destination compute farm endpoints with low latency and full line rate bandwidth of 100 Gbs with feedback (back-pressure) from the destination compute farm.  ESnet used P4 programming on the FPGA to use meta-data in the UDP packet stream to route packets with a common tag to dynamically configurable endpoints controlled by the endpoint farm. Control plane programming tasks included back-pressure notifications from destination endpoints and notification processing by the FPGA host CPU to dynamically re-configure routing tables for the FPGA P4 code.
+
<br><hr><br>
 +
<div class="orbitron"><font size="+1">System Overview:</font></div>''EJFAT is a collaboration between Energy Sciences Network (ESnet) and Thomas Jefferson National Laboratory (JLab) for proof of concept engineering for accelerated load balancer (LB) using dynamic IP4/6 address forwarding. Dynamic because the forwarding address is chosen dynamically from a collection of destination endpoints based on near real-time destination workload conditions, and accelerated because the forwarding is accomplished with low fixed latency at line rates of up to 200Gbps per FPGA, where in general a functioning LB may consist of up to four FPGAs acting as one logical DP for a total bandwidth capacity of over 1 Tbps. The low, fixed latency is achieved by utilization of an appropriately programmed Field Programmable Gate Array (FPGA) to effect the Data Plane (DP) functions of the LB.  
  
−
= EJFAT Overview =
+
== EJFAT System Status ==
 +
=== ejfat-1 ===
 +
# 100Gbps NIC: ejfat-1-daq  129.57.177.8
 +
# 10Gbps NIC:  ejfat-1      129.57.177.131
 +
# U280 FPGA:  ejfat-1-dp    129.57.177.{9-16} - '''LAG'd for 200Gbps'''
 +
# LB CP: ejfat-1 129.57.177.131,  latest Stable branch
 +
# LB: DP latest Stable FW
 +
# CP Web UI port 8081
  
−
This collaboration between ESnet and JLab for FPGA Accelerated Transport (EJFAT) seeks a network data transport capability to aggregate and dynamically route selected UDP traffic with endpoint feedback.
+
=== ejfat-2 ===
−
EJFAT will add meta-data to UDP data streams to be used both by the intervening FPGA, acting as a work Load Balancer (LB), to aggregate data packets from multiple logical input streams and dynamically route to endpoints and for an endpoint Reassembly Engine (RE) to perform custom reassembly resulting from network equipment fragmentation.
+
# 100Gbps NIC: ejfat-2-daq  129.57.177.2
−
While the aggregation and routing meta-data included as the header in the payload is generic in design, it is being first utilized for streamed (non-triggered) data from the JLab DAQ to the back-end compute farm.
+
# 10Gbps NIC:  ejfat-2      129.57.177.132
−
In the initial JLab deployment context, the FPGA will time-stamp aggregate across detector Data Acquisition System (DAQ) channels for the purpose of load balancing work to individual compute farm destinations in a farm status aware manner (see Figure X in Appendix X), where '''work''' here is concerned with using data from an individual time-stamp across all DAQ channels to identify or reconstruct detector '''events'''.
+
# 100Gbps U280 FPGA: ejfat-2-dp 129.57.177.{17-24}
−
This load balancing of computational work is under direct control of the compute farm via dynamic management of routing information communicated to the FPGA host CPU which is passed on to the FPGA.
+
# LB CP: ejfat-2 129.57.177.132,  latest Stable branch
−
As the aggregated/routed data is opaque to this design, it should be reusable for other data streams with aggregation/routing needs.
+
# LB: DP latest Stable FW
 +
# CP Web UI port 8082
  
−
= Read Out Controller Processing =
+
=== ejfat-3 ===
−
The Read Out Controllers (ROC) of the JLab DAQ system will be enhanced to stream data via UDP and include new meta-data prepended to the original payload that serves the needs of the compute destination LB and the destination fragmentation RE. Figure X is a diagram of the new data stream processing requirements for the JLaB DAQ system.
+
# 200Gbps NIC: ejfat-3-daq  129.57.177.3
 +
# 10Gbps NIC:  ejfat-3      129.57.177.133
 +
# '''Two U280s installed - LAG'd for 400Gbps'''
 +
# FW Containers built by Stacey
  
−
This new meta-data, populated by the JLab DAQ system, consists of two parts, the first for the LB and the second for the RE.
+
=== ejfat-4 ===
 +
# 100Gbps NIC: ejfat-4-daq  129.57.177.4
 +
# 10Gbps NIC:  ejfat-4      129.57.177.134
 +
# '''XDP experiments'''
 +
# 100Gbps U280 FPGA: ejfat-4-dp 129.57.177.{41-48}
 +
# LB CP: ejfat-4 129.57.177.134, <s>latest Stable branch</s>
 +
# LB: DP <s>latest Stable FW</s>
  
−
== Load Balancer Meta-Data ==
+
=== ejfat-5 ===
−
The LB meta-data (Figure X, cyan section) is 96 bits that in order consists of
+
# 200Gbps NIC: ejfat-5-daq  129.57.177.5
−
<ul>
+
# 10Gbps NIC:  ejfat-5      129.57.177.135
−
<li>'''LB Control Word''' is 32 bits (bits 0-31) such that
+
# LB CP: ejfat-5 129.57.177.135, <s>latest Stable branch</s>
−
<ul>
+
# 100Gbps U280 FPGA: ejfat-5-dp 129.57.177.{49-56}
−
<li>bits 0-7 ASCII character ’L’</li>
+
# LB: DP <s>latest Stable FW</s>
−
<li>bits 8-15 ASCII character ’B’</li>
+
# '''Optical Taps Installed'''
−
<li>bits 16-23 LB '''version number''' starting at 1 (constant for run duration)</li>
 
−
<li>bits 24-31 '''Protocol Number''' (very useful for protocol decoders e.g., wireshark/tshark )</li>
 
−
</ul></li>
 
−
<li>'''Tick''' (Time stamp or proxy) is a 64 bit quantity (bits 32-95) that for a DAQ run duration
 
−
<ul>
 
−
<li>Monotonically increases</li>
 
−
<li>Unique</li>
 
−
<li>Never rolls over</li>
 
−
<li>Never resets</li>
 
−
<li>Serves as a common tag across multiple DAQ ROC channels/packets related to the same time-stamped data transfer.</li>
 
−
</ul></li>
 
−
</ul>
 
  
−
In standard IETF RFC format:
+
=== ejfat-6 ===
−
<pre>protocol &#39;L:8,B:8,Version:8,Protocol:8,Tick:64&#39;
+
# 200Gbps NIC: ejfat-6-daq  129.57.177.6
−
0                  1                  2                  3 
+
# 10Gbps NIC:  ejfat-6       129.57.177.136
−
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+
# DAOS experiments
−
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+
# '''Using Ubuntu 24.04 LTS'''
−
|      L      |       B      |    Version    |    Protocol  |
+
# FW containers built
−
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+
# Waiting for podman compose installation
−
|                                                              |
 
−
+                              Tick                            +
 
−
|                                                              |
 
−
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+</pre>
 
  
−
== Reassembly Engine Meta-Data ==
+
=== ejfat-fs ===
−
The '''RE meta-data''' (Figure X, yellow section) is 64 bits and consists of
+
# 100Gbps NIC: ejfat-fs-daq  129.57.177.7
−
<ul>
+
# 10Gbps NIC:  ejfat-fs      129.57.177.130
−
<li>bits 0-3 Version number</li>
+
# Hosts NVME memory/disk
−
<li>bits 4-13 Reserved</li>
+
# 100Gbps U280 FPGA: ejfat-fs-dp 129.57.177.{65-72}
−
<li>bit 14 indicates first packet</li>
+
# LB CP: ejfat-fs 129.57.177.130, latest Stable branch
−
<li>bit 15 indicates last packet</li>
+
# LB: DP latest Stable FW
−
<li>bits 16-31 ROC Id</li>
+
# CP Web UI port 8080
−
<li>bits 32-63 packet sequence number or optionally data offset byte number for reassembly</li>
 
−
</ul>
 
−
In standard IETF RFC format:
 
−
<pre>protocol &#39;Version:4,Rsvd:10,First:1,Last:1,ROC-ID:16,Offset:32&#39;
 
−
0                  1                  2                  3 
 
−
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
 
−
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 
−
|Version|        Rsvd      |F|L|            ROC-ID            |
 
−
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 
−
|                            Offset                            |
 
−
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+</pre>
 
  
−
The resultant DAQ data stream is shown just below the block diagram in Figure X and depicts the stream UDP packet structure from the DAQ system to the LB. Individual packets are meta-data tagged both for the LB, to route based on '''tick''' to the proper compute node, and for the RE with packet '''offset''' spanning the collection of packets for a single '''tick''' for eventual destination reassembly.
+
== Presentations/Papers ==
−
The depicted sequence is only illustrative, and no assumption about the order of packets with respect to either '''tick''' or '''offset''' should be made by the LB or the RE.
+
{| class="wikitable"
−
[[File:Esnet-JLab-network-diagram-v002d-roc.png|border|1600px|link=|"ROC Data Stream Processing"]]
+
|-
 +
!date
 +
!presenter
 +
!Event
 +
!links
 +
|-
 +
|2021-03-01
 +
|G. Heyes
 +
|EJFAT Proposal
 +
|[https://jeffersonlab.sharepoint.com/:w:/r/sites/SciComp/_layouts/15/Doc.aspx?sourcedoc=%7B65DA331C-40E4-4761-B643-251BFA309C45%7D&file=20210525%20ASCR%20BRN%20Solicitation%20v4.docx&action=default&mobileredirect=true Word]
 +
|-
 +
|2021-10-21
 +
|M. S. Goodrich
 +
|Div Brief
 +
|[https://jeffersonlab.sharepoint.com/sites/SciComp/Shared%20Documents/EPSCI/EJFAT/EJFAT_for_Div.pdf?CT=1638970015731&OR=ItemsView PDF]
 +
|-
 +
|2021-11-05
 +
|M. S. Goodrich
 +
|Canisius College
 +
|[https://jeffersonlab.sharepoint.com/sites/SciComp/Shared%20Documents/EPSCI/EJFAT/canisius.pdf?CT=1638970328329&OR=ItemsView PDF]
 +
|-
 +
|2021-12-03
 +
|S. Sheldon
 +
|ESnet LB Tutorial
 +
|[https://jeffersonlab.sharepoint.com/:v:/r/sites/SciComp/Shared%20Documents/EPSCI/EJFAT/ESnet_EJFAT_Tut.mp4?csf=1&web=1&e=4nDeZ2 MP4]
 +
|-
 +
|2021-12-10
 +
|Y. Kumar
 +
|SRO iX Presentation
 +
|[https://jeffersonlab.sharepoint.com/:p:/r/sites/SciComp/Shared%20Documents/EPSCI/EJFAT/EJFAT_SRO_iX.pptx?d=w78e41e5ddab04d21a4c26f93ac84b7d6&csf=1&web=1&e=gkaCDS PPTX]
 +
|-
 +
|2022-08-05
 +
|M. S. Goodrich
 +
|RT-2022 Presentation
 +
|[https://jeffersonlab.sharepoint.com/:p:/r/sites/SciComp/Shared%20Documents/EPSCI/EJFAT/JLab%20EJFAT-msg.pptx?d=w7a8e53d19a584fefb1405fa8ff190b1e&csf=1&web=1&e=50bX4g PPTX]
 +
|-
 +
|2022-08-05
 +
|M. S. Goodrich, et al.
 +
|RT-2022 Proceedings
 +
|[https://jeffersonlab.sharepoint.com/:b:/r/sites/SciComp/Shared%20Documents/EPSCI/EJFAT/EJFAT_rt2022.pdf?csf=1&web=1&e=NFHXHM PDF]
 +
|-
 +
|2022-10-20
 +
|S. Sheldon, et al.
 +
|INDIS-2022
 +
|[https://jeffersonlab.sharepoint.com/:b:/r/sites/SciComp/Shared%20Documents/EPSCI/EJFAT/Indis_Paper_2022-3.pdf?csf=1&web=1&e=tmhpfA PDF]
 +
|-
 +
|2022-10-24
 +
|M. S. Goodrich
 +
|ACAT-2022 Presentation
 +
|[https://jeffersonlab.sharepoint.com/:p:/r/sites/SciComp/Shared%20Documents/EPSCI/EJFAT/EJFAT-acat2022.pptx?d=wc024332f3cf7440eae15e4f6f3646897&csf=1&web=1&e=QEwIcx PPTX]
 +
|-
 +
|2023-03-17
 +
|M. S. Goodrich, et al.
 +
|ACAT-2022 Proceedings
 +
|[https://jeffersonlab.sharepoint.com/:b:/r/sites/SciComp/Shared%20Documents/EPSCI/EJFAT/EJFAT_ACAT_2022_QL_sub.pdf?csf=1&web=1&e=dR566P PDF]
 +
|-
 +
|2023-05-11
 +
|M. S. Goodrich, et al.
 +
|CHEP-2023 Presentation
 +
|[https://jeffersonlab.sharepoint.com/:p:/r/sites/SciComp/Shared%20Documents/EPSCI/EJFAT/EJFAT-chep2023.pptx?d=w605623a55051446e9d2bcca80f64eda6&csf=1&web=1&e=NHSloC PPTX]
 +
|-
 +
|2023-10-12
 +
|D. Howard, et al.
 +
|CHEP-2023 Conference Publication
 +
|[https://jeffersonlab.sharepoint.com/:b:/r/sites/SciComp/Shared%20Documents/EPSCI/EJFAT/chep2023_proceedings.pdf?csf=1&web=1&e=FO7f8j PDF]
 +
|-
 +
|2024-03-11
 +
|M. S. Goodrich, et al.
 +
|ACAT-2024 Presentation
 +
|[https://jeffersonlab.sharepoint.com/:p:/r/sites/SciComp/Shared%20Documents/EPSCI/EJFAT/Acat2024.pptx?d=wb4c9cc47a8eb4b299c3dab1aaa379a36&csf=1&web=1&e=Kct82Y} PPTX]
 +
|-
 +
|2024-04-10
 +
|M. S. Goodrich, et al.
 +
|RT-2024 Presentation
 +
|[https://jeffersonlab.sharepoint.com/:p:/r/sites/SciComp/Shared%20Documents/EPSCI/EJFAT/rt2024.pptx?d=w0dba99dbb67f481f9a39907dbec384b8&csf=1&web=1&e=1XISCm} PPTX]
 +
|-
 +
|2024-07-31
 +
|M. S. Goodrich, et al.
 +
|ACAT-2024 Proceedings
 +
|[https://jeffersonlab.sharepoint.com/:b:/r/sites/SciComp/Shared%20Documents/EPSCI/EJFAT/ACAT_2024.pdf?csf=1&web=1&e=HkQedP PDF]
 +
|-
 +
|2024-10-02
 +
|S. Veseli​, APS/SDM
 +
|APS/ALS - EJFAT
 +
|[https://jeffersonlab.sharepoint.com/:p:/r/sites/SciComp/Shared%20Documents/EPSCI/EJFAT/AlsEjfatMeeting-20241002.pptx?d=wcaa3a21ffd3a466f979bf3f5fbaab457&csf=1&web=1&e=BSOlI7 PPTX]
 +
|}
  
−
= UDP Header =
+
== EJFAT Weekly EPSCI Meetings ==
−
The UDP Header will be populated as follows:
 
−
<ul>
 
−
<li>: '''Source Port''' = lower 16 bits of Load Balancer '''Tick''' (for LAG switch entropy)</li>
 
−
<li>: '''Destination Port''' = Value that indicates LB should perform load balancing (else packet is discarded) = 'LB' = 0x4c42</li>
 
−
</ul>
 
−
The resultant DAQ data stream is shown below the block diagram and depicts the stream UDP packet structure. Individual packets
 
  
−
= ROC Aggregation Switch =
+
[[EJFAT Weekly EPSCI Meetings]]
−
Individual ROC channels will be aggregated for maximum throughput by a switch using Link Aggregation Protocol (LAG) or similar where the network traffic downstream of the switch will be addressed to the LB FPGA (see (Figure X, Appendix X).
 
  
−
If the LAG configured switch proves to be incapable of meeting line rate throughput (100Gbs), then an additional FPGA(s) can be engineered to subsume this function as depicted in Figure X.
+
== EJFAT Weekly Collaboration Meetings ==
  
−
[[File:Esnet-JLab-network-diagram-v002a-roc-1.png|border|"ROC Channel Load Balancing"]]
+
[[EJFAT Weekly Meetings]]
  
−
= LB Processing =
+
== Technical Design Overview ==
−
The FPGA resident LB aggregates data across all DAQ channels for a single discrete '''tick''' and routes this aggregated data to individual end compute nodes in cooperation with the FPGA host chassis CPU using algorithms designed for the host CPU and feedback received from the end compute node farm, maintaining complete opacity of the UDP payload to the LB (except for the LB meta-data).
 
  
−
== Data Plane (FPGA) Processing ==
+
[[EJFAT Technical Design Overview]]
−
[[File:LB-data-pln.png|border|"Data Plane Flow Chart"]]
 
−
<ul>
 
−
<li>Packet Parsing Stage
 
−
<ul>
 
−
<li>Headers defined in the previous stage will be parsed and made available for the remaining stages.</li>
 
−
<li>The Event Payload MUST NOT be parsed by the load balancer.</li>
 
−
</ul></li>
 
−
<li>Input Packet Filter Stage
 
−
<ul>
 
−
<li>Implemented as a P4 table with the following properties</li>
 
−
<li>Max Entries: 32</li>
 
−
<li>Key:
 
−
<ul>
 
−
<li>(Exact Match) EtherType</li>
 
−
<li>(Exact Match) (96b 0 <span class="math"> ∥ </span> IPv4 Dst) OR IPv6 Dst</li>
 
−
<li>(Binary Match) UDP Dst Port</li>
 
−
</ul></li>
 
−
<li>Value: None</li>
 
−
<li>A miss in this table MUST result in the packet being discarded since it is not intended for the load balancer.</li>
 
−
<li>The P4 code in this stage must also check both the Magic and Version fields in the LB Header. A mismatch from the expected values MUST result in the packet being discarded.</li>
 
−
</ul></li>
 
−
<li>Calendar Epoch Assignment Stage
 
−
<ul>
 
−
<li>Implemented as a P4 table with the following properties</li>
 
−
<li>Max Entries: 128</li>
 
−
<li>Key: (Ternary Match) 64b LB Event Number (Timestamp)</li>
 
−
<li>Value: 32b Calendar Epoch</li>
 
−
</ul></li>
 
−
<li>Load Balance Calendar to Member Map Stage
 
−
<ul>
 
−
<li>Implemented as a P4 table with the following properties</li>
 
−
<li>Max Entries: 2048</li>
 
−
<li>Key:
 
−
<ul>
 
−
<li>(Exact Match) 32b Calendar Epoch</li>
 
−
<li>(Exact Match) 9b Calendar Slot (ie. LB Event Number &amp; 0x1FF)</li>
 
−
</ul></li>
 
−
<li>Value: 16b LB Member ID</li>
 
−
</ul></li>
 
−
<li>Load Balance Member Info Lookup Stage
 
−
<ul>
 
−
<li>Implemented as a P4 table with the following properties</li>
 
−
<li>Max Entries: 1K</li>
 
−
<li>Key:
 
−
<ul>
 
−
<li>(Exact Match) 16b EtherType (IPv4 or IPv6)</li>
 
−
<li>(Exact Match) 16b LB Member ID</li>
 
−
</ul></li>
 
−
<li>Value:
 
−
<ul>
 
−
<li>8b Action ID
 
−
<ul>
 
−
<li>1 = IPv4 Rewrite</li>
 
−
<li>2 = IPv6 Rewrite</li>
 
−
</ul></li>
 
−
<li>IPv4 Rewrite Action
 
−
<ul>
 
−
<li>48b MAC DA (for next-hop router)</li>
 
−
<li>32b IPv4 Dst</li>
 
−
<li>16b UDP Dst Port</li>
 
−
</ul></li>
 
−
<li>IPv6 Rewrite Action
 
−
<ul>
 
−
<li>48b MAC DA (for next-hop router)</li>
 
−
<li>128b IPv6 Dst</li>
 
−
<li>16b UDP Dst Port</li>
 
−
</ul></li>
 
−
</ul></li>
 
−
<li>'''Note:''' Rewrite Action type must match the input packet’s EtherType. E.g. An Input IPv4 packet cannot use the IPv6 Rewrite Action and vice versa.</li>
 
−
<li>'''Note:''' Before applying the rewrite actions, the original packet’s MAC DA should be copied into the MAC SA. This will ensure that the outgoing packet will be sent from exactly the MAC address that the original packet was destined to. This will help to keep the MAC FDB entries in the adjacent switches from expiring.</li>
 
−
<li>'''Note:''' Packets will be transmitted back out the port they were received on.</li>
 
−
</ul></li>
 
−
</ul>
 
  
−
The resultant LB data stream is shown just above the block diagram in Figure X and depicts the stream UDP packet structure from the LB to the RE concerning an arbitrary '''single''' destination compute node. Individual packets here are still meta-data tagged both for the LB and RE. <strong>The RE for a target compute node will see a collection of packets that share a common '''tick'''</strong>.
+
[[UDP Packet Header Formats]]
  
−
The depicted sequence is only illustrative, and no assumption about the order of packets with respect to the '''offset''' should be made by the RE.
+
[https://jeffersonlab.sharepoint.com/:p:/r/sites/HPDF/_layouts/15/Doc.aspx?sourcedoc=%7BEABA533A-E516-4C57-BE85-BBF594F5E918%7D&file=Jan%2010%20HPDF%20Conceptual%20Machine%20Design%20Concept.pptx&action=edit&mobileredirect=true IRIAD/EJFAT Testbed]
  
−
== LB Data Plane P4 Code ==
+
== UDP Transmission Performance ==
−
<pre>
 
−
1    // -*- Mode: c -*-
 
−
2    #include <core.p4>
 
−
3    #include <xsa.p4>
 
−
4
 
−
5    struct intrinsic_metadata_t {
 
−
6    bit<64> ingress_global_timestamp;
 
−
7    bit<64> egress_global_timestamp;
 
−
8    bit<16> mcast_grp;
 
−
9    bit<16> egress_rid;
 
−
10    }
 
−
11
 
−
12    header ethernet_t {
 
−
13    bit<48> dstAddr;
 
−
14    bit<48> srcAddr;
 
−
15    bit<16> etherType;
 
−
16    }
 
−
17
 
−
header ipv6_t {
 
−
bit<4>  version;
 
−
bit<8>  trafficClass;
 
−
bit<20>  flowLabel;
 
−
bit<16>  payloadLen;
 
−
bit<8>  nextHdr;
 
−
bit<8>  hopLimit;
 
−
bit<128> srcAddr;
 
−
bit<128> dstAddr;
 
−
}
 
  
−
header ipv4_t {
+
[[EJFAT UDP General Information]]
−
bit<4>  version;
 
−
bit<4>  ihl;
 
−
bit<8>  diffserv;
 
−
bit<16> totalLen;
 
−
bit<16> identification;
 
−
bit<3>  flags;
 
−
bit<13> fragOffset;
 
−
bit<8>  ttl;
 
−
bit<8>  protocol;
 
−
bit<16> hdrChecksum;
 
−
bit<32> srcAddr;
 
−
bit<32> dstAddr;
 
−
}
 
  
−
header ipv4_opt_t {
+
[[EJFAT UDP General Performance Considerations]]
−
varbit<320> options; // IPv4 options - length = (ipv4.hdr_len - 5) * 32
 
−
}
 
  
−
header udp_t {
+
[[EJFAT UDP Packet Receiving and Core Switching]]
−
bit<16> srcPort;
 
−
bit<16> dstPort;
 
−
bit<16> totalLen;
 
−
bit<16> checksum;
 
−
}
 
  
−
header udplb_t {
+
[[EJFAT UDP Packet Sending and NUMA Nodes]]
−
bit<16> magic; /* "LB" */
 
−
bit<8>  version; /* version 0 */
 
−
bit<8>  proto;
 
−
bit<64> tick;
 
−
}
 
−
#define SIZEOF_UDPLB_HDR 12
 
  
−
struct short_metadata {
+
[[EJFAT UDP Single Thread Packet Sending and Receiving]]
−
bit<64> ingress_global_timestamp;
 
−
bit<9>  egress_spec;
 
−
bit<1>  processed;
 
−
bit<16> packet_length;
 
−
}
 
−
 
 
−
struct headers {
 
−
ethernet_t ethernet;
 
−
ipv4_t ipv4;
 
−
ipv4_opt_t ipv4_opt;
 
−
ipv6_t ipv6;
 
−
udp_t udp;
 
−
udplb_t udplb;
 
−
}
 
  
−
// User-defined errors
+
[[Testing Load Balancer Bandwidth]]
−
error {
 
−
InvalidIPpacket,
 
−
InvalidUDPLBmagic,
 
−
InvalidUDPLBversion
 
−
}
 
  
−
parser ParserImpl(packet_in packet, out headers hdr, inout short_metadata short_meta, inout standard_metadata_t smeta) {
+
== HOW-TOs ==
−
state start {
 
−
transition parse_ethernet;
 
−
}
 
  
−
state parse_ethernet {
+
[[How to use Control Plane Web UI]]
−
packet.extract(hdr.ethernet);
 
−
transition select(hdr.ethernet.etherType) {
 
−
16w0x0800: parse_ipv4;
 
−
16w0x86dd: parse_ipv6;
 
−
default: accept;
 
−
}
 
−
}
 
−
state parse_ipv4 {
 
−
packet.extract(hdr.ipv4);
 
−
verify(hdr.ipv4.version == 4 && hdr.ipv4.ihl >= 5, error.InvalidIPpacket);
 
−
packet.extract(hdr.ipv4_opt, (((bit<32>)hdr.ipv4.ihl - 5) * 32));
 
−
transition select(hdr.ipv4.protocol) {
 
−
8w17: parse_udp;
 
−
default: accept;
 
−
}
 
−
}
 
−
state parse_ipv6 {
 
−
packet.extract(hdr.ipv6);
 
−
verify(hdr.ipv6.version == 6, error.InvalidIPpacket);
 
−
transition select(hdr.ipv6.nextHdr) {
 
−
8w17: parse_udp;
 
−
default: accept;
 
−
}
 
−
}
 
−
state parse_udp {
 
−
packet.extract(hdr.udp);
 
−
transition select(hdr.udp.dstPort) {
 
−
16w0x4c42: parse_udplb;
 
−
default: accept;
 
−
}
 
−
}
 
  
−
state parse_udplb {
+
[[How to Monitor Prometheus]]
−
packet.extract(hdr.udplb);
 
−
verify(hdr.udplb.magic == 0x4c42, error.InvalidUDPLBmagic);
 
−
verify(hdr.udplb.version == 1, error.InvalidUDPLBversion);
 
−
transition accept;
 
−
}
 
−
}
 
  
−
control MatchActionImpl(inout headers hdr, inout short_metadata short_meta, inout standard_metadata_t smeta) {
+
[https://wiki.jlab.org/epsciwiki/index.php/Install_an_EJFAT_Load_Balancer Install a Load Balancer]
  
−
//
+
[https://jeffersonlab.sharepoint.com/:t:/r/sites/SciComp/Shared%20Documents/EPSCI/EJFAT/lbtest.txt?csf=1&web=1&e=PNz0DM Test a Load Balancer]
−
// DstFilter - a gate
 
−
//
 
  
−
bit<128> meta_ipdst = 128w0;
+
[[How to setup ejfat nodes]]
−
 
−
action drop() {
 
−
smeta.drop = 1;
 
−
}
 
−
 
−
table dst_filter_table {  // layer 2/3 filter
 
−
actions = {
 
−
drop;
 
−
NoAction;
 
−
}
 
−
key = {
 
−
hdr.ethernet.dstAddr : exact; // this a unicast packet for us at layer 2 (MAC) ?
 
−
hdr.ethernet.etherType : exact // ipv4 or ipv6
 
−
meta_ipdst : exact; // normalized 128b IP (not MAC) address from either ipv4/6
 
−
}
 
−
size = 32;
 
−
default_action = drop;
 
−
}
 
  
−
//
+
[[How to install, build and use gRPC]]
−
// EpochAssign
 
−
//
 
  
−
bit<32> meta_epoch = 0;
+
[[How to install, build and use XDP related packages]]
−
 
−
action do_assign_epoch(bit<32> epoch) {
 
−
meta_epoch = epoch;
 
−
}
 
  
−
table epoch_assign_table {
+
[https://jeffersonlab.sharepoint.com/:b:/r/sites/SciComp/Shared%20Documents/EPSCI/EJFAT/CP_PID_Sched.pdf?csf=1&web=1&e=JpffJ4 How to Compute Schedule Density from PID Signals]
−
actions = {
 
−
do_assign_epoch;
 
−
drop;
 
−
}
 
−
key = {
 
−
hdr.udplb.tick : lpm; // lpm is the keystone to routing everywhere
 
−
}
 
−
size = 128;
 
−
default_action = drop;
 
−
}
 
  
−
//
+
[https://linuxconfig.org/how-to-enable-jumbo-frames-in-linux Enable Jumbo Frames]
−
// LoadBalanceCalendar
 
−
//
 
  
−
// Use lsbs of tick to select a calendar slot
+
Network Path MTU Discovery support in the Linux Kernel:
−
bit<9> calendar_slot = (bit<9>) hdr.udplb.tick & 0x1FF;  # 9 bits
 
−
bit<16> meta_member_id = 0;
 
  
−
action do_assign_member(bit<16> member_id) {
+
<pre>
−
meta_member_id = member_id;
+
file: /proc/sys/net/ipv4/tcp_mtu_probing
−
}
+
variable: net.ipv4.tcp_mtu_probing (integer; default: 0; since Linux 2.6.17):
  
−
table load_balance_calendar_table {
+
tcp_mtu_probing - INTEGER
−
actions = {
+
Controls TCP Packetization-Layer Path MTU Discovery.  Takes three values:
−
do_assign_member;
+
  0 - Disabled
−
drop;
+
  1 - Disabled by default, enabled when an ICMP black hole detected
−
}
+
  2 - Always enabled, use initial MSS of tcp_base_mss.
−
key = {
+
</pre>
−
meta_epoch : exact;
 
−
calendar_slot : exact;
 
−
}
 
−
size = 2048;
 
−
default_action = drop;
 
−
}
 
−
 
−
//
 
−
// MemberInfoLookup
 
−
//
 
−
 
 
−
// Cumulative checksum delta due to field rewrites
 
−
bit<16> ckd = 0;
 
−
bit<16> new_udp_dst = 0;
 
−
 
 
−
action cksum_sub(inout bit<16> cksum, in bit<16> a) {
 
−
bit<17> sum = (bit<17>) cksum;
 
−
 
 
−
sum = sum + (bit<17>)(a ^ 0xFFFF);
 
−
sum = (sum & 0xFFFF) + (sum >> 16);
 
−
 
 
−
cksum = sum[15:0];
 
−
}
 
−
 
 
−
action cksum_add(inout bit<16> cksum, in bit<16> a) {
 
−
bit<17> sum = (bit<17>) cksum;
 
−
 
 
−
sum = sum + (bit<17>)a;
 
−
sum = (sum & 0xFFFF) + (sum >> 16);
 
−
 
 
−
cksum = sum[15:0];
 
−
}
 
−
 
 
−
action cksum_swap(inout bit<16> cksum, in bit<16> old, in bit<16> new) {
 
−
cksum_sub(cksum, old);
 
−
cksum_add(cksum, new);
 
−
}
 
−
 
 
−
action do_ipv4_member_rewrite(bit<48> mac_dst, bit<32> ip_dst, bit<16> udp_dst) {
 
−
// Calculate IPv4 and UDP pseudo header checksum delta using rfc1624 method
 
−
 
 
−
cksum_swap(ckd, hdr.ipv4.dstAddr[31:16], ip_dst[31:16]);
 
−
cksum_swap(ckd, hdr.ipv4.dstAddr[15:00], ip_dst[15:00]);
 
−
cksum_swap(ckd, hdr.ipv4.totalLen, hdr.ipv4.totalLen - SIZEOF_UDPLB_HDR);
 
−
 
 
−
// Apply the accumulated delta to the IPv4 header checksum
 
−
hdr.ipv4.hdrChecksum = hdr.ipv4.hdrChecksum ^ 0xFFFF;
 
−
cksum_add(hdr.ipv4.hdrChecksum, ckd);
 
−
hdr.ipv4.hdrChecksum = hdr.ipv4.hdrChecksum ^ 0xFFFF;
 
−
 
 
−
hdr.ethernet.dstAddr = mac_dst;
 
−
hdr.ipv4.dstAddr = ip_dst;
 
−
hdr.ipv4.totalLen = hdr.ipv4.totalLen - SIZEOF_UDPLB_HDR;
 
−
 
 
−
new_udp_dst = udp_dst;
 
−
}
 
−
 
 
−
action do_ipv6_member_rewrite(bit<48> mac_dst, bit<128> ip_dst, bit<16> udp_dst) {
 
−
// Calculate UDP pseudo header checksum delta using rfc1624 method
 
−
 
 
−
cksum_swap(ckd, hdr.ipv6.dstAddr[63:48], ip_dst[63:48]);
 
−
cksum_swap(ckd, hdr.ipv6.dstAddr[47:32], ip_dst[47:32]);
 
−
cksum_swap(ckd, hdr.ipv6.dstAddr[31:16], ip_dst[31:16]);
 
−
cksum_swap(ckd, hdr.ipv6.dstAddr[15:00], ip_dst[15:00]);
 
−
 
 
−
cksum_swap(ckd, hdr.ipv6.payloadLen, hdr.ipv6.payloadLen - SIZEOF_UDPLB_HDR);
 
−
 
 
−
hdr.ethernet.dstAddr = mac_dst;
 
−
hdr.ipv6.dstAddr = ip_dst;
 
−
hdr.ipv6.payloadLen = hdr.ipv6.payloadLen - 12;
 
−
new_udp_dst = udp_dst;
 
−
}
 
−
 
 
−
table member_info_lookup_table {
 
−
actions = {
 
−
do_ipv4_member_rewrite;
 
−
do_ipv6_member_rewrite;
 
−
drop;
 
−
}
 
−
key = {
 
−
hdr.ethernet.etherType : exact;
 
−
meta_member_id : exact;
 
−
}
 
−
size = 1024;
 
−
default_action = drop;
 
−
}
 
−
 
 
−
// Entry Point
 
−
apply {
 
−
bool hit;
 
−
 
 
−
//
 
−
// DstFilter
 
−
//
 
−
 
−
// Normalized IP destination address for both ipv4 and ipv6
 
−
if (hdr.ipv4.isValid()) {
 
−
meta_ipdst = (bit<96>) 0 ++ (bit<32>) hdr.ipv4.dstAddr;
 
−
} else if (hdr.ipv6.isValid()) {
 
−
meta_ipdst = hdr.ipv6.dstAddr;
 
−
}
 
−
 
−
// .apply() looks up key and applies action specified by control plane on match
 
−
hit = dst_filter_table.apply().hit;
 
−
if (!hit) {
 
−
return;
 
−
}
 
−
 
 
−
//
 
−
// EpochAssign
 
−
//
 
−
 
 
−
if (!hdr.udplb.isValid()) {
 
−
return;
 
−
}
 
−
 
 
−
hit = epoch_assign_table.apply().hit;
 
−
if (!hit) {
 
−
return;
 
−
}
 
−
 
 
−
//
 
−
// LoadBalanceCalendar
 
−
//
 
  
−
hit = load_balance_calendar_table.apply().hit;
+
== REFERENCEs ==
−
if (!hit) {
 
−
return;
 
−
}
 
  
−
//
+
[https://jeffersonlab.sharepoint.com/:x:/r/sites/DataCenter/_layouts/15/Doc.aspx?sourcedoc=%7B3F832940-1BA2-4183-A00A-5085C5A353D6%7D&file=IRIAD-testbed-Inventory.xlsx&action=default&mobileredirect=true EJFAT Config Planning]
−
// MemberInfoLookup
 
−
//
 
  
−
hit = member_info_lookup_table.apply().hit;
+
[https://www.jlab.org/news/releases/california-streamin-jefferson-lab-esnet-achieve-coast-coast-feed-real-time-physics JLab EJFAT News Release]
−
if (!hit) {
 
−
return;
 
−
}
 
  
−
//
+
[https://jeffersonlab.sharepoint.com/:i:/r/sites/SciComp/Shared%20Documents/EPSCI/EJFAT/JIRIAF%20on%20FABRIC.png?csf=1&web=1&e=TOGEPr EJFAT on FABRIC]
−
// UpdateUDPChecksum
 
−
//
 
  
−
// Calculate UDP pseudo header checksum delta using rfc1624 method
+
[https://jeffersonlab.sharepoint.com/:b:/r/sites/SciComp/Shared%20Documents/EPSCI/EJFAT/E2SAR.drawio.pdf?csf=1&web=1&e=E0Uqlh EJFAT API]
  
−
cksum_swap(ckd, hdr.udp.dstPort, new_udp_dst);
+
[https://docs.google.com/document/d/1ssw8sye7jExtPCJVejloe8hNkyWOcxEQzVmm45xs5-w/edit#heading=h.b8k68ix2wf30 LB Pipeline]
−
cksum_swap(ckd, hdr.udp.totalLen, hdr.udp.totalLen - SIZEOF_UDPLB_HDR);
 
  
−
// Subtract out the bytes of the UDP load-balance header
+
[https://docs.google.com/document/d/1qEo51MZeUPM3-DA2CK6jAccrU0r1QtPfl5i3aPS2SKM/edit?exids=71471482,71471477#heading=h.69350544ggm5 Getting Started with EJFAT]
−
cksum_sub(ckd, hdr.udplb.magic);
 
−
cksum_sub(ckd, hdr.udplb.version ++ hdr.udplb.proto);
 
−
cksum_sub(ckd, hdr.udplb.tick[63:48]);
 
−
cksum_sub(ckd, hdr.udplb.tick[47:32]);
 
−
cksum_sub(ckd, hdr.udplb.tick[31:16]);
 
−
cksum_sub(ckd, hdr.udplb.tick[15:00]);
 
  
−
// Write the updated checksum back into the packet
+
[https://jeffersonlab.sharepoint.com/:w:/r/sites/ITDivision/proposals/_layouts/15/Doc.aspx?sourcedoc=%7B33ffd720-9356-471f-8880-b0c56c5593a5%7D&action=view&wdAccPdf=0&wdparaid=39A41B49 IRIAD Workplan]
−
hdr.udp.checksum = hdr.udp.checksum ^ 0xFFFF;
 
−
cksum_add(hdr.udp.checksum, ckd);
 
−
hdr.udp.checksum = hdr.udp.checksum ^ 0xFFFF;
 
  
−
// Update destination port and fix up length to adapt to dropped udplb header
+
[https://wiki.jlab.org/epsciwiki/index.php/SRO_Grand_Challenge SRO Grand Challenge]
−
hdr.udp.dstPort = new_udp_dst;
 
−
hdr.udp.totalLen = hdr.udp.totalLen - SIZEOF_UDPLB_HDR;
 
−
}
 
−
}
 
  
−
control DeparserImpl(packet_out packet, in headers hdr, inout short_metadata short_meta, inout standard_metadata_t smeta) {
+
[https://my.es.net/?_gl=1*pchcca*_ga*MjAyODE5NDE3OC4xNzEwOTYwMDI4*_ga_9Y9H16804B*MTcxMDk2MDAyOC4xLjAuMTcxMDk2MDAyOC4wLjAuMA..&s=JLAB&st=esnet_site ESnet Logical Map]
−
apply {
 
−
packet.emit(hdr.ethernet);
 
−
packet.emit(hdr.ipv4);
 
−
packet.emit(hdr.ipv4_opt);
 
−
packet.emit(hdr.ipv6);
 
−
packet.emit(hdr.udp);
 
−
}
 
−
}
 
  
−
XilinxPipeline(
+
[http://linux-ip.net/html/tools-ip-neighbor.html IP Neighbor]
−
ParserImpl(),
 
−
MatchActionImpl(),
 
−
DeparserImpl()
 
−
) main;
 
  
−
</pre>
+
[https://robotframework.org/robotframework/latest/RobotFrameworkUserGuide.html Robot Framework]
−
<ul>
 
−
<li>lines 13-14: These are MAC addresses</li>
 
−
<li>line 45: IPV4 options length = (ipv4 <span class="math"> × </span> hdrlen - 5) <span class="math"> × </span> 32</li>
 
−
<li>line 56: literally “LB” = 0x4c42; c.f. section [sec:Load Balancer Meta-Data] and line 8 section [sec:rocpcap].</li>
 
−
<li>line 57: LB version; c.f. line 9 section [sec:rocpcap]</li>
 
−
<li>line 58: protocol; c.f. line 10 section [sec:rocpcap], section [sec:roc]</li>
 
−
<li>line 150, 301: Gate to ensure in packet is for LB</li>
 
−
<li>line 174: '''epoch''' is determined by '''largest''' prefix match of '''tick'''; c.f. section X lines 18,25 for epochs 0,1 respectively</li>
 
−
<li>line 185: lower 9 bits of '''tick''' used for member slot indexing on round-robin basis; all 512 member slots require population for all '''active''' epochs; c.f. line 33 section [sec:lbcntrlpln]</li>
 
−
</ul>
 
  
−
== LB Control Plane ==
+
[https://science.osti.gov/-/media/ascr/ascac/pdf/meetings/202306/Brown_IRI_ASCAC_2023206.pdf IRI Vision]
−
This is the Control Plane CLI configuration script:
 
−
<ul>
 
−
<pre>
 
−
1    table_add
 
−
2    dst_filter_table
 
−
3            NoAction
 
−
4            0x00aabbccddee                      # set FPGA MAC = 00.170.187.204.221.238
 
−
5            0x0800                              # ipv4
 
−
6            0x0000000000000000000000000a010203  # set FPGA IPV4 meta_ipdst = 10.1.2.3
 
−
7            =>
 
−
8    table_add
 
−
9        dst_filter_table
 
−
        NoAction
 
−
        0x00aabbccddee                      # set FPGA MAC = 00.170.187.204.221.238
 
−
        0x86dd                              # ipv6
 
−
        0xfe800000000000000000000000000002  # set FPGA IPV6 meta_ipdst = 254.128.00.0 ??  0.3
 
−
        =>
 
−
table_add
 
−
    epoch_assign_table
 
−
        do_assign_epoch
 
−
        0x0000000000000000/0    # tick/lpm (match high 0b); all ticks = epoch 0
 
−
        =>
 
−
            0x00000000          # epoch
 
−
            64                  # priority (low is higher)
 
−
table_add
 
−
    epoch_assign_table
 
−
        do_assign_epoch
 
−
        0x0000000000000010/60  # tick/lpm (match high 60b); 16 ticks = epoch 1
 
−
        =>
 
−
            0x00000001          # epoch
 
−
            5                  # priority (low is higher)
 
−
table_add
 
−
    load_balance_calendar_table
 
−
        do_assign_member
 
−
        0x00000000  # meta_epoch
 
−
        0x00a      # calendar_slot or tick lsb ; 0x00a = 10; see line #42 ROC PCAP Generator Script
 
−
        =>
 
−
            0x0000  # member_id
 
−
table_add
 
−
    load_balance_calendar_table
 
−
        do_assign_member
 
−
        0x00000001  # meta_epoch
 
−
        0x014      # calendar_slot
 
−
        =>
 
−
            0x0000  # member_id
 
−
table_add
 
−
    member_info_lookup_table
 
−
        do_ipv4_member_rewrite
 
−
        0x0800              # ipv4
 
−
        0x0000              # meta_member_id
 
−
        =>
 
−
            0x112233445566  # dst MAC
 
−
            0xaabbccdd      # dst IPV4 IP
 
−
            0x4556          # dest Port
 
−
table_add
 
−
    member_info_lookup_table
 
−
        do_ipv6_member_rewrite
 
−
        0x86dd                                  # ipv6
 
−
        0x0000                                  # meta_member_id
 
−
        =>
 
−
            0x112233445566                      # dst MAC
 
−
            0xfe800000000000000000000000000003  # dst IPV6 IP
 
−
            0x4556                              # dest Port
 
  
−
run_traffic packets
+
[https://arxiv.org/pdf/2111.05155 A horizontally scalable online processing system for trigger-less data acquisition]
  
−
</pre>
+
[https://arxiv.org/pdf/2212.11032 The-triggerless-data-acquisition-system-of-the-XENONnT-experiment]
−
<li>lines 1,8: Sets the LB MAC address, IP address for IPV4, IPV6 respectively.</li>
 
−
<li>line 15: Set up Epoch 0 to match all ticks at low priority (=64)</li>
 
−
<li>line 22: Set up Epoch 1 to match ticks for which the 2nd least significant nibble=1 and the most significant nibble is arbitrary designating 16 possible tick values for Epoch 1 at higher priority (=5).</li>
 
−
<li>line 29: Designate member (end-node) 0 for calendar slot 0xa, Epoch 0</li>
 
−
<li>line 36: Designate member (end-node) 0 for calendar slot 0x14, Epoch 1</li>
 
−
<li>line 43: Designate member 0 for IPV4 packets as MAC=0x11223344556, IP=0xaabbccdd, Port=0x4556</li>
 
−
<li>line 52: Designate member 0 for IPV6 packets as MAC=0x11223344556,IP=0xfe8000000000000000000000000000, Port=0x4556</li>
 
−
</ul>
 
  
−
== Control Plane (Host CPU) Processing ==
+
[https://indico.cern.ch/event/783429/contributions/3378959/attachments/1829959/2996545/khennessy_cepc_dune_daq_v1.pdf DUNE triggerless DAQ]
  
−
[[File:LB-cntrl-pln.png|border|"Control Plane Flow Chart"]]
+
[https://indico.jlab.org/event/378/contributions/6050/attachments/5093/6351/20200513_JLab_Streaming_Readout.pdf Streaming Mode DAQ at JLab]
−
<ul>
 
−
<li>Software Initialization Steps
 
−
<ul>
 
−
<li>PROPOSAL: A low-level C Software library will be provided to allow insertion/deletion of table entries into each of the P4 tables. All other SW will likely need to be written by the user of the LB pipeline.</li>
 
−
<li>Populate Input Packet Filter Table
 
−
<ul>
 
−
<li>Table Insert <span class="math">⟨</span>0x800, LB IPv4 Addr, LB UDP Dst<span class="math">⟩</span></li>
 
−
<li>Table Insert <span class="math">⟨</span>0x86dd, LB IPv6 Addr, LB UDP Dst<span class="math">⟩</span></li>
 
−
</ul></li>
 
−
<li>Populate Load Balance Member Table: For each LB Member
 
−
<ul>
 
−
<li>Allocate next free Member ID number from SW pool</li>
 
−
<li>Table Insert
 
−
<ul>
 
−
<li>K: <span class="math">⟨</span>0x800, Member ID<span class="math">⟩</span></li>
 
−
<li>V: <span class="math">⟨</span>IPv4 Rewrite, 0, Next-Hop MAC DA, Worker IPv4 Dst, Worker UDP Dst Port<span class="math">⟩</span></li>
 
−
</ul></li>
 
−
<li>Table Insert
 
−
<ul>
 
−
<li>K: <span class="math">⟨</span>0x86dd, Member ID<span class="math">⟩</span></li>
 
−
<li>V: <span class="math">⟨</span>IPv6 Rewrite, 0, Next-Hop MAC DA, Worker IPv6 Dst, Worker UDP Dst Port<span class="math">⟩</span></li>
 
−
</ul></li>
 
−
</ul></li>
 
−
<li>Populate Load Balance to Member Map Table
 
−
<ul>
 
−
<li>Allocate next free Calendar Epoch number from SW pool</li>
 
−
<li>Assign all active LB Member IDs to the 512 Calendar Slots
 
−
<ul>
 
−
<li>Any members can occur between 0-512 times in the calendar</li>
 
−
<li>A member occurring more times in the calendar has a higher “weight” and will be more likely to be assigned an event within this Calendar Epoch</li>
 
−
<li>All 512 slots MUST have a member assigned to them or events that target the empty slot will be entirely discarded by the load balancer</li>
 
−
</ul></li>
 
−
<li>For each Calendar Slot Table Insert
 
−
<ul>
 
−
<li>K: <span class="math">⟨</span>Calendar Epoch, Calendar Slot<span class="math">⟩</span></li>
 
−
<li>V: <span class="math">⟨</span>LB Member ID<span class="math">⟩</span></li>
 
−
</ul></li>
 
−
</ul></li>
 
−
<li>Populate the Calendar Epoch Assignment Table
 
−
<ul>
 
−
<li>Assign all Event IDs (Timestamps) to the newly allocated Calendar Epoch</li>
 
−
<li>Table Insert
 
−
<ul>
 
−
<li>K: <span class="math">⟨</span>*<span class="math">⟩</span></li>
 
−
<li>V: <span class="math">⟨</span>Calendar Epoch<span class="math">⟩</span></li>
 
−
</ul></li>
 
−
</ul></li>
 
−
<li>The load balancer will now assign each Rx’d packet to exactly one of the LB members based on the Event ID contained in the packet. The mapping will remain consistent for any given Event ID within an Epoch since the Calendar and Member tables cannot change within a given Epoch.</li>
 
−
</ul></li>
 
−
<li>Making Changes to the Load Balancer ConfigurationThis section assumes that the load balancer is in-service and as such, care must be taken to avoid service disruption during reconfiguration. If the load balancer is out-of-service, you can reconfigure it using the Initialization steps above without care for disruption.
 
−
Any Epoch that is reachable (connected) via the Calendar Epoch Assignment table MUST not be changed. In-service reconfiguration of the load balancer is done by the following steps.
 
−
<ul>
 
−
<li>Allocate the next free Calendar Epoch ID: Once we’re done all the rest of the updates, we’ll activate this new Epoch</li>
 
−
<li>Insert new entries into the Load Balance Member Table for any entries that need to be different in the next Epoch</li>
 
−
<li>Compute and insert an entirely new calendar into the Load Balance to Member Map Table using the next Calendar Epoch ID</li>
 
−
<li>Choose an Event ID in the (near) future which will become the boundary between the current Epoch and the new Epoch.</li>
 
−
<li>Compute a set of Ternary prefix matches over the Event ID space which describe the entire range of Event IDs from the start of the current Epoch up to the start of the new Epoch.</li>
 
−
<li>Program the ternary prefix matches into the Calendar Epoch Assignment Table</li>
 
−
<li>Update the wildcard match in the Calendar Epoch Assignment Table to point to the new Epoch</li>
 
−
</ul></li>
 
−
<li>The new Epoch is activated and MUST NOT be changed</li>
 
−
<li>After waiting an appropriate time for all events from the previous Epoch to have quiesced, perform the following cleanup steps.
 
−
<ul>
 
−
<li>Delete the ternary prefix matches for the previous Epoch from the Calendar Epoch Assignment Table. This disconnects all references for the previous Epoch to the rest of the pipeline tables.</li>
 
−
<li>Delete the Calendar for the previous Epoch</li>
 
−
<li>Delete the Member rewrites for the previous Calendar</li>
 
−
</ul></li>
 
−
</ul>
 
  
−
[[File:esnet-JLab-network-diagram-v002d-lb.png|border|1500px|link=|"Load Balancer/Host CPU Processing"]]
+
[http://www.scholarpedia.org/article/Real-time_data_analysis_in_particle_physics Real-time data analysis in particle physics]
−
= Reassembly Engine Processing =
 
−
Time-Stamp aggregated data transferred through network equipment will be fragmented and require reassembly by the RE on behalf of the targeted compute farm destination node. Several candidate designs are being considered as depicted in Figure x:
 
−
<ul>
 
−
<li>RE resident () in each compute node in an FPGA accelerated Network Interface Card (NIC) (e.g., Xilinx SN1000)</li>
 
−
<li>RE resident () in each compute node CPU operating system.</li>
 
−
<li>RE centralized () in an FPGA residing in a compute farm switch.</li>
 
−
</ul>
 
−
Additionally, end compute nodes are individually responsible for informing the LB host CPU of status such that the LB host CPU can make informed decisions in comprising an effective load balancing strategy for the FPGA resident LB.
 
−
[[File:esnet-JLab-network-diagram-v002d-re.png|border|"Compute Farm Reassembly/Feedback"]]
 
−
= Initial Test Configuration =
 
−
Figure X depicts a notional EJFAT initial test configuration using commonly available Unix utilities and not initially using a network switch. The sequence of this test configuration is as follows:
 
−
<ul>
 
−
<li>Generate a representative ROC PCAP file using the script in section [sec:rocpcap]. This data should emulate a data stream from the DAQ system exhibiting desired sequences in the stream that will test the functionality of the LB.</li>
 
−
<li>Configure the LB P4 load balancer per the guidelines set forth in section [sec:lbcntrlpln].</li>
 
−
<li>Use '''tcpreplay''' to send the ROC PCAP file via the test host CPU NIC into the FPGA resident LB via the FPGA’s (bidirectional) QSFP28 optical port.</li>
 
−
<li>Use '''tcpdump''' to capture the LB response sent back into the test host CPU NIC into an LB PCAP file</li>
 
−
<li>Use '''Wireshark'''/tshark or other to decode, render, and examine the LB PCAP file to ascertain if the LB provided the correct response. Decoding should be facilitated by using the '''Protocol''' field in the LB meta-data specified in section [sec:Load Balancer Meta-Data] with add-ins for the bit structures defined in sections [sec:Load Balancer Meta-Data], [sec:Reassembly Engine Meta-Data].</li>
 
−
<li>Use an '''LB config''' process running on the host CPU to setup test conditions with the destination compute farm to alter the LBs routing strategy using the information in sections [sec:cntrl-pln], [sec:lbcntrlpln].</li>
 
−
</ul>
 
−
[[File:ejfat-u280_tst0.png|border|EJFAT Test Configuration"]]
 
−
== ROC PCAP Generator Script ==
 
−
The following Scapy test script can be used to generate the ROC PCAP file shown in Figure X:
 
−
<pre>
 
−
#!/usr/bin/env python3
 
  
−
from scapy.all import *
+
[https://indico.cern.ch/event/659612/contributions/2690262/attachments/1591386/2518642/triggerintro4.pdf Intro to Triggering]
  
−
class UDPLB(Packet):
+
[https://wiki.jlab.org/epsciwiki/images/8/8b/SRO_LDRD_Test_Plan_2024v0.8.pdf SRO Test Plan]
−
    name = "UDP LB Packet"
 
−
    fields_desc = [
 
−
        XShortField("magic", 0x4C42),
 
−
        XByteField("version", 1),
 
−
        XByteField("proto", 0),
 
−
        XLongField("tick", 0),
 
−
    ]
 
  
−
class EVIO6Seg(Packet):
+
== Edge to Core Test Equipment: ==
−
    name = "EVIO6 Segment"
 
−
    fields_desc = [
 
−
        BitField("version", 0, 4),
 
−
        BitField("reserved", 0, 10),
 
−
        FlagsField("flags", 0, 2,
 
−
                  ["last",
 
−
                    "first",]),
 
−
        XShortField("rocid", 0),
 
−
        XIntField("offset", 0),]
 
  
−
bind_layers(UDPLB, EVIO6Seg, {'proto': 1})
+
# [https://jeffersonlab.sharepoint.com/:x:/r/sites/SciComp/Shared%20Documents/EPSCI/EJFAT/Edge-to-Core-Test-Stand-12102021.xlsx?d=w8de06c441cd442fd8d3f1b7d7983028d&csf=1&web=1&e=wKS9Lh Price Estimate Spreadsheet]
−
 
+
# [https://jeffersonlab.sharepoint.com/:b:/r/sites/SciComp/Shared%20Documents/EPSCI/EJFAT/EJFAT-Test-Stand-Network-Map.pdf?csf=1&web=1&e=iWvvet Networking Diagram], [[Media:20240209_EJFAT_diagram.pdf | Updated (PDF)]] (from Brent 2024-02-09)
−
EVIO6_BLOB_SIZE = 1050
+
# [https://misportal.jlab.org/reqs/pr/viewPr.do?prNum=408549 PR408549] : Requisition 1 of 2 :
−
evio6_blob = bytearray(EVIO6_BLOB_SIZE)
+
## [https://jeffersonlab.sharepoint.com/:w:/r/sites/SciComp/Shared%20Documents/EPSCI/EJFAT/EJFAT-Test-Stand-Servers-SOW.docx?d=w19107d52332948a0b2924b13939c3f64&csf=1&web=1&e=CS6Ub8 Statement of Work for Servers]
−
EVIO6_SEG_SIZE = 100
+
## 1/13/2022: EJFAT team decided to solicit two bid responses, one with MLX NIC and one without. Response from Procurement is "I can ask for the two separate quotes. If you are going to purchase both option (with & without add-in cards), once I receive the quotes back, you will have submit a new PR to cover the option (without add-in cards)."
−
 
+
## 1/18/2022: Question from KOI Computers: "please clarify what the part number for the NVIDIA Dual Port ConnectX-6". Replied with part # MCX623106AN-CDAT.
−
with scapy.utils.PcapWriter('packets_in.pcap', linktype=DLT_EN10MB) as w:
+
## 1/24/2022: Requisition currently open for bid responses from vendors. Due date is COB 1/24/2022.
−
    w.write_header([])
+
## 1/27/2022: PO awarded to Atipa for 6 servers and 1 file-server with FPGA and MLX SmartNIC. Expected delivery date from vendor is 5/31/2022.
−
    for offset in range(0, EVIO6_BLOB_SIZE, EVIO6_SEG_SIZE):
+
# [https://misportal.jlab.org/reqs/pr/viewPr.do?prNum=408870 PR408870] [https://misportal.jlab.org/reqs/pr/viewPr.do?prNum=408938 PR408938] Requisition 2 of 2: Statement of Work for Switches & Cables
−
        flags = []
+
## 1/14/2022: PRs for the switches, transceivers and fiber have been submitted. I added (4) 2km 100G transceivers to support dual 100G connections between the switches.  We can always upgrade to 400G in the future, if needed.
−
        if offset == 0:
+
# [https://misportal.jlab.org/reqs/pr/viewPr.do?prNum=409850 PR409850] [https://developer.nvidia.com/arm-hpc-devkit NVIDIA ARM HPC Developer Kit]
−
            # first segment
+
## Hardware Specifications for dev kit
−
            flags.append("first")
+
##: [[Model]] GIGABYTE G242-P32, 2U server
−
        if offset + EVIO6_SEG_SIZE > EVIO6_BLOB_SIZE:
+
##: [[CPU]] 1x Ampere Altra Q80-30 (Arm processor)
−
            # last (possibly short) segment
+
##: [[Memory]] 512G DDR4 memory
−
            flags.append("last")
+
##: [[Storage]] 6TB SAS/ SATA 3.5″
−
 
+
##: [[GPU]] 2x NVIDIA A100 GPU
−
        p = Ether(dst="00:aa:bb:cc:dd:ee", src="00:11:22:33:44:55")/IP(dst="10.1.2.3", src="10.1.2.2")/UDP(sport=50000,dport=0x4c42)/UDPLB(tick=10)/EVIO6Seg(rocid=0xabc, flags=flags, offset=offset)/Raw(load=evio6_blob[offset:offset+EVIO6_SEG_SIZE])
+
##: [[Network]] 2x NVIDIA® BlueField®-2 E-Series DPU, 200GbE/HDR single-port QSFP56, PCIe Gen4 x16, secure boot enabled, crypto disabled, 16GB on-board DDR, 1GbE OOB management
−
        w.write_packet(p)
 
−
        #print(p.show())
 
−
 
 
−
        p = Ether(dst="00:aa:bb:cc:dd:ee", src="00:01:02:03:04:05")/IPv6(dst="fe80::2", src="fe80::1")/UDP(sport=12345,dport=0x4c42)/UDPLB(tick=20)/EVIO6Seg(rocid=0x123, flags=flags, offset=offset)/Raw(load=evio6_blob[offset:offset+EVIO6_SEG_SIZE])
 
−
        w.write_packet(p)
 
−
        #print(p.show())
 
−
</pre>
 
  
−
== Scapy Installation ==
+
== Resources ==
−
Scapy may be installed by executing the following command at the Unix command line:
+
* [https://jeffersonlab.sharepoint.com/:b:/r/sites/SciComp/Shared%20Documents/EPSCI/EJFAT/u280_po_Signed_21-M0862%20-%20Avnet.pdf?csf=1&web=1&e=PmJfdu First FPGA PO]
−
<pre>    pip3 install --user scapy</pre>
+
* [https://www.jlab.org TBD]
−
== ROC PCAP Meta Data File ==
 
−
In the following file:
 
−
Each row (1...n) in this file matches with the corresponding packet (1...n) in the packets<span>_</span>in.pcap file. ( Lines 42, 46 in section [sec:rocpcap] )
 
−
The simulator loads a line from this file and uses it to initialize the short<span>_</span>metadata struct which is processed along with the corresponding packet data.
 
−
You should find references to “short<span>_</span>metadata” struct in the p4 program.
 
−
I have no idea what the sim does when it has more packets in the .pcap file than the number of lines in the .meta file. In general, all of these fields may be relevant. For the load balancer specifically, the only field that matters is really the “egress<span>_</span>spec” field. On the way in, it contains the *ingress* port. It isn’t touched by the p4 program since we always want the output packet to go back out the exact same interface that it arrived on (after we rewrite the destination header fields of course).
 
−
The p4bm simulator does something like this:
 
−
<pre>   
 
−
open packets_in.pcap for read
 
−
open packets_in.meta for read
 
−
open packets_out.pcap for write
 
−
open packets_out.meta for write
 
−
while (!done) {
 
−
    read packet p from packets_in.pcap
 
−
    read metadata row m from packets_in.meta
 
−
    (out_packet, out_meta) = simulate_the_p4(p, m)
 
−
    write out_packet to packets_out.pcap
 
−
    write out_meta to packets_out.meta</pre>
 
−
The pcap-generator.py script isn’t directly used by the p4bm sim. That’s just something I wrote to help you to make up input packets for the simulator so you didn’t have to make your input packets (as seen in packets<span>_</span>in.pcap) by hand.
 
−
There is a 1:1 correspondence between calls to write<span>_</span>packet() in the pcap-generator.py script and packets in the packets<span>_</span>in.pcap file.
 
−
The .py program spits out 2 interleaved ROC transfers (one transfer is carried in IPv4, the other is in IPv6). Each ROC transfer starts out as 1050 bytes of EVIO6 data. It is then segmented into 100 byte segments, ready to be put into a packet. Each segment gets a EVIO6 Segmentation header added to keep track of which segment it is, then each segment gets a UDP Load Balancer header added to give the load-balancer its context.
 
−
Each call to write<span>_</span>packet() spits out one of the segments in either a IPv4 or IPv6 encapsulation.
 
−
The program might be more clear if you were to duplicate the loop (one loop for IPv4 and one loop for IPv6) and put a single call to write<span>_</span>packet() in each of the loops. That might make it more clear that it was 2 transfers since they would no longer be interleaved.
 
−
The 2 lines in packets<span>_</span>in.meta will be used for the first 2 packets in the packets<span>_</span>in.pcap file. As currently written, one of those lines will be for the first segment in IPv4 and the second line will be for the first segment in IPv6. As I mentioned before, I’m actually not certain what happens after that in the simulator. In practice, it shouldn’t be important for this pipeline since it doesn’t get modified by the p4 program at all.
 
−
== Tshark Plug-in for LB Meta-Data ==
 
−
In the following file:
 
−
== Tshark Plug-in for Payload Seg ==
 
−
In the following file:
 
−
<span>99</span>
 
−
Author, A.N and Another, A. N., 2010, MNRAS, 431, 28.
 
−
= Appendix: EJFAT Processing =
 
−
[[File:esnet-jlab-network-diagram-v002d.png|border|1600px|link=|"EJFAT Load Balanced Transport"]]
 

Latest revision as of 20:52, 19 December 2024

Welcome to the EJFAT Wiki

(ESnet / JLaB FPGA Accelerated Transport)


System Overview:

EJFAT is a collaboration between Energy Sciences Network (ESnet) and Thomas Jefferson National Laboratory (JLab) for proof of concept engineering for accelerated load balancer (LB) using dynamic IP4/6 address forwarding. Dynamic because the forwarding address is chosen dynamically from a collection of destination endpoints based on near real-time destination workload conditions, and accelerated because the forwarding is accomplished with low fixed latency at line rates of up to 200Gbps per FPGA, where in general a functioning LB may consist of up to four FPGAs acting as one logical DP for a total bandwidth capacity of over 1 Tbps. The low, fixed latency is achieved by utilization of an appropriately programmed Field Programmable Gate Array (FPGA) to effect the Data Plane (DP) functions of the LB.

EJFAT System Status

ejfat-1

  1. 100Gbps NIC: ejfat-1-daq 129.57.177.8
  2. 10Gbps NIC: ejfat-1 129.57.177.131
  3. U280 FPGA: ejfat-1-dp 129.57.177.{9-16} - LAG'd for 200Gbps
  4. LB CP: ejfat-1 129.57.177.131, latest Stable branch
  5. LB: DP latest Stable FW
  6. CP Web UI port 8081

ejfat-2

  1. 100Gbps NIC: ejfat-2-daq 129.57.177.2
  2. 10Gbps NIC: ejfat-2 129.57.177.132
  3. 100Gbps U280 FPGA: ejfat-2-dp 129.57.177.{17-24}
  4. LB CP: ejfat-2 129.57.177.132, latest Stable branch
  5. LB: DP latest Stable FW
  6. CP Web UI port 8082

ejfat-3

  1. 200Gbps NIC: ejfat-3-daq 129.57.177.3
  2. 10Gbps NIC: ejfat-3 129.57.177.133
  3. Two U280s installed - LAG'd for 400Gbps
  4. FW Containers built by Stacey

ejfat-4

  1. 100Gbps NIC: ejfat-4-daq 129.57.177.4
  2. 10Gbps NIC: ejfat-4 129.57.177.134
  3. XDP experiments
  4. 100Gbps U280 FPGA: ejfat-4-dp 129.57.177.{41-48}
  5. LB CP: ejfat-4 129.57.177.134, latest Stable branch
  6. LB: DP latest Stable FW

ejfat-5

  1. 200Gbps NIC: ejfat-5-daq 129.57.177.5
  2. 10Gbps NIC: ejfat-5 129.57.177.135
  3. LB CP: ejfat-5 129.57.177.135, latest Stable branch
  4. 100Gbps U280 FPGA: ejfat-5-dp 129.57.177.{49-56}
  5. LB: DP latest Stable FW
  6. Optical Taps Installed

ejfat-6

  1. 200Gbps NIC: ejfat-6-daq 129.57.177.6
  2. 10Gbps NIC: ejfat-6 129.57.177.136
  3. DAOS experiments
  4. Using Ubuntu 24.04 LTS
  5. FW containers built
  6. Waiting for podman compose installation

ejfat-fs

  1. 100Gbps NIC: ejfat-fs-daq 129.57.177.7
  2. 10Gbps NIC: ejfat-fs 129.57.177.130
  3. Hosts NVME memory/disk
  4. 100Gbps U280 FPGA: ejfat-fs-dp 129.57.177.{65-72}
  5. LB CP: ejfat-fs 129.57.177.130, latest Stable branch
  6. LB: DP latest Stable FW
  7. CP Web UI port 8080

Presentations/Papers

date presenter Event links
2021-03-01 G. Heyes EJFAT Proposal Word
2021-10-21 M. S. Goodrich Div Brief PDF
2021-11-05 M. S. Goodrich Canisius College PDF
2021-12-03 S. Sheldon ESnet LB Tutorial MP4
2021-12-10 Y. Kumar SRO iX Presentation PPTX
2022-08-05 M. S. Goodrich RT-2022 Presentation PPTX
2022-08-05 M. S. Goodrich, et al. RT-2022 Proceedings PDF
2022-10-20 S. Sheldon, et al. INDIS-2022 PDF
2022-10-24 M. S. Goodrich ACAT-2022 Presentation PPTX
2023-03-17 M. S. Goodrich, et al. ACAT-2022 Proceedings PDF
2023-05-11 M. S. Goodrich, et al. CHEP-2023 Presentation PPTX
2023-10-12 D. Howard, et al. CHEP-2023 Conference Publication PDF
2024-03-11 M. S. Goodrich, et al. ACAT-2024 Presentation PPTX
2024-04-10 M. S. Goodrich, et al. RT-2024 Presentation PPTX
2024-07-31 M. S. Goodrich, et al. ACAT-2024 Proceedings PDF
2024-10-02 S. Veseli​, APS/SDM APS/ALS - EJFAT PPTX

EJFAT Weekly EPSCI Meetings

EJFAT Weekly EPSCI Meetings

EJFAT Weekly Collaboration Meetings

EJFAT Weekly Meetings

Technical Design Overview

EJFAT Technical Design Overview

UDP Packet Header Formats

IRIAD/EJFAT Testbed

UDP Transmission Performance

EJFAT UDP General Information

EJFAT UDP General Performance Considerations

EJFAT UDP Packet Receiving and Core Switching

EJFAT UDP Packet Sending and NUMA Nodes

EJFAT UDP Single Thread Packet Sending and Receiving

Testing Load Balancer Bandwidth

HOW-TOs

How to use Control Plane Web UI

How to Monitor Prometheus

Install a Load Balancer

Test a Load Balancer

How to setup ejfat nodes

How to install, build and use gRPC

How to install, build and use XDP related packages

How to Compute Schedule Density from PID Signals

Enable Jumbo Frames

Network Path MTU Discovery support in the Linux Kernel: 

file: /proc/sys/net/ipv4/tcp_mtu_probing
variable: net.ipv4.tcp_mtu_probing (integer; default: 0; since Linux 2.6.17):

tcp_mtu_probing - INTEGER
	Controls TCP Packetization-Layer Path MTU Discovery.  Takes three values:
	  0 - Disabled
	  1 - Disabled by default, enabled when an ICMP black hole detected
	  2 - Always enabled, use initial MSS of tcp_base_mss.

REFERENCEs

EJFAT Config Planning

JLab EJFAT News Release

EJFAT on FABRIC

EJFAT API

LB Pipeline

Getting Started with EJFAT

IRIAD Workplan

SRO Grand Challenge

ESnet Logical Map

IP Neighbor

Robot Framework

IRI Vision

A horizontally scalable online processing system for trigger-less data acquisition

The-triggerless-data-acquisition-system-of-the-XENONnT-experiment

DUNE triggerless DAQ

Streaming Mode DAQ at JLab

Real-time data analysis in particle physics

Intro to Triggering

SRO Test Plan

Edge to Core Test Equipment:

  1. Price Estimate Spreadsheet
  2. Networking Diagram, Updated (PDF) (from Brent 2024-02-09)
  3. PR408549 : Requisition 1 of 2 :
    1. Statement of Work for Servers
    2. 1/13/2022: EJFAT team decided to solicit two bid responses, one with MLX NIC and one without. Response from Procurement is "I can ask for the two separate quotes. If you are going to purchase both option (with & without add-in cards), once I receive the quotes back, you will have submit a new PR to cover the option (without add-in cards)."
    3. 1/18/2022: Question from KOI Computers: "please clarify what the part number for the NVIDIA Dual Port ConnectX-6". Replied with part # MCX623106AN-CDAT.
    4. 1/24/2022: Requisition currently open for bid responses from vendors. Due date is COB 1/24/2022.
    5. 1/27/2022: PO awarded to Atipa for 6 servers and 1 file-server with FPGA and MLX SmartNIC. Expected delivery date from vendor is 5/31/2022.
  4. PR408870 PR408938 Requisition 2 of 2: Statement of Work for Switches & Cables
    1. 1/14/2022: PRs for the switches, transceivers and fiber have been submitted. I added (4) 2km 100G transceivers to support dual 100G connections between the switches. We can always upgrade to 400G in the future, if needed.
  5. PR409850 NVIDIA ARM HPC Developer Kit
    1. Hardware Specifications for dev kit
      Model GIGABYTE G242-P32, 2U server
      CPU 1x Ampere Altra Q80-30 (Arm processor)
      Memory 512G DDR4 memory
      Storage 6TB SAS/ SATA 3.5″
      GPU 2x NVIDIA A100 GPU
      Network 2x NVIDIA® BlueField®-2 E-Series DPU, 200GbE/HDR single-port QSFP56, PCIe Gen4 x16, secure boot enabled, crypto disabled, 16GB on-board DDR, 1GbE OOB management

Resources

Retrieved from "https://wiki.jlab.org/epsciwiki/index.php?title=EJFAT&oldid=5963"