EJFAT EPSCI Meeting Oct. 30, 2024

The meeting time is 2:30pm.

Connection Info:

You can connect using Teams Link. (Click "Expand" to the right for details -->):

Meeting URL
	https://teams.microsoft.com/l/meetup-join/19%3ameeting_M2ExMzNiYmItNTNiMi00NzE4LTkwYWItOTQwYjJmMTJiMDhh%40thread.v2/0?context=%7b%22Tid%22%3a%22b4d7ee1f-4fb3-4f06-9037-2b5b522042ab%22%2c%22Oid%22%3a%22ff03cbe2-66d5-4741-bf32-ff1c69fa2fcc%22%7d

Meeting ID
273 279 114 915

Passcode
Rkypzi

Agenda:

1. Previous meeting

2. Announcements:
   1. SuperComputing24 Atlanta, GA from Nov 17-22, 2024
   2. Mike PD Duty 11/06/2024 - 11/20/2024
3. Status
4. Topics
   1. Local CP testing
      1. Working on tooling
      2. Looking at Aug 29 IRI test data
      3. Unable to reproduce anomalous balancing on Aug29 (Derek also unable to reproduce)
   2. IRI Test Development:
      1. Last Test Thursday Oct 10
      2. LB version = ESnet ??? version
      3. Data Source:
         1. JLAB, CLAS12, pre-triggered events - 1 channel
      4. Data Sink:
         1. Perlmutter - 80 nodes
         2. ORNL/ESnet/JLab IRI Testbed / Defiant - 4 nodes allocated
         3. JLab - 7 nodes available
         4. FABRIC - nodes available
         5. ERSAP
      5. Test Plans - JLab, ESnet, NERSC:
      6. Prometheus Dashboards
      7. The Prometheus dashboard can be accessed on port 1717 of the ejfat-fs node. The test data is located at "100g-nersc-ornl / ejfat-nersc-ornl". The test time interval is around UTC 17:05 to 18:20 on August 29, 2024. To log in to Grafana, please use the username and password "ejfat".
   3. JLab FEG/SRO
      1. will use interim UDP solution for event sync
      2. Special Events Issue - Completely Out-of-band
         1. Cloud message queue solutions include Kafka and RabbitMQ, ...
         2. LB isolation from any non-LB processing
   4. E2SAR 0.1.2
      1. segmentation/reassembly complete
      2. .deb packages for Ubuntu 20, 22 and 24 are now available (they contain E2SAR library, headers, executables as well as appropriate versions of gRPC and Boost dependencies, all installed under /usr/local), as well as the latest Docker image
   5. ALS: Next is Tech Talk
   6. IB
   7. ejfat-3 - two FPGA DP built, running - with 4-port LAG at switch, needs CP installation
   8. ejfat-6 - Ubuntu 24.04 installed - esnet-smartnic-fw build succeeds with podman, issues with podman compose
   9. SC poster submitted - demo in works
   10. Storage
       1. SSD drives on ejfat-fs - 20TB used of 28TB - mounted for EJFAT farm - permissions issue
       2. Ram Disks: 1TB Total Mem on ejfat-fs, 0.5 TB others
       3. Repurposing /dev/sdb to be used for user storage
       4. Storage Areas NOT to be backed up could be marked as scratch
       5. Have an opportunity to consolidate wares on SSD for consistent SC backup procedure ?.
   11. Experiment Halls - beam returns late January/February 2025
   12. Ubuntu 20.04 LTS - support ends in 2025 - next ESnet target 22.04
   13. CP: Control Web UI (127.0.0.1:8081) needs SSH tunnel
   14. EJFAT II
   15. ESnet interested in partnering for beachhead in FPGA/GPU AI space
       1. Separate Project
       2. FAST program coming up
       3. May get free help from Xilinx
       4. Might target VERSA release
5. Resources:
   1. HPDF
   2. EJFAT API
   3. EJFAT Status
6. U280’s are discontinued.
   1. New LB purchases U55C
   2. U55C bitfiles available 1 year out
   3. U280 Supported indefinitely
7. ACAT 2024 Paper - In Review

Notes

1. LLDP needs IOMMU
2. FW containers need boot init script
3. EJFAT nodes:
   1. 16 NUMA domains
   2. DPDK must run portmode driver on CPU in NUMA domain of FPGA for LLDP messages
4. Event splitting being investigated with E2SAR debug tools
5. EJFAT II
   1. architecture change in control/data paths for FPGA (SRIOV)
   2. adding PCIE AES
6. AOT

---