EJFAT Group Meeting Dec. 1, 2022
Jump to navigation
Jump to search
The meeting time is 11:00am.
Connection Info:
You can connect using ZoomGov Video conferencing (ID: 161 012 5238). (Click "Expand" to the right for details -->):
Meeting URL https://jlab-org.zoomgov.com/j/16184166709 Meeting ID 161 8416 6709 Passcode (none) Want to dial in from a phone? Dial one of the following numbers: US: +1 669 254 5252 or +1 646 828 7666 or +1 551 285 1373 or +1 669 216 1590 or 833 568 8864 (Toll Free) Enter the meeting ID and passcode followed by # Connecting from a room system? Dial: bjn.vc or 199.48.152.152 and enter your meeting ID & passcode
Agenda:
- Previous meeting
- Announcements:
- Status:
- Using ESnet FPGA f/w build 28 April
- Specs
- Jumbo Frames
- arp, ping, ICMP filtering
- Port entropy
- Script based LB Control Plane
- EJFAT VLAN Open for Business:
- Hosts running Ubuntu 20.04
- 1 Gbs i/f are: ejfat-1, ejfat-2, ejfat-3, ejfat-3, ejfat-5, ejfat-6, ejfat-fs
- 100 Gbs i/f are: ejfat-1-daq, ejfat-2-daq, ejfat-3-daq, ejfat-3-daq, ejfat-5-daq, ejfat-6-daq, ejfat-fs-daq
- LBs: 172.19.22.241-247, indra-s2
- 172.19.22.241 - Currently reserved for Carl
- 172.19.22.242 - Currently reserved for Stacey - BIOS / Kernel mods pending
- 172.19.22.243 - Currently reserved for Vardan
- 172.19.22.247 - Currently reserved for Mike
- indra-s2 upgraded to Ubuntu 20.04 LB sucessfully installed, now on EJFAT VLAN / DAQ lo-speed networks via Indra-Lab switch
- /daq-fs/gyurjyan self-contained ERSAP event processing package
- Spare EJFAT equip loaners:
- (4) DAQ dev machines indra-s[1-3] 129.57.29/109.23[0-2]
- alkaid: 24 Xeon Gold 3.4 GHz cores, 100Gbs
- indra-s1: 24 Xeon Gold 3.0 GHz cores, 100Gbs
- indra-s2: 32 Xeon Gold 3.2 GHz cores, 100Gbs
- indra-s3: 32 Xeon Gold 2.3 GHz cores, 100Gbs, 750GB ram disk
- (4) DAQ Farm machines dafarm6[1-4] currently on 129.57.29.17[1-4] - each 32 Xeon 2.0Ghz cores - 1 Gbs NIC + (4) 10Gbs Spare NICs
- (4) Unbuilt DAQ Farm machines - each 32 Xeon 2.0Ghz cores - 1 Gbs NIC + (4) 10Gbs Spare NICs
- (4) DAQ dev machines indra-s[1-3] 129.57.29/109.23[0-2]
- PR408870 PR408938 (2) 100Gbs Arista switches
- All equipment recd
- Arista switch #1 for EJFAT VLAN to be installed (swapped) soonest
- Arista switch #2 - TBD
- FPGA LB Throughput - max sustained 90Gbs with s/w data generation
- RT 2022 Paper - submitted August 22 - up to 8 mos. review process
- Using ESnet FPGA f/w build 28 April
- Current Activities
- Wrapping up EJFAT-I - moving to IRIAD / EJFAT II start of CY 2023
- Tring to pass packets from 172.19.22.* and 129.57.109.*
- Test plan for all pieces required for integration test
- Plan A: CLAS12 files -> Carl C++ Packetizer -> LB -> Carls's C++ Reassembler -> ERSAP
- Plan B: CLAS12 files -> ERSAP C++ Packetizer -> LB -> Mike's C++ Reassembler -> (tcp) -> ERSAP
- Plan C: Simulated Packets -> LB -> Mike's C++ Reassembler -> Simulated Host Loading/Feedback
- Bug in LB DP - not forwarding packets of length < (18 bytes + LB Header size)
- Control Plane
- Compute Farm Feed Back Monitor
- Have working RL (Q-Learning) Schedule Density Adjuster (to be integrated)
- Need to define Optimality Criterion for Schedule Density
- DP Supervisor
- Demonstrate CP based flexibility/elasticity
- Paper for ACAT 2022 Conf Proc. (TBD)
- Resubmitting Paper for RT 2022 Conf Proc. (IEEE-TNS)
- ESnet Update:
- New toolkit for detailed FPGA packet tracking
- Uses DPDK
- Can send packets via PCIe to FPGA
- Working on direct delivery of data from FPGA to host memory via DMA
- Proposing special packet for CP event_id sync with sender - discuss with DAQ group
- IPV6 neighbor discovery - in process
- LB F/W Installation Manual - with PCI buffer allocation assurance steps
- In ESnet Legal Review
- Support C libraries for LB Host Control Plane - needs completion of C API doc
- ESnet smartnic open-source GitHub repo - in legal review
- ESnet private, forkable Jlab P4 and simulations GitHub repo - in legal review
- FPGA LB data generation capability
- New toolkit for detailed FPGA packet tracking
- Back Burner / Downstream / Notes:
- FireHose Benchmark
- Network Performance:
- cpusets
- Pktgen-DPDK
- Receive Side Scaling
- Packet Loss Studies / LB Round Robin Studies - migrated to Compute Center / EJFAT / EJFAT VLAN
- DAQ/VTP Data Generation Test Harness - 40 Gbs initially - after FEG ???
- Host NICs
- Host S/W Reassembly - better algorithms, buffering, asynchronicity, etc.
- EJFAT UDP Transmission Performance
- Need better parameters for event reassembly/reconstruction
- DPDK will own NIC bypassing kernel, etc. - ESnet reports can stream 100 Gbps using DPDK; Pktgen-DPDK
- Look at ROCE / NIC
- SPDK for hyper storage performance
- ESnet UDP tuning
- SLURM env for EJFAT VLAN (Hess)
- Vivado Licesnses for new machines - AI/ML ? D. Lawrence POC
- Open Nic Shell
- Hall-B FT calorimeter and hodoscope streaming readout test
- May be able to use Abbott's indra-s1 setup
- May be able to use new VTP f/w with Hall-B VTP's
- CODA 3.10 + ERSAP for new VTP f/w
- CODA 2.0 (non-streaming) for old VTP f/w
- Diagram
- Hall-B to start taking data June 8
- Hall B VTPs on .167. subnet
- HOSS
- parallelize writing of raw data files
- distribute raw data across multiple compute nodes for calibration skims
- 1 Gbs at hi-luminosity
- Hall-D comms with DAQ 109 subnet require network customization; (EJFAT subnet)
- Hall-D EJFAT use case
- Hall-D EJFAT Network Diagram
- IPV6 testing
- AOT