JIRIAF Meeting Oct. 12 2023
Jump to navigation
Jump to search
Connection Info:
You can connect using the following link (Meeting ID: 161 690 3130). (Click "Expand" to the right for details -->):
One tap mobile: US: +16692545252,,1608518798# or +16468287666,,1608518798#
Meeting URL: https://jlab-org.zoomgov.com/j/1616903130?pwd=cjg3U0Y4SndXL05SeFBmQjVHZkhrQT09&from=addon
Meeting ID: 161 690 3130
Passcode: 018094
Join by Telephone
For higher quality, dial a number based on your current location.
Dial:
US: +1 669 254 5252 or +1 646 828 7666 or +1 551 285 1373 or +1 669 216 1590 or 833 568 8864 (Toll Free)
Meeting ID: 161 690 3130
International numbers
Join by SIP
1616903130@sip.zoomgov.com
Join by H.323
161.199.138.10 (US West)
161.199.136.10 (US East)
Meeting ID: 160 851 8798
Passcode: 018094
Agenda:
- Announcements
- Current status of the JRM
- Running JRM as a SWIF Job
- SSH tunnel between the processing node (JRM) and the JIRIAF server (Kubernetes API server).
- "Hello World" workflow within the JRM.
- Digital tween for an abstract actor of the data processing system.
- Start with a simple digital agent for a processor node (JRM). Use queuing theory to define the state of the JRM in terms of
- traffic intensity, time data quantum spends in a data pipeline, etc.
- Use workflow-provided measures to define data arrival and servicing rates
- Use other indirect measures (e.g., reassembly engine fifo levels for streaming processing case) to estimate these rates if workflow does not provide information.
- Define physical actor control channels.
- Start with a simple digital agent for a processor node (JRM). Use queuing theory to define the state of the JRM in terms of
- JIRIAF Available Resources DB Table
- Suggested schema:
- computing_suite (e.g., HPC, GPU, etc.)
- resource_ID (unique identifier for each resource)
- resource_type (e.g., CPU, GPU, memory, etc.)
- number_of_cores
- memory
- available_walltime (decreasing over time)
- Available wall time will decrease over time.
- The wall time decrease below a certain threshold will trigger resource release.
- The wall time for the opportunistic resource (e.g., commercial cloud resource) will be clearly indicated in the table.
- Suggested schema:
- JIRIAF User Resource Requests DB Table
- Suggested schema:
- name (user or group)
- resource_type (e.g., CPU, GPU, memory, etc.)
- number_of_cores
- memory
- wall time
- workflow_container_info (e.g., container ID, Docker image, etc.)
- data_type (streaming or batch files)
- can_it_be_horizontally_distributed/scaled (yes/no)
- Important question: Who is responsible for data provisioning, JIRIAF, or user workflow management system?
- Suggested schema:
- JIRIAF Workflow Resource Matching Service (JMS)
- Choice of technology/language: Python?
- The algorithm must be able to fit multiple user requests into a bigger resource (workflow Tetris).
- Running CLAS12 Data-Stream Processing Pipeline
- Docker image for the CLAS12 data-stream processing pipeline.
- Status of the Front-End
- JIRIAF Entry Web Page
- Connection to MariaDB Back-End
- JIRIAF REST API
- Authentication of JIRIAF Users
- AOT
Useful References