Difference between revisions of "JIRIAF Meeting Oct. 12 2023"
Jump to navigation
Jump to search
(Created page with " === Connection Info: === <div class="toccolours mw-collapsible mw-collapsed"> You can connect using [https://jlab-org.zoomgov.com/j/1616903130?pwd=cjg3U0Y4SndXL05SeFBmQjVHZk...") |
|||
Line 38: | Line 38: | ||
# Announcements | # Announcements | ||
# Current status of the JRM | # Current status of the JRM | ||
− | ## Running JRM as a | + | ## Running JRM as a SWIF Job |
## SSH tunnel from the processing node (JRM) to the JIRIAF server, running Kubernetes API server. | ## SSH tunnel from the processing node (JRM) to the JIRIAF server, running Kubernetes API server. | ||
## "Hello World" workflow within the JRM. | ## "Hello World" workflow within the JRM. | ||
− | # JIRIAF Available Resources DB Table | + | ## Digital tween for an abstract actor of the data processing system. |
+ | ### Start with a simple digital agent for a processor node (JRM) | ||
+ | ### Use queuing theory to define the state of the JRM in terms of | ||
+ | #### traffic intensity, time data quantum spends in a data pipeline, etc. | ||
+ | #### Use workflow-provided measures to define data arrival and servicing rates | ||
+ | #### Use other indirect measures (e.g., reassembly engine fifo levels for streaming processing case) to estimate these rates if workflow does not provide information. | ||
+ | ### Define physical actor control channels. | ||
+ | # JIRIAF Available Resources DB Table | ||
## Suggested schema: | ## Suggested schema: | ||
### computing_suite (e.g., HPC, GPU, etc.) | ### computing_suite (e.g., HPC, GPU, etc.) | ||
Line 52: | Line 59: | ||
#### The wall time decrease below a certain threshold will trigger resource release. | #### The wall time decrease below a certain threshold will trigger resource release. | ||
#### The wall time for the opportunistic resource (e.g., commercial cloud resource) will be clearly indicated in the table. | #### The wall time for the opportunistic resource (e.g., commercial cloud resource) will be clearly indicated in the table. | ||
− | # JIRIAF User Resource Requests DB Table | + | # JIRIAF User Resource Requests DB Table |
## Suggested schema: | ## Suggested schema: | ||
### name (user or group) | ### name (user or group) |
Revision as of 14:45, 12 October 2023
Connection Info:
You can connect using the following link (Meeting ID: 161 690 3130). (Click "Expand" to the right for details -->):
One tap mobile: US: +16692545252,,1608518798# or +16468287666,,1608518798#
Meeting URL: https://jlab-org.zoomgov.com/j/1616903130?pwd=cjg3U0Y4SndXL05SeFBmQjVHZkhrQT09&from=addon
Meeting ID: 161 690 3130
Passcode: 018094
Join by Telephone
For higher quality, dial a number based on your current location.
Dial:
US: +1 669 254 5252 or +1 646 828 7666 or +1 551 285 1373 or +1 669 216 1590 or 833 568 8864 (Toll Free)
Meeting ID: 161 690 3130
International numbers
Join by SIP
1616903130@sip.zoomgov.com
Join by H.323
161.199.138.10 (US West)
161.199.136.10 (US East)
Meeting ID: 160 851 8798
Passcode: 018094
Agenda:
- Announcements
- Current status of the JRM
- Running JRM as a SWIF Job
- SSH tunnel from the processing node (JRM) to the JIRIAF server, running Kubernetes API server.
- "Hello World" workflow within the JRM.
- Digital tween for an abstract actor of the data processing system.
- Start with a simple digital agent for a processor node (JRM)
- Use queuing theory to define the state of the JRM in terms of
- traffic intensity, time data quantum spends in a data pipeline, etc.
- Use workflow-provided measures to define data arrival and servicing rates
- Use other indirect measures (e.g., reassembly engine fifo levels for streaming processing case) to estimate these rates if workflow does not provide information.
- Define physical actor control channels.
- JIRIAF Available Resources DB Table
- Suggested schema:
- computing_suite (e.g., HPC, GPU, etc.)
- resource_ID (unique identifier for each resource)
- resource_type (e.g., CPU, GPU, memory, etc.)
- number_of_cores
- memory
- available_walltime (decreasing over time)
- Available wall time will decrease over time.
- The wall time decrease below a certain threshold will trigger resource release.
- The wall time for the opportunistic resource (e.g., commercial cloud resource) will be clearly indicated in the table.
- Suggested schema:
- JIRIAF User Resource Requests DB Table
- Suggested schema:
- name (user or group)
- resource_type (e.g., CPU, GPU, memory, etc.)
- number_of_cores
- memory
- wall time
- workflow_container_info (e.g., container ID, Docker image, etc.)
- data_type (streaming or batch files)
- can_it_be_horizontally_distributed/scaled (yes/no)
- Important question: Who is responsible for data provisioning, JIRIAF, or user workflow management system?
- Suggested schema:
- JIRIAF Workflow Resource Matching Service (JMS)
- Choice of technology/language: Python?
- The algorithm must be able to fit multiple user requests into a bigger resource (workflow Tetris).
- Running CLAS12 Data-Stream Processing Pipeline
- Docker image for the CLAS12 data-stream processing pipeline.
- Status of the Front-End
- JIRIAF Entry Web Page
- Connection to MariaDB Back-End
- JIRIAF REST API
- Authentication of JIRIAF Users
- AOT
Useful References