Difference between revisions of "JIRIAF Meeting Sep. 28 2023"
Jump to navigation
Jump to search
(Created page with " === Connection Info: === <div class="toccolours mw-collapsible mw-collapsed"> You can connect using [https://jlab-org.zoomgov.com/j/1602836193?pwd=TkdVUnlkbTVVek5XV01TN3Jzd0...") |
|||
Line 39: | Line 39: | ||
## The initial manuscript outlining our project has been submitted to the EPJ. | ## The initial manuscript outlining our project has been submitted to the EPJ. | ||
## The JIRIAF LDRD project has been chosen to receive funding for its second year of operation. | ## The JIRIAF LDRD project has been chosen to receive funding for its second year of operation. | ||
+ | # Remote Access to jiriaf2301/02 | ||
+ | ## Verify if remote access to jiriaf2301/02 is resolved. | ||
+ | ### xrdp installation. | ||
+ | ## Discuss the need for acceptable remote access performance on jiriaf2301/03 for prototyping purposes. | ||
# Running Remote JRM with Virtual Kubelet (vk-cmd) | # Running Remote JRM with Virtual Kubelet (vk-cmd) | ||
## Discuss the setup of running a remote JRM using the k8s API server on jiriaf2301. | ## Discuss the setup of running a remote JRM using the k8s API server on jiriaf2301. | ||
+ | ## Please be advised that the JRM job constitutes a resource allocation mechanism, and the execution of the actual workflow will take place on this allocated resource after the decision made by JMS. | ||
## Share any recent developments or challenges in this area. | ## Share any recent developments or challenges in this area. | ||
− | # Running JRM as a SLURM Job | + | # Running JRM as a SLURM/SWIF Job |
## Explore the process of running JRM as a SLURM job. | ## Explore the process of running JRM as a SLURM job. | ||
− | ## Discuss | + | ### Discuss the advantages and benefits of utilizing SWIF for running JRM. |
− | |||
− | |||
## Address any questions or concerns regarding this choice. | ## Address any questions or concerns regarding this choice. | ||
− | # JCS and k8s API Server | + | # JCS and k8s API Server |
− | ## | + | ## Discuss how the JCS and/or k8s API server will update the "available resources" table in the MariaDB. |
− | ## Discuss how | + | ## Discuss mechanisms interacting with the k8s API server. |
+ | ## Discuss how JRM job records are maintained in the database. | ||
# Running CLAS12 Data-Stream Processing Pipeline | # Running CLAS12 Data-Stream Processing Pipeline | ||
## Outline the steps to prepare a Docker image for the CLAS12 data-stream processing pipeline. | ## Outline the steps to prepare a Docker image for the CLAS12 data-stream processing pipeline. | ||
− | ## Discuss running the pipeline container as a | + | ## Discuss running the pipeline container as a k8s POD inside a JRM on a Perlmutter node at NERSC. |
− | |||
− | |||
− | |||
# Status of the Front-End | # Status of the Front-End | ||
## Provide an update on the current status of the front-end system. | ## Provide an update on the current status of the front-end system. | ||
Line 67: | Line 68: | ||
# Authentication of JIRIAF Users | # Authentication of JIRIAF Users | ||
## Explore the authentication mechanisms used by SWIF and OSG. | ## Explore the authentication mechanisms used by SWIF and OSG. | ||
− | ## Discuss the feasibility of adopting these mechanisms for JIRIAF. | + | ### Discuss the feasibility of adopting these mechanisms for JIRIAF. |
# Metadata Characterizing User Resources and Processing Requests | # Metadata Characterizing User Resources and Processing Requests | ||
## Discuss the metadata related to user resources and processing requests, including data source, core requirements, memory, disk space, walk time, preferred site, and other relevant factors. | ## Discuss the metadata related to user resources and processing requests, including data source, core requirements, memory, disk space, walk time, preferred site, and other relevant factors. | ||
# JIRIAF Workflow Request Matching Server (JMS) | # JIRIAF Workflow Request Matching Server (JMS) | ||
− | ## | + | ## Describe the role and mechanisms of JMS in running Kubernetes pods. |
## Share any updates or changes in this area. | ## Share any updates or changes in this area. | ||
# JMS Matching Algorithms | # JMS Matching Algorithms | ||
− | ## Discuss different matching algorithms | + | ## Discuss different matching algorithms JMS uses, such as exact matching and distributed resource allocation. |
## Explore the implications and benefits of these algorithms. | ## Explore the implications and benefits of these algorithms. | ||
# AOT | # AOT |
Revision as of 16:27, 28 September 2023
Connection Info:
You can connect using the following link (Meeting ID: 160 851 8798). (Click "Expand" to the right for details -->):
One tap mobile: US: +16692545252,,1608518798# or +16468287666,,1608518798#
Meeting URL: https://jlab-org.zoomgov.com/j/1608518798?pwd=NnU3cW1ZZFhTTUQ2Y0hIRU5JTWg0UT09&from=addon
Meeting ID: 160 851 8798
Passcode: 205601
Join by Telephone
For higher quality, dial a number based on your current location.
Dial:
US: +1 669 254 5252 or +1 646 828 7666 or +1 551 285 1373 or +1 669 216 1590 or 833 568 8864 (Toll Free)
Meeting ID: 160 851 8798
International numbers
Join by SIP
1608518798@sip.zoomgov.com
Join by H.323
161.199.138.10 (US West)
161.199.136.10 (US East)
Meeting ID: 160 851 8798
Passcode: 205601
Agenda:
- Announcements
- The initial manuscript outlining our project has been submitted to the EPJ.
- The JIRIAF LDRD project has been chosen to receive funding for its second year of operation.
- Remote Access to jiriaf2301/02
- Verify if remote access to jiriaf2301/02 is resolved.
- xrdp installation.
- Discuss the need for acceptable remote access performance on jiriaf2301/03 for prototyping purposes.
- Verify if remote access to jiriaf2301/02 is resolved.
- Running Remote JRM with Virtual Kubelet (vk-cmd)
- Discuss the setup of running a remote JRM using the k8s API server on jiriaf2301.
- Please be advised that the JRM job constitutes a resource allocation mechanism, and the execution of the actual workflow will take place on this allocated resource after the decision made by JMS.
- Share any recent developments or challenges in this area.
- Running JRM as a SLURM/SWIF Job
- Explore the process of running JRM as a SLURM job.
- Discuss the advantages and benefits of utilizing SWIF for running JRM.
- Address any questions or concerns regarding this choice.
- Explore the process of running JRM as a SLURM job.
- JCS and k8s API Server
- Discuss how the JCS and/or k8s API server will update the "available resources" table in the MariaDB.
- Discuss mechanisms interacting with the k8s API server.
- Discuss how JRM job records are maintained in the database.
- Running CLAS12 Data-Stream Processing Pipeline
- Outline the steps to prepare a Docker image for the CLAS12 data-stream processing pipeline.
- Discuss running the pipeline container as a k8s POD inside a JRM on a Perlmutter node at NERSC.
- Status of the Front-End
- Provide an update on the current status of the front-end system.
- JIRIAF Entry Web Page
- Discuss whether a web server running on jiriaf2301 is required for the JIRIAF entry web page.
- Connection to MariaDB Back-End
- Confirm if there is a connection to the MariaDB back-end for populating the "user-job-request" table.
- Status of the JIRIAF REST API
- Provide an update on the current status and functionality of the JIRIAF REST API.
- Authentication of JIRIAF Users
- Explore the authentication mechanisms used by SWIF and OSG.
- Discuss the feasibility of adopting these mechanisms for JIRIAF.
- Explore the authentication mechanisms used by SWIF and OSG.
- Metadata Characterizing User Resources and Processing Requests
- Discuss the metadata related to user resources and processing requests, including data source, core requirements, memory, disk space, walk time, preferred site, and other relevant factors.
- JIRIAF Workflow Request Matching Server (JMS)
- Describe the role and mechanisms of JMS in running Kubernetes pods.
- Share any updates or changes in this area.
- JMS Matching Algorithms
- Discuss different matching algorithms JMS uses, such as exact matching and distributed resource allocation.
- Explore the implications and benefits of these algorithms.
- AOT
Useful References