JIRIAF Meeting Jan. 11 2024
Jump to navigation
Jump to search
Connection Info:
Expand
You can connect using the following link (Meeting ID: 160 126 6529). (Click "Expand" to the right for details -->):
Agenda:
- Announcements
- Welcome Patrick onboard.
- Problem getting an NSLS-II data-intensive workflow for migration.
- New sites for deployments: ORNL and ANL.
- Ticket (INC0114103) requesting Jiriaf nodes to access DOE computing facilities like NERSC and ORNL.
- The list of IP addresses and ports to present to the security and networking team.
- Ticket (INC0114103) requesting Jiriaf nodes to access DOE computing facilities like NERSC and ORNL.
- NERSC allocation
- Jiriaf's project request has been approved (as of 01.02).
- Nick Taylor assigned more time for the m3792 project (EJFAT-EsNet).
- Summary of the project's undertakings and key achievements
- M3
- Define mechanisms to act on user workflows, such as reducing previously allocated resources to the user workflow/application.
- M4
- JCS design and development
- Starting VKs (Jiriaf nodes) through the k8s API management system
- Jiriaf node naming convention and labeling
- Jiriaf k8s cluster autoscaling (with possible AI support)
- Defining workflows/pods in the cluster that are unschedulable
- JCS and Jiriaf database relationship. Tables, such as
- available resource, user requests, and user workflow status.
- Examine the site resources database table (constantly updated by SWIF2) and submit SWIF2 requests to launch nodes and allocate/lease resources.
- Communicate with the k8s App server, ensuring submitted jobs are running, updating JIRIAF's available resource DB table.
- Develop a resource-request matching algorithm that compares user requests with the available resources.
- Define and suggest metadata structure for requests for accurate matching.
- Starting VKs (Jiriaf nodes) through the k8s API management system
- JCS design and development
- M3
- M5
- JIRIAF k8s node/vk_cmd: Implement a function that can use ConfigMap configuration to write files in pods.
- Anatomy of the node/vk launch script.
- VK hardware monitor server
- JIRIAF k8s node/vk_cmd: Implement a function that can use ConfigMap configuration to write files in pods.
- Future milestones
- Accepting opportunistic workflow primarily designed for streaming purposes.
- Mathematical model for simulating the abstract processor/actor within the JIRIAF ecosystem.
- Definition of the parameters and functionalities of the distributed workflow agent model and initiation of its design.
- M5
- Slides for upcoming presentations in preparation for the publications
- Slide describing JIRIAF virtual k8s cluster creation, emphasizing its dynamic nature.
- Slide showing Prometheus integration to monitor JIRIAF k8s cluster and pods.
- Start working on a paper describing JIRIAF resource acquisition and workflow deployment within a dynamic k8s cluster.
- AOT
Useful References