JIRIAF Meeting Jan. 25 2024
Jump to navigation
Jump to search
Connection Info:
Expand
You can connect using the following link (Meeting ID: 160 126 6529). (Click "Expand" to the right for details -->):
Agenda:
- Announcements
- New site for deployments: ORNL.
- Tomorrow (01.26.24) starts biweekly meetings with ORNL (EJFAT collaboration)
- "Direct" access to Perlmutter cluster login nodes from jiriaf2301-02
- Access granted to the ESnet Perlmutter server from jiriaf2301-02.
- curl -s -G --data-urlencode 'match[]={__name__=~".+"}' https://prometheus-ejfat.es.net/federate
- NERSC allocation
- 250-hour allocation on Perlmutter: m4636 project (JIRIAF)
- 300-hour allocation on Perlmutter: m3792 project (EJFAT-EsNet).
- Abstract submitted to ACAT
- Invitation from Derek to give a talk on JIRIAF at the NERSC Data Day
- Summary of the project's undertakings
- JFE
- Forms to submit user workflow requests
- Login and authentication
- Workflow description: Processing type (batch, streaming, opportunistic-streaming, etc.), Docker image location, Resource requirements (core type, core count, memory, disk, time, data provisioning details).
- Research if k8s provides facilities for this (e.g., k8s dashboard)
- Forms to submit user workflow requests
- Visualize Jiriaf database tables.
- Dynamic updates.
- JCS and JMS
- Starting VKs (Jiriaf nodes) through the k8s API management system
- Jiriaf node naming convention and labeling
- JCS and Jiriaf database relationship. Tables, such as
- available resource, user requests, and user workflow status.
- Examine the site resources database table (constantly updated by SWIF2) and submit SWIF2 requests to launch nodes and allocate/lease resources.
- JIRIAF k8s cluster autoscaling (with possible AI support)
- Estimate wait time in the queue before and during the processing (Queueing theory)
- Defining workflows/pods in the cluster that are unschedulable
- Communicate with the k8s App server, ensuring submitted jobs are running, updating JIRIAF's available resource DB table.
- Develop a resource-request matching algorithm that compares user requests with the available resources.
- Define and suggest metadata structure for requests for accurate matching.
- JRM
- Implement a function using ConfigMap configuration to write files in pods.
- Define mechanisms to act on user workflows, such as reducing previously allocated resources to the user workflow/application.
- VK hardware monitor server
- node/vk launch script.
- Repository for all such scripts and k8s YAML configuration files
- JFE
- AOT
Useful References