JIRIAF Meeting Jan. 25 2024

From epsciwiki
Revision as of 15:49, 25 January 2024 by Gurjyan (talk | contribs) (→‎Agenda:)
Jump to navigation Jump to search


Connection Info:

You can connect using the following link (Meeting ID: 160 126 6529). (Click "Expand" to the right for details -->):


Agenda:

  • Announcements
  • New site for deployments: ORNL.
    • Tomorrow (01.26.24) starts biweekly meetings with ORNL (EJFAT collaboration)
  • "Direct" access to Perlmutter cluster login nodes from jiriaf2301-02
  • Access granted to the ESnet Perlmutter server from jiriaf2301-02.
  • NERSC allocation
    • 250-hour allocation on Perlmutter: m4636 project (JIRIAF)
    • 300-hour allocation on Perlmutter: m3792 project (EJFAT-EsNet).
  • Abstract submitted to ACAT
  • Invitation from Derek to give a talk on JIRIAF at the NERSC Data Day
  • Summary of the project's undertakings
    • JFE
      • Forms to submit user workflow requests
        • Login and authentication
        • Workflow description: Processing type (batch, streaming, opportunistic-streaming, etc.), Docker image location, Resource requirements (core type, core count, memory, disk, time, data provisioning details).
          • Research if k8s provides facilities for this (e.g., k8s dashboard)
    • Visualize Jiriaf database tables.
      • Dynamic updates.
    • JCS and JMS
      • Starting VKs (Jiriaf nodes) through the k8s API management system
      • Jiriaf node naming convention and labeling
      • JCS and Jiriaf database relationship. Tables, such as
        • available resource, user requests, and user workflow status.
        • Examine the site resources database table (constantly updated by SWIF2) and submit SWIF2 requests to launch nodes and allocate/lease resources.
        • JIRIAF k8s cluster autoscaling (with possible AI support)
          • Estimate wait time in the queue before and during the processing (Queueing theory)
      • Defining workflows/pods in the cluster that are unschedulable
      • Communicate with the k8s App server, ensuring submitted jobs are running, updating JIRIAF's available resource DB table.
      • Develop a resource-request matching algorithm that compares user requests with the available resources.
      • Define and suggest metadata structure for requests for accurate matching.
    • JRM
      • Implement a function using ConfigMap configuration to write files in pods.
      • Define mechanisms to act on user workflows, such as reducing previously allocated resources to the user workflow/application.
      • VK hardware monitor server
      • node/vk launch script.
      • Repository for all such scripts and k8s YAML configuration files
  • AOT

Useful References



Minutes: