Difference between revisions of "JIRIAF Meeting Jan. 12 2023"

From epsciwiki
Jump to navigation Jump to search
(Created page with " === Connection Info: === <div class="toccolours mw-collapsible mw-collapsed"> You can connect using [https://jlab-org.zoomgov.com/j/11601588698 https://jlab-org.zoomgov.com/...")
 
 
(9 intermediate revisions by one other user not shown)
Line 3: Line 3:
 
=== Connection Info: ===
 
=== Connection Info: ===
 
<div class="toccolours mw-collapsible mw-collapsed">
 
<div class="toccolours mw-collapsible mw-collapsed">
You can connect using [https://jlab-org.zoomgov.com/j/11601588698 https://jlab-org.zoomgov.com/j/1601588698] (Meeting ID: 160 158 8698). <font size="-3">(Click "Expand" to the right for details -->):</font>
+
You can connect using [https://jlab-org.zoomgov.com/j/1608518798 https://jlab-org.zoomgov.com/j/1608518798] (Meeting ID: 160 851 8798). <font size="-3">(Click "Expand" to the right for details -->):</font>
  
 
<div class="mw-collapsible-content">
 
<div class="mw-collapsible-content">
  
 
<p>
 
<p>
One tap mobile: US: +16692545252,,1601588698# or +16468287666,,1601588698# <br>
+
One tap mobile: US: +16692545252,,1608518798#   or +16468287666,,1608518798# <br>
Meeting URL: https://jlab-org.zoomgov.com/j/1601588698?pwd=Q3FRR0xjd1lheU5vS3NHcTZBdkp6QT09&from=addon<br>
+
Meeting URL: https://jlab-org.zoomgov.com/j/1608518798?pwd=NnU3cW1ZZFhTTUQ2Y0hIRU5JTWg0UT09&from=addon<br>
Meeting ID: 160 158 8698<br>
+
Meeting ID: 160 851 8798<br>
Passcode: 508845<br>
+
Passcode: 205601<br>
 
</p>
 
</p>
 
<p>
 
<p>
Line 17: Line 17:
 
For higher quality, dial a number based on your current location.<br>
 
For higher quality, dial a number based on your current location.<br>
 
Dial: <br>
 
Dial: <br>
US: +1 669 254 5252 or ++1 646 828 7666 or ++1 669 216 1590 or ++1 415 449 4000 or 833 568 8864 (Toll Free)<br>
+
US: +1 669 254 5252 or +1 646 828 7666 or +1 551 285 1373  or +1 669 216 1590  or 833 568 8864 (Toll Free)<br>
Meeting ID: 160 158 8698<br>
+
Meeting ID: 160 851 8798<br>
 
</p>
 
</p>
 
<p>
 
<p>
 
International numbers<br>
 
International numbers<br>
Join from an H.323/SIP room system<br>
+
Join by SIP<br>
H.323: 161.199.138.10 (US West)<br>
+
1608518798@sip.zoomgov.com<br>
 +
Join by H.323<br>
 +
161.199.138.10 (US West)<br>
 
161.199.136.10 (US East)<br>
 
161.199.136.10 (US East)<br>
Meeting ID: 160 158 8698<br>
+
Meeting ID: 160 851 8798<br>
Passcode: 508845 <br>
+
Passcode: 205601<br>
SIP: 11601588698@sip.zoomgov.com<br>
 
Passcode: 923910 <br>
 
 
</p>
 
</p>
 
</div>
 
</div>
Line 38: Line 38:
 
# Announcements
 
# Announcements
  
# Streaming CODA
+
# [https://userweb.jlab.org/~gurjyan/presentations/jiriaf/jiriaf1.12.23.pdf Introduction]
## CODA PAAG and SAAG components
+
# User Request
## JCEdit-5.0: CODA COOL streaming front-end
+
## Super Facility API
###  New CODA types: PAAG and SAAG
+
# User workflow/application migration
## AFECS updates to support new CODA types
+
## Containers
## CODA connection mechanism to a data-processing pipeline
+
### Docker Hub
### Remote ET FIFO
+
## Distributed file systems
## CODA EPICS interface
+
### [https://cernvm.cern.ch/fs/ CernVM-FS]
### ActiveMQ
+
# Data provisioning
 
+
## [https://rucio.cern.ch RUCIO]
# Prototype GEM detector readout
+
## [https://blueskyproject.io/tiled/ TILED]
## instructions at [https://wiki.jlab.org/epsciwiki/index.php/SAMPA_SRO SAMPA-SRO]
+
## [https://userweb.jlab.org/~gurjyan/presentations/jiriaf/DAOS_update_JLab.pdf Intel DAOS]
## DSPDecoder engine (on hold)
+
# Orchestration
## Formatting output data in EVIO-6 (in progress)
+
## [https://airflow.apache.org Apache AirFlow]
## ML-based anomaly detection: Adaptive sliding window (ADWIN) algorithm-based actor development (completed, testing)
 
## Cluster finding in a stream of data
 
## New GEM detector installation
 
## Integration with the streaming CODA data acquisition system.
 
 
 
# EJFAT
 
## Port CLAS12 CLARA services into ERSAP (completed)
 
## EJFAT reassemble - ERSAP BE interface (completed)
 
## Successfully run CLAS12 reconstruction application on ejfat-1, streaming data from ejfat-5 through LB on ejfat-1
 
## Corrupted reassembled events
 
## Introducing data lake as a stream-source actor (testing)
 
## PID loop in the BE receiver actor (completed)
 
 
 
# TriDAS
 
 
 
# ERSAP planning
 
## Development efforts
 
### Monitoring
 
#### Support for InfluxDB (done)
 
#### Support for Prometheus (active)
 
##### Prometheus ERSAP Exporter (active)
 
### Tiered data storage
 
#### Redis as a source actor (on hold)
 
#### Data-lake as a source actor 
 
##### Developed EjfatReaderEngine that includes a large ring buffer. (Testing)
 
##### Develop EjfatReaderEngine that talks to ET (designing)
 
### Data stream transport
 
#### Implementing Kafka as internal transport (research)
 
#### Support for P2P (client-server) communication (on hold)
 
### Transient data format
 
#### Use EVIO-6 as default
 
#### Evaluate Apache Arrow as a transient data format (active)
 
#### EvioArrow, ArrowEvio convertor engines (designing)
 
### Application graphical designer (research)
 
#### Extend COOL to describe data-stream processing pipelines
 
##### Considering using Spring Boot for web interface and object persistency (research) 
 
#### Engine provisioning and deployment improvements (on hold)
 
### CLI improvements
 
### New language bindings (on hold)
 
### General-purpose data-stream processing engines (on hold)
 
### Benchmarking and testing (active)
 
## Commissioning and field studies
 
### EIC prototype detector components data stream readout and processing
 
#### Upcoming beam tests at the JLAB
 
 
# AOT
 
# AOT
  
 
==== Useful References ====
 
==== Useful References ====
 
<small>
 
<small>
* [[SRO Meetings]]
+
* [[JIRIAF Meetings]]
* [https://clasweb.jlab.org/wiki/index.php/Streaming_Read-out_Wiki_page CLAS12 SRO page]
+
* [https://teams.microsoft.com/_#/FileBrowserTabApp/JIRIAF?threadId=19:9a90d2e6643f40af9491c50faf8143d2@thread.skype&ctx=channel JIRIAF Channel]
* [https://jeffersonlab.sharepoint.com/:w:/s/SciComp/EQxPpI6h8VRFtlSKLHeLWusBYCOQUySVpEBKCIlR09u5wA?email=davidl%40jlab.org&e=NCC22f ERSAP Document]
+
</small>
* [https://jeffersonlab.sharepoint.com/:w:/r/sites/SciComp/Shared%20Documents/EPSCI/SRO/Streaming/Development/WMS.docx?d=w286e2f0b5d174426a0ddd06a0766e57d&csf=1&e=8wGGZR Vardan's WMS draft]
 
* [https://jeffersonlab.sharepoint.com/:b:/r/sites/SciComp/Shared%20Documents/EPSCI/SRO/Streaming/20191217%20Streaming%20Paper/20200302%20JLab%20Streaming%20Readout.pdf?csf=1&e=kDX7yr Graham's Paper describing SRO]
 
* [[Media:20200609_data_format.pdf |Dave Abbott's slides on data format]]
 
</small>
 
  
 
<hr>
 
<hr>
  
 
=== Minutes: ===
 
=== Minutes: ===

Latest revision as of 15:02, 19 January 2023


Connection Info:

You can connect using https://jlab-org.zoomgov.com/j/1608518798 (Meeting ID: 160 851 8798). (Click "Expand" to the right for details -->):

One tap mobile: US: +16692545252,,1608518798# or +16468287666,,1608518798#
Meeting URL: https://jlab-org.zoomgov.com/j/1608518798?pwd=NnU3cW1ZZFhTTUQ2Y0hIRU5JTWg0UT09&from=addon
Meeting ID: 160 851 8798
Passcode: 205601

Join by Telephone
For higher quality, dial a number based on your current location.
Dial:
US: +1 669 254 5252 or +1 646 828 7666 or +1 551 285 1373 or +1 669 216 1590 or 833 568 8864 (Toll Free)
Meeting ID: 160 851 8798

International numbers
Join by SIP
1608518798@sip.zoomgov.com
Join by H.323
161.199.138.10 (US West)
161.199.136.10 (US East)
Meeting ID: 160 851 8798
Passcode: 205601


Agenda:

  1. Announcements
  1. Introduction
  2. User Request
    1. Super Facility API
  3. User workflow/application migration
    1. Containers
      1. Docker Hub
    2. Distributed file systems
      1. CernVM-FS
  4. Data provisioning
    1. RUCIO
    2. TILED
    3. Intel DAOS
  5. Orchestration
    1. Apache AirFlow
  6. AOT

Useful References



Minutes: