Difference between revisions of "ERSAP"

From epsciwiki
Jump to navigation Jump to search
 
(37 intermediate revisions by 2 users not shown)
Line 2: Line 2:
  
 
<br><hr><br>
 
<br><hr><br>
−
<div class="orbitron"><font size="+1">System Overview:</font></div>''ERSAP is a reactive actor/micro-service based data-stream processing framework,  a hardware/software system that is capable of processing unbounded streams of continuous data at scale over distributed heterogeneous resources in real-time. The goal of ERSAP is to design a framework that will provide a flexible environment to design and, most importantly, allow future evolution of the data processing applications. An environment that will encourage implementation of new ideas and new technologies while preserving integrity of existing data processing pipelines. ERSAP encourages to perform functional decomposition of the overall data processing application into small, mono-functional artifacts/actors. Artifacts that are easy to understand, develop, deploy and debug. The fact that these artifacts/actors are programmatically independent, they can be scaled and optimized independently, which is impossible to do for components of the monolithic application. One of the important advantages is the ERSAP fault tolerance, where independent actors can come and go on the stream without forcing entire application to crash, and it is easy to locate the faulty actor in the chain. Due the fact that the actors are loosely coupled, and that the message (the only coupling between actors) carries the context, they can run on heterogenous environments with different memory and/or processing power requirements. The active actors can be physically separated from each other, so the deployment of a new set of actors does not put the whole system on hold, and the updates are less intrusive. The development of the actors and actor engines handling specific areas of the system can be performed by independent development teams, with their own release cycles and with it's own pace. The ERSAP is not a ground up design. It is based on our experience of micro-service based, off-line data processing applications that are running more than a decade in CLAS12, and CODA distributed, triggered based DAQ that is running more than quarter of a century at the JLAB. Also, it will be utilizing JANA framework to develop high performance data processing engines.  
+
<div class="orbitron"><font size="+1">System Overview:</font></div>'' ERSAP is a reactive actor/micro-service based data-stream processing framework,  a hardware/software system that is capable of processing unbounded streams of continuous data at scale over distributed heterogeneous resources in real-time. The goal of ERSAP is to design a framework that will provide a flexible environment to design and, most importantly, allow the future evolution of data processing applications. An environment that will encourage the implementation of new ideas and new technologies while preserving the integrity of existing data processing pipelines. ERSAP encourages to perform functional decomposition of the overall data processing application into small, mono-functional artifacts/actors. Artifacts that are easy to understand, develop, deploy, and debug. Fact that these artifacts/actors are programmatically independent, they can be scaled and optimized independently, which is impossible to do for components of the monolithic application. One of the important advantages is the ERSAP fault tolerance, where independent actors can come and go on the stream without forcing the entire application to crash, and it is easy to locate the faulty actor in the chain. Due to the fact that the actors are loosely coupled and that the message (the only coupling between actors) carries the context, they can run on heterogenous environments with different memory and/or processing power requirements. The active actors can be physically separated from each other, so the deployment of a new set of actors does not put the whole system on hold, and the updates are less intrusive. The development of the actors and actor engines handling specific areas of the system can be performed by independent development teams, with their own release cycles, and at their own pace. The ERSAP is not a ground-up design. It is based on our experience of micro-service-based, off-line data processing applications that are running for more than a decade in CLAS12, and CODA distributed, triggered-based DAQ that is running for more than a quarter of a century at the JLAB. Also, it will be utilizing the JANA framework to develop high-performance data processing engines.  
 +
<br><hr><br>
 +
<div class="orbitron"><font size="+0">Genealogy:</font></div>
 +
[https://userweb.jlab.org/~gurjyan/presentations/ersap/ERSAP_genealogy.pdf History and computing mode]
 +
<br><hr><br>
 +
<div class="orbitron"><font size="+0">Roadmap:</font></div>
 +
[https://userweb.jlab.org/~gurjyan/presentations/ERSAPv2.0.pdf ERSAP v2.0]
 +
<br><hr><br>
 
<table align="left" cellpadding="3" width="100%">
 
<table align="left" cellpadding="3" width="100%">
 
   <tr bgcolor="#d0e0d0">
 
   <tr bgcolor="#d0e0d0">
Line 17: Line 24:
 
       <li>[https://github.com/JeffersonLab/ersap-vtp.git ERSAP CODA engines]
 
       <li>[https://github.com/JeffersonLab/ersap-vtp.git ERSAP CODA engines]
 
       <li>[https://github.com/JeffersonLab/ersap-jana.git ERSAP JANA engines]
 
       <li>[https://github.com/JeffersonLab/ersap-jana.git ERSAP JANA engines]
 +
      <li>[https://github.com/JeffersonLab/ersap-sampa.git ERSAP SAMPA SRO engines]
 +
      <li>[https://github.com/JeffersonLab/ersap-actor ERSAP actors]
 
     <li>ERSAP DataLake
 
     <li>ERSAP DataLake
−
       <li>[[SAMPA SRO]] instructions
+
       <li>[[SAMPA SRO]] INDRA-ASTRA project instructions
 
       </ul>
 
       </ul>
 
     </td>
 
     </td>
Line 25: Line 34:
 
     <td>
 
     <td>
 
       <ul>
 
       <ul>
 +
      <li>[[SRO Meetings]]
 
       <li>[[TriDAS-ERSAP Collaboration Meetings]]
 
       <li>[[TriDAS-ERSAP Collaboration Meetings]]
 
       </ul>
 
       </ul>
Line 31: Line 41:
 
</table>
 
</table>
 
<br><hr><br>
 
<br><hr><br>
−
<div class="orbitron"><font size="+1">Presentations/Papers</font></div>
+
<div class="orbitron"><font size="+1">Presentations/Papers:</font></div>
  
 
{| class="wikitable"
 
{| class="wikitable"
 
|-
 
|-
−
!date
+
!Date
−
!presenter
+
!Presenter
 
!Event
 
!Event
−
!links
+
!Link
 +
|-
 +
|2023-11-28
 +
|Vardan Gyurjyan
 +
| [https://indico.bnl.gov/event/20010/ SRO-X1 Workshop]
 +
| [https://indico.bnl.gov/event/20010/timetable/#20231128.detailed pdf]
 +
|-
 +
|2023-06-17
 +
|Vardan Gyurjyan
 +
| ERSAP: Towards Better NP Data-Stream Analytics with Flow-Based Programming. IEEE Transactions on Nuclear Science, June 2023, Volume: 70, Issue: 6, Pages: 966-970, Print ISSN: 0018-9499, Online ISSN: 1558-1578 
 +
| Digital Object Identifier:[https://ieeexplore.ieee.org/search/searchresult.jsp?newsearch=true&queryText=10.1109%2FTNS.2023.3242548  10.1109/TNS.2023.3242548]
 +
|-
 +
|2023-04-08
 +
|Vardan Gyurjyan
 +
| [https://indico.jlab.org/event/459/contributions/11397/ 26TH International Conference on Computing in High Energy & Nuclear Phusics]
 +
| [https://indico.jlab.org/event/459/contributions/11397/ pptx]
 +
|-
 +
 
 
|-
 
|-
−
|2021-12-01
+
|2022-08-04
−
|T. Jeske
+
|Vardan Gyurjyan
−
|[https://indico.cern.ch/event/855454/ ACAT2021]
+
| [https://indico.cern.ch/event/1109460/ 23rd IEEE Real Time Conference]
−
|([https://indico.cern.ch/event/855454/contributions/4597451/attachments/2357033/4023012/ACAT_2021_v1.pdf PDF] [https://indico.cern.ch/event/855454/contributions/4597451/attachments/2357033/4023011/ACAT_2021_v1.pptx PPTX])
+
| [https://indico.cern.ch/event/1109460/contributions/4893265/attachments/2489038/4274259/ersap_rt2022.pdf pdf]
 
|-
 
|-
−
|2021-11-30
 
−
|D. Lawrence
 
−
|DOE Nuclear Physics PI Exchange Meeting
 
−
|([https://docs.google.com/presentation/d/15BbsDtUm5jJQHJQuun3GtV1Qo272vGzIGEA0ihrGOec/edit?usp=sharing slides])
 
 
|-
 
|-
−
|2021-11-17
+
|2021-12-08
−
|N. Jarvis
+
|Vardan Gyurjyan
−
|[https://halldweb.jlab.org/wiki-private/index.php/June_30,_2021,_Analysis_Working_Group GlueX Production & Analysis Meeting]
+
| [https://indico.phy.ornl.gov/event/112/ EIC Workshop IX on Streaming Readout]
−
|([https://halldweb.jlab.org/doc-private/DocDB/ShowDocument?docid=5371 slides])
+
| [https://indico.phy.ornl.gov/event/112/contributions/476/attachments/493/1343/sro-IX.pptx pptx]
 
|-
 
|-
−
|2021-11-05
+
|2021-04-30
−
|T. Britton
+
|Vardan Gyurjyan
−
|Canisius College Seminar
+
| [https://indico.mit.edu/event/1/ EIC Workshop VIII on Streaming Readout]
−
|([https://docs.google.com/presentation/d/1Vb26-fTx5mxDYqcG_BOXRi4w9kdJnRFTr0C5BwM2vDU/edit#slide=id.gfd51fecf9e_0_105 slides], [[Media:2021.11.05.AIEC_Canisius.pdf|PDF]])
+
| [https://indico.mit.edu/event/1/contributions/80/attachments/37/61/gyurjyan_sro_VIII.pptx pptx]
 
|-
 
|-
−
|2021-10-07
+
|2020-11-8
−
|N. Jarvis
+
|Vardan Gyurjyan
−
|IOP NPG Colloquium
+
| [https://indico.bnl.gov/event/9458/ EIC SRO Workshop VII]
−
|([https://halldweb.jlab.org/doc-public/DocDB/ShowDocument?docid=5324 slides])
+
| [https://indico.bnl.gov/event/9458/contributions/43291/attachments/31389/49551/sro_workshop_VII.pptx pptx]
 
|-
 
|-
−
|2021-09-09
+
|2020-10-12
−
|T. Jeske
+
|Vardan Gyurjyan
−
|[https://indico.bnl.gov/event/10699/ AI4EIC Workshop]
+
| [https://indico.cern.ch/event/737461/ 22nd IEEE Real Time Conference]
−
|([https://indico.bnl.gov/event/10699/contributions/53860/attachments/37034/60994/AI4EIC_Jeske.pdf slides])
+
| [https://indico.cern.ch/event/737461/contributions/3725113/attachments/2105289/3568916/RT2020_gyurjyan_33.pdf poster]
 
|-
 
|-
−
|2021-07-26
+
|2020-03-12
−
|T. Jeske
+
|Vardan Gyurjyan
−
|[https://indico.jlab.org/event/462/ JLab A.I. Town Hall]
+
| ERSAP Design Requirements
−
|([https://indico.jlab.org/event/462/contributions/8849/attachments/7306/10085/AI_TownHall2021-5.pdf slides])
+
| [https://jeffersonlab.sharepoint.com/:w:/s/SciComp/EQxPpI6h8VRFtlSKLHeLWusBYCOQUySVpEBKCIlR09u5wA?email=davidl%40jlab.org&e=NCC22f docx]
 
|-
 
|-
−
|2021-06-30
+
|2020-01-01
−
|T. Jeske
+
|Graham Heyes
−
|[https://halldweb.jlab.org/wiki-private/index.php/June_30,_2021,_Analysis_Working_Group GlueX Production & Analysis Meeting]
+
| JLAB SRO paper
−
|([https://jeffersonlab.sharepoint.com/:b:/s/DataScience/EWbGo2Xl_K1CgxEh0TvKeLABIrdrl5YCIFCrzB9hcJSbuQ?e=iXivoJ slides])
+
|[https://jeffersonlab.sharepoint.com/:b:/r/sites/SciComp/Shared%20Documents/EPSCI/SRO/Streaming/20191217%20Streaming%20Paper/20200302%20JLab%20Streaming%20Readout.pdf?csf=1&e=kDX7yr pdf]
 
|}
 
|}

Latest revision as of 14:35, 4 December 2023

Welcome to the ERSAP Wiki

(Environment for Real-time Streaming, Acquisition and Processing)


System Overview:

ERSAP is a reactive actor/micro-service based data-stream processing framework, a hardware/software system that is capable of processing unbounded streams of continuous data at scale over distributed heterogeneous resources in real-time. The goal of ERSAP is to design a framework that will provide a flexible environment to design and, most importantly, allow the future evolution of data processing applications. An environment that will encourage the implementation of new ideas and new technologies while preserving the integrity of existing data processing pipelines. ERSAP encourages to perform functional decomposition of the overall data processing application into small, mono-functional artifacts/actors. Artifacts that are easy to understand, develop, deploy, and debug. Fact that these artifacts/actors are programmatically independent, they can be scaled and optimized independently, which is impossible to do for components of the monolithic application. One of the important advantages is the ERSAP fault tolerance, where independent actors can come and go on the stream without forcing the entire application to crash, and it is easy to locate the faulty actor in the chain. Due to the fact that the actors are loosely coupled and that the message (the only coupling between actors) carries the context, they can run on heterogenous environments with different memory and/or processing power requirements. The active actors can be physically separated from each other, so the deployment of a new set of actors does not put the whole system on hold, and the updates are less intrusive. The development of the actors and actor engines handling specific areas of the system can be performed by independent development teams, with their own release cycles, and at their own pace. The ERSAP is not a ground-up design. It is based on our experience of micro-service-based, off-line data processing applications that are running for more than a decade in CLAS12, and CODA distributed, triggered-based DAQ that is running for more than a quarter of a century at the JLAB. Also, it will be utilizing the JANA framework to develop high-performance data processing engines.


Genealogy:

History and computing mode



Roadmap:

ERSAP v2.0



Projects Meetings



Presentations/Papers:
Date Presenter Event Link
2023-11-28 Vardan Gyurjyan SRO-X1 Workshop pdf
2023-06-17 Vardan Gyurjyan ERSAP: Towards Better NP Data-Stream Analytics with Flow-Based Programming. IEEE Transactions on Nuclear Science, June 2023, Volume: 70, Issue: 6, Pages: 966-970, Print ISSN: 0018-9499, Online ISSN: 1558-1578 Digital Object Identifier:10.1109/TNS.2023.3242548
2023-04-08 Vardan Gyurjyan 26TH International Conference on Computing in High Energy & Nuclear Phusics pptx
2022-08-04 Vardan Gyurjyan 23rd IEEE Real Time Conference pdf
2021-12-08 Vardan Gyurjyan EIC Workshop IX on Streaming Readout pptx
2021-04-30 Vardan Gyurjyan EIC Workshop VIII on Streaming Readout pptx
2020-11-8 Vardan Gyurjyan EIC SRO Workshop VII pptx
2020-10-12 Vardan Gyurjyan 22nd IEEE Real Time Conference poster
2020-03-12 Vardan Gyurjyan ERSAP Design Requirements docx
2020-01-01 Graham Heyes JLAB SRO paper pdf
Retrieved from "https://wiki.jlab.org/epsciwiki/index.php?title=ERSAP&oldid=4215"