Alert.png The wiki is deprecated and due to be decommissioned by the end of September 2022.
The content is being migrated to other supports, new updates will be ignored and lost.
If needed you can get in touch with EGI SDIS team using operations @ egi.eu.

Difference between revisions of "Applications on Demand Service - architecture"

From EGIWiki
Jump to navigation Jump to search
 
(46 intermediate revisions by 2 users not shown)
Line 1: Line 1:
{{DeprecatedAndMovedTo|new_location=https://docs.egi.eu/users/applications-on-demand/}}
{{Template:LTOS_menubar}} {{TOC_right}}  
{{Template:LTOS_menubar}} {{TOC_right}}  


Line 17: Line 19:
<!--* The <b>User Registration Portal (URP)</b> is a web portal which is used to authenticate users interested to access the Infrastructure. For the authentication, the portal relies on the EGI AAI CheckIn service which provides AAI services for both users and service providers. The EGI AAI CheckIn service supports different IdPs, eduGain, the international inter-federation  service, and social credentials (e.g. Facebook, Google). During the authentication process, users can provide information about their contact, institutions and research topic. Such information will be taken into account by the operator(s) to evaluate whether the user is entitled to access and use the resources and the applications available in the Platform. This User Registration Portal (URP) is accessible at http://access.egi.eu-->
<!--* The <b>User Registration Portal (URP)</b> is a web portal which is used to authenticate users interested to access the Infrastructure. For the authentication, the portal relies on the EGI AAI CheckIn service which provides AAI services for both users and service providers. The EGI AAI CheckIn service supports different IdPs, eduGain, the international inter-federation  service, and social credentials (e.g. Facebook, Google). During the authentication process, users can provide information about their contact, institutions and research topic. Such information will be taken into account by the operator(s) to evaluate whether the user is entitled to access and use the resources and the applications available in the Platform. This User Registration Portal (URP) is accessible at http://access.egi.eu-->


* A <b>catch-all</b> VO called ‘vo.access.egi.eu’ and a <b>pre-allocated pool of HTC and cloud resources</b> configured for supporting the EGI Applications on Demand research activities. This resource pool currently includes cloud resources from Italy (INFN-Catania and INFN-Bari) and Spain (CESGA) and HTC clusters from Belgium (VUB), Italy (INFN-Catania and INFN-Bari), Poland (CYFRONET) and Spain (CESGA).
* A <b>catch-all</b> VO called ‘vo.access.egi.eu’ and a <b>pre-allocated pool of HTC and cloud resources</b> configured for supporting the EGI Applications on Demand research activities. This resource pool currently includes cloud resources from Italy (INFN-Catania), Spain (CESGA) and France (IN2P3-IRES).


* The <b>X.509 credentials factory service</b> (also called eToken server)[1]  is a standard-based solution developed for central management of robot certificates and the provision of Per-User Sub-Proxy (PUSP) certificates. PUSP allows to identify the individual user that operates using a common robot certificate.  Permissions to request robot proxy certificates, and contact the server, are granted only to Science Gateways/portal integrated in the Platform and only when the user is authorized.
* The <b>X.509 credentials factory service</b> (also called eToken server)[1]  is a standard-based solution developed for central management of robot certificates and the provision of Per-User Sub-Proxy (PUSP) certificates. PUSP allows to identify the individual user that operates using a common robot certificate.  Permissions to request robot proxy certificates, and contact the server, are granted only to Science Gateways/portal integrated in the Platform and only when the user is authorized.


* A set of <b>Science Gateways/portal</b> to host applications that users can run accessing the EGI Applications on Demand service. Currently the following three frameworks are used: The Catania Science Gateway (CSG), the WS-PGRADE/gUSE  portal and the Elastic Cloud Computing Cluster (EC3).
* A set of <b>Science Gateways/portal</b> to host applications that users can run accessing the EGI Applications on Demand service. Currently the following three frameworks are used: The FutureGateway Science Gateway (FGSG), the WS-PGRADE/gUSE  portal and the Elastic Cloud Computing Cluster (EC3).


* The <b>EGI VMOps dashboard</b> provides a Graphical User Interface (GUI) to perform Virtual Machine (VM) management operations on the EGI Federated Cloud. The dashboard introduces a user-friendly environment where users can create and manage VMs along with associated storage devices on any of the EGI Federated Cloud providers. It provides a complete view of the deployed applications and a unified resource management experience, independently of the technology driving each of the resource centres of the federation. Users can create new infrastructure topologies, which include a set of VMs, their associated storage and contextualization, a wizard-like builder that guides them through the selection of the virtual appliances, virtual organisation, resource provider, and the final customisation of the VMs that will be deployed. Its tight integration with the AppDB Cloud Marketplace allows for an automatic discovery of the appliances which are supported at each resource provider Once a topology has been created, VMOps allows management actions to be applied both on the set of VMs comprising a topology and on fine-grained actions on each individual VM.
* The <b>EGI VMOps dashboard</b> provides a Graphical User Interface (GUI) to perform Virtual Machine (VM) management operations on the EGI Federated Cloud. The dashboard introduces a user-friendly environment where users can create and manage VMs along with associated storage devices on any of the EGI Federated Cloud providers. It provides a complete view of the deployed applications and a unified resource management experience, independently of the technology driving each of the resource centres of the federation. Users can create new infrastructure topologies, which include a set of VMs, their associated storage and contextualization, a wizard-like builder that guides them through the selection of the virtual appliances, virtual organisation, resource provider, and the final customisation of the VMs that will be deployed. Its tight integration with the AppDB Cloud Marketplace allows for an automatic discovery of the appliances which are supported at each resource provider Once a topology has been created, VMOps allows management actions to be applied both on the set of VMs comprising a topology and on fine-grained actions on each individual VM.
* The <b>EGI Notebooks</b>, build on top of JupyterHub offers an open-source web application where users can create and share documents that contain live code, equations, visualization and explanatory text.




Line 33: Line 37:
Current available resources grouped by categories:
Current available resources grouped by categories:
* <u>Cloud Resources</u>:
* <u>Cloud Resources</u>:
** 132 vCPU cores
** 246 vCPU cores
** 576GB of RAM
** 781 GB of RAM
** 3TB of object storage
** 2.1TB of object storage


* <u>High-Throughtput Resources</u>:
* <u>High-Throughtput Resources</u>:
** ~13Million HEPSPEC
** 9.5Million HEPSPEC
** 1.4 TB of disk storage
** 1.4 TB of disk storage


For more details about the resources allocated for supporting this service, please click [[here|here]].
For more details about the resources allocated for supporting this service, please click [[here|here]].
<br/>


== Useful Links  ==
== Operational Level Agreements ==
* EC3: https://documents.egi.eu/document/3370
* FGSG: https://documents.egi.eu/document/2782
* WS-PGRADE/gUSE: https://documents.egi.eu/document/3368
* Cloud resource providers: https://documents.egi.eu/document/2773
<br/>


=== Catch-all VO ===
== GGUS Support Units (SUs) ==
* VO ID card: http://operations-portal.egi.eu/vo/view/voname/vo.access.egi.eu
* FutureGateway Science Gateway (FGSG):  
* Name: <code>vo.access.egi.eu</code>
** The FGSG GGUS SU is available under the Core Services category
* Scope: Global
* Elastic Cloud Computing Cluster (EC3):  
* Disciplines: Support Activities
** The EC3 GGUS SU is available under the Core Services category
* VOMS servers: voms1/voms2.grid.cesnet.cz
* WS-PGRADE/gUSE portal:  
* VO Membership Management: https://perun.metacentrum.cz/perun-registrar-cert/?vo=vo.access.egi.eu
** The WS-PGRADE/gUSE GGUS SU is available under the Core Services category
* Contacts:
* The EGI Notebooks:  
** EGI Support Team: applications-platform-support@mailman.egi.eu
** The EGI Notebooks GGUS SU is available under the EGI Notebooks category
** Managers:
* The EGI VMOps dashboard:
*** Giuseppe La Rocca (<giuseppe.larocca@egi.eu>)
** The AppDB GGUS SU is available under the Core Services category
*** Diego Scardaci (<diego.scardaci@egi.eu>)
*** Gergely Sipos (<gergely.sipos@egi.eu>)
<br/>
<br/>


Line 70: Line 77:
== Monitoring ==
== Monitoring ==


* The EGI Applications on Demand service components have been registered in the [http://gocdb.egi.eu/ GOCDB] and connected with the EGI monitoring system based on [http://argo.egi.eu ARGO]. By default ARGO automatically gathered the services endpoints from the GOCDB  and implements simple https checks using standard NAGIOS  probes. If necessary new ones can be easily developed and added.
* The EGI Applications on Demand service components have been registered in the [http://gocdb.egi.eu/ GOCDB] and connected with the EGI monitoring system based on [http://argo.egi.eu ARGO].  
* To monitor the status of the service components (e.g. GRIDOPS-CSFG, GRIDOPS-LTOS, GRIDOPS-WS-PGRADE and GRIDOPS-EC3), check the following [http://argo.egi.eu/lavoisier/opsmon_reports?accept=html ARGO] report page.
* By default ARGO automatically gathered the services endpoints from the GOCDB  and implements simple 'https' checks using standard NAGIOS  probes. If necessary new ones can be easily developed and added.
* The following service components are monitored by ARGO:
** EGI-FGSG,  
** EGI-NOTEBOOKS,  
** GRIDOPS-WS-PGRADE,
** GRIDOPS-APPDB, and  
** GRIDOPS-EC3
* To monitor the service components, check the following [http://argo.egi.eu/egi/OPS-MONITOR-Critical ARGO] report page.
<br/>
<br/>


== Accounting ==
== Accounting ==
* Accounting data about the VO users can be checked here: https://accounting.egi.eu/  
* Accounting data about the VO users can be checked here: https://accounting.egi.eu/  
* From the EGI Accounting Portal it is possible to check the accounting metrics generated for both grid- and cloud-based resources supporting the vo.access.egi.eu VO. From the top-menu click on 'Restrict View' and 'VO Admin' to check the accounting data of platform users.
* From the EGI Accounting Portal it is possible to check the accounting metrics generated for both grid- and cloud-based resources supporting the vo.access.egi.eu VO.  
* From the top-menu click on <u>'Restrict View'</u> and <u>'VO Admin'</u> to check the accounting data of platform users.
 
 
[[Image:EGI_AoDs_Accounting.png|center|800px|EGI_AoDs_Accounting.png]]
 
== Policies & documents ==
*[https://documents.egi.eu/public/ShowDocument?docid=2635 Acceptable Use Policy (AUP) and Conditions of Use of the 'EGI Applications on Demand  (AoD) service']<br/>
*[https://documents.egi.eu/public/ShowDocument?docid=2734 EGI Applications on Demand (AoD) service Security Policy]<br/>
*[https://documents.egi.eu/public/ShowDocument?docid=3127 Doc vetting manual for the EGI Support team]
<br/>
<br/>


== Policies ==
== Useful Links  ==
*[https://documents.egi.eu/public/ShowDocument?docid=2635 Acceptable Use Policy (AUP) and Conditions of Use of the 'EGI Applications on Demand  (AoD) service']<br>
* VO ID card: http://operations-portal.egi.eu/vo/view/voname/vo.access.egi.eu
*[https://documents.egi.eu/public/ShowDocument?docid=2734 EGI Applications on Demand (AoD) service Security Policy]<br>
* Name: <code>vo.access.egi.eu</code>
*[https://documents.egi.eu/public/ShowDocument?docid=3127 Doc vetting manual for the EGI Support team] (DRAFT)
* Scope: Global
* Disciplines: Support Activities
* VOMS servers: voms1/voms2.grid.cesnet.cz
* VO Membership Management: https://perun.metacentrum.cz/perun-registrar-cert/?vo=vo.access.egi.eu
* Contacts:
** EGI Support Team: applications-platform-support@mailman.egi.eu
** Managers:
*** Giuseppe La Rocca (<giuseppe.larocca@egi.eu>)
*** Diego Scardaci (<diego.scardaci@egi.eu>)
*** Gergely Sipos (<gergely.sipos@egi.eu>)
<br/>


== Roadmap  ==
== Roadmap  ==


*<strike>Integration of the JupyterHub as a Service (mid 2018)</strike>
*<strike>Integration of the JupyterHub as a Service (mid 2018)</strike>
*Integration of the Galaxy as a Service (2018)
*<strike>Upgrade of the CSG to Liferay 7 to use the Future Gateway (FG) API server developed in the context of the INDIGO-DataCloud project (2019)</strike>
*Extend the portfolio of applications adopting the COntinuous DElivery of Research Applications in a Distributed Environment (CODE-RADE) [1] approach (mid 2018)
*<strike>Integration the open-source serverless Computing Platform for Data-Processing Applications</strike>
*Investigate the integration of the Ophidia framework for supporting data-intensive analysis (2018)
*Upgrade of the CSG to Liferay 7 to use the Future Gateway (FG) API server developed in the context of the INDIGO-DataCloud project (2018)  
*<strike>Improve the user's experience in the EC3 portal (mid 2018)</strike>
*<strike>Improve the user's experience in the EC3 portal (mid 2018)</strike>
*Integration of CWL
*<strike>Integrate the HNSciCloud voucher schemes in the EC3 portal (2018)</strike>
*<strike>Joined the IN2P3-IRES cloud provider in the vo.access.egi.eu VO (2019)</strike>
*<strike>Configure no.1 instance of PROMINENCE service for the vo.access.egi.eu VO (2019)</strike>
*<strike>Create an Ansible receipt to run big data workflows with ECAS/Ophidia framework in EGI (2019)</strike>
*<strike>Agree OLAs with additional cloud providers of the EGI Federation (2019)</strike>


[1] https://github.com/SouthAfricaDigitalScience <!--
<!--


= Roadmap  =
= Roadmap  =

Latest revision as of 09:19, 23 July 2021

Alert.png This article is Deprecated and has been moved to https://docs.egi.eu/users/applications-on-demand/.



Applications on Demand Service menu: Home Documentation for providers Documentation for developers Architecture




Applications on Demand (AoD) Service Information pages

The EGI Applications on Demand service (AoD) is the EGI’s response to the requirements of researchers who are interested in using applications in a on-demand fashion together with the compute and storage environment needed to compute and store data.

The Service can be accessed through the EGI Marketplace

Service architecture

The EGI Applications on Demand service architecture is presented in Figure.


AoDs_architecturev2.jpg


This service architecture is composed by the following components:

  • A catch-all VO called ‘vo.access.egi.eu’ and a pre-allocated pool of HTC and cloud resources configured for supporting the EGI Applications on Demand research activities. This resource pool currently includes cloud resources from Italy (INFN-Catania), Spain (CESGA) and France (IN2P3-IRES).
  • The X.509 credentials factory service (also called eToken server)[1] is a standard-based solution developed for central management of robot certificates and the provision of Per-User Sub-Proxy (PUSP) certificates. PUSP allows to identify the individual user that operates using a common robot certificate. Permissions to request robot proxy certificates, and contact the server, are granted only to Science Gateways/portal integrated in the Platform and only when the user is authorized.
  • A set of Science Gateways/portal to host applications that users can run accessing the EGI Applications on Demand service. Currently the following three frameworks are used: The FutureGateway Science Gateway (FGSG), the WS-PGRADE/gUSE portal and the Elastic Cloud Computing Cluster (EC3).
  • The EGI VMOps dashboard provides a Graphical User Interface (GUI) to perform Virtual Machine (VM) management operations on the EGI Federated Cloud. The dashboard introduces a user-friendly environment where users can create and manage VMs along with associated storage devices on any of the EGI Federated Cloud providers. It provides a complete view of the deployed applications and a unified resource management experience, independently of the technology driving each of the resource centres of the federation. Users can create new infrastructure topologies, which include a set of VMs, their associated storage and contextualization, a wizard-like builder that guides them through the selection of the virtual appliances, virtual organisation, resource provider, and the final customisation of the VMs that will be deployed. Its tight integration with the AppDB Cloud Marketplace allows for an automatic discovery of the appliances which are supported at each resource provider Once a topology has been created, VMOps allows management actions to be applied both on the set of VMs comprising a topology and on fine-grained actions on each individual VM.
  • The EGI Notebooks, build on top of JupyterHub offers an open-source web application where users can create and share documents that contain live code, equations, visualization and explanatory text.


[1] Valeria Ardizzone, Roberto Barbera, Antonio Calanducci, Marco Fargetta, E. Ingrà, Ivan Porro, Giuseppe La Rocca, Salvatore Monforte, R. Ricceri, Riccardo Rotondo, Diego Scardaci, Andrea Schenone: The DECIDE Science Gateway. Journal of Grid Computing 10(4): 689-707 (2012)

Available resources

Current available resources grouped by categories:

  • Cloud Resources:
    • 246 vCPU cores
    • 781 GB of RAM
    • 2.1TB of object storage
  • High-Throughtput Resources:
    • 9.5Million HEPSPEC
    • 1.4 TB of disk storage

For more details about the resources allocated for supporting this service, please click here.

Operational Level Agreements


GGUS Support Units (SUs)

  • FutureGateway Science Gateway (FGSG):
    • The FGSG GGUS SU is available under the Core Services category
  • Elastic Cloud Computing Cluster (EC3):
    • The EC3 GGUS SU is available under the Core Services category
  • WS-PGRADE/gUSE portal:
    • The WS-PGRADE/gUSE GGUS SU is available under the Core Services category
  • The EGI Notebooks:
    • The EGI Notebooks GGUS SU is available under the EGI Notebooks category
  • The EGI VMOps dashboard:
    • The AppDB GGUS SU is available under the Core Services category


Links for administrators

User approval:


Monitoring

  • The EGI Applications on Demand service components have been registered in the GOCDB and connected with the EGI monitoring system based on ARGO.
  • By default ARGO automatically gathered the services endpoints from the GOCDB and implements simple 'https' checks using standard NAGIOS probes. If necessary new ones can be easily developed and added.
  • The following service components are monitored by ARGO:
    • EGI-FGSG,
    • EGI-NOTEBOOKS,
    • GRIDOPS-WS-PGRADE,
    • GRIDOPS-APPDB, and
    • GRIDOPS-EC3
  • To monitor the service components, check the following ARGO report page.


Accounting

  • Accounting data about the VO users can be checked here: https://accounting.egi.eu/
  • From the EGI Accounting Portal it is possible to check the accounting metrics generated for both grid- and cloud-based resources supporting the vo.access.egi.eu VO.
  • From the top-menu click on 'Restrict View' and 'VO Admin' to check the accounting data of platform users.


EGI_AoDs_Accounting.png

Policies & documents


Useful Links


Roadmap

  • Integration of the JupyterHub as a Service (mid 2018)
  • Upgrade of the CSG to Liferay 7 to use the Future Gateway (FG) API server developed in the context of the INDIGO-DataCloud project (2019)
  • Integration the open-source serverless Computing Platform for Data-Processing Applications
  • Improve the user's experience in the EC3 portal (mid 2018)
  • Integrate the HNSciCloud voucher schemes in the EC3 portal (2018)
  • Joined the IN2P3-IRES cloud provider in the vo.access.egi.eu VO (2019)
  • Configure no.1 instance of PROMINENCE service for the vo.access.egi.eu VO (2019)
  • Create an Ansible receipt to run big data workflows with ECAS/Ophidia framework in EGI (2019)
  • Agree OLAs with additional cloud providers of the EGI Federation (2019)