Alert.png The wiki is deprecated and due to be decommissioned by the end of September 2022.
The content is being migrated to other supports, new updates will be ignored and lost.
If needed you can get in touch with EGI SDIS team using operations @ egi.eu.

Difference between revisions of "ServiceGuides"

From EGIWiki
Jump to navigation Jump to search
m
Line 7: Line 7:
The [https://www.egi.eu/services/cloud-compute/%7CEGI Federated Cloud] is a standards-based, open cloud system that federates institutional clouds to offer a scalable computing platform for data and/or compute driven applications and services in research and science.  
The [https://www.egi.eu/services/cloud-compute/%7CEGI Federated Cloud] is a standards-based, open cloud system that federates institutional clouds to offer a scalable computing platform for data and/or compute driven applications and services in research and science.  


==== Training Materials ====
'''Training Materials'''


*[https://documents.egi.eu/public/ShowDocument?docid=3349 Research Computational Infrastructure: Cloud infrastructures], [ppt]. This training was given by Giuseppe La Rocca & Gergely Sipos, 16-17 Aug 2018, during [http://www.codata.org/datatrieste2018 CODATA-RDA Research Data Science Summer Schoo 2018l]. It provides up-to-dated information about EGI cloud computing and EGI services (AppDB, Jupyter Notebook), including demo and exercise. It also gives an overview about EOSC and EOSC-hub project.
*[https://documents.egi.eu/public/ShowDocument?docid=3349 Research Computational Infrastructure: Cloud infrastructures], [ppt]. This training was given by Giuseppe La Rocca & Gergely Sipos, 16-17 Aug 2018, during [http://www.codata.org/datatrieste2018 CODATA-RDA Research Data Science Summer Schoo 2018l]. It provides up-to-dated information about EGI cloud computing and EGI services (AppDB, Jupyter Notebook), including demo and exercise. It also gives an overview about EOSC and EOSC-hub project.
Line 19: Line 19:
*[https://www.digitalinfrastructures.eu/sites/default/files/DI4R_fedcloud-jocci-V6_0.pptx The EGI Federated Cloud for application developers (API access)], [ppt]. This training is jointly given by Giuseppe La Rocca, Enol Fernandez del Castillo, and Boris Parak, 28 Sep 2016, during [https://www.digitalinfrastructures.eu/content/egi-federated-cloud-developers DI4R 2016]. The topics covers EGI and EGI Federated Cloud, Access the training infrastructure. It also includes hand-on exercises.
*[https://www.digitalinfrastructures.eu/sites/default/files/DI4R_fedcloud-jocci-V6_0.pptx The EGI Federated Cloud for application developers (API access)], [ppt]. This training is jointly given by Giuseppe La Rocca, Enol Fernandez del Castillo, and Boris Parak, 28 Sep 2016, during [https://www.digitalinfrastructures.eu/content/egi-federated-cloud-developers DI4R 2016]. The topics covers EGI and EGI Federated Cloud, Access the training infrastructure. It also includes hand-on exercises.


==== User Manual ====
'''User Manual'''


*https://wiki.egi.eu/wiki/Federated_Cloud_user_support provides detailed information about the service and guideline of access the service
*
https://wiki.egi.eu/wiki/Federated_Cloud_user_support provides detailed information about the service and guideline of access the service


==== Publications ====
'''Publications'''


*[https://www.egi.eu/wp-content/uploads/2016/08/The-EGI-Federated-Cloud-paper.pdf The EGI Federated Cloud e-Infrastructure] (2015), Enol Fernández-del-Castillo, Diego Scardaci, Álvaro López García, in the proceeding [https://www.holacloud.eu/cloud-forward-2015-conference/ HOLACONF-Cloud Forward: From Distributed to Complete Computing, 6-8 October 2015, Scuola Normale Superiore, Pisa]. doi: 10.1016/j.procs.2015.09.235
*[https://www.egi.eu/wp-content/uploads/2016/08/The-EGI-Federated-Cloud-paper.pdf The EGI Federated Cloud e-Infrastructure] (2015), Enol Fernández-del-Castillo, Diego Scardaci, Álvaro López García, in the proceeding [https://www.holacloud.eu/cloud-forward-2015-conference/ HOLACONF-Cloud Forward: From Distributed to Complete Computing, 6-8 October 2015, Scuola Normale Superiore, Pisa]. doi: 10.1016/j.procs.2015.09.235
Line 33: Line 34:
[https://www.egi.eu/services/cloud-container/ The Cloud Container Compute] offers guaranteed computational resources in a secure and isolated environment with standard API access, without the overhead of managing the operating system.  
[https://www.egi.eu/services/cloud-container/ The Cloud Container Compute] offers guaranteed computational resources in a secure and isolated environment with standard API access, without the overhead of managing the operating system.  


==== Training Material ====
'''Training Material'''


*[https://documents.egi.eu/public/RetrieveFile?docid=3353&version=1&filename=egi-container-compute-training-20180913.pptx Training Material]. [ppt]. This training is given by Enol Fernandez, 17 Jul 2018, during [https://indico.egi.eu/indico/event/4100/ EOSC-hub tech-talk: Cloud, containers, orchestration]. It introduces the service architecture and interfaces, options and attributes.
*
[https://documents.egi.eu/public/RetrieveFile?docid=3353&version=1&filename=egi-container-compute-training-20180913.pptx Training Material]. [ppt]. This training is given by Enol Fernandez, 17 Jul 2018, during [https://indico.egi.eu/indico/event/4100/ EOSC-hub tech-talk: Cloud, containers, orchestration]. It introduces the service architecture and interfaces, options and attributes.


==== User Manual ====
'''User Manual'''


*https://wiki.egi.eu/wiki/Federated_Cloud_Containers: provides information about the docker image and configurations details.
*
https://wiki.egi.eu/wiki/Federated_Cloud_Containers: provides information about the docker image and configurations details.


== Application on Demand  ==
== Application on Demand  ==
Line 45: Line 48:
[https://www.egi.eu/services/applications-on-demand/ Application on Demand] service allows user-friendly access to a portfolio of applications and application hosting frameworks (Science Gateways, VREs) that are configured to use the dedicated pool of cloud computing and HTC clusters from EGI. The service also allows users to run their own simulation/analysis models with custom input data.  
[https://www.egi.eu/services/applications-on-demand/ Application on Demand] service allows user-friendly access to a portfolio of applications and application hosting frameworks (Science Gateways, VREs) that are configured to use the dedicated pool of cloud computing and HTC clusters from EGI. The service also allows users to run their own simulation/analysis models with custom input data.  


==== Training Material ====
'''Training Material'''


*[https://indico.egi.eu/indico/event/3378/contribution/0/material/slides/0.pptx Overview of the EGI Applications on Demand (AoD) service]. [ppt]. This training is provided by Gergely Sipos and Giuseppe La Rocca, 13 Jul 2017, during [https://indico.egi.eu/indico/event/3378/ Webinar: EGI Applications On Demand Service]. It gives an overview of the service components, information details on how to access, available services provided via the framework. An introductory webinar (video recording) can be found at: http://connect.ct.infn.it/p8w34jegx9s/
*
[https://indico.egi.eu/indico/event/3378/contribution/0/material/slides/0.pptx Overview of the EGI Applications on Demand (AoD) service]. [ppt]. This training is provided by Gergely Sipos and Giuseppe La Rocca, 13 Jul 2017, during [https://indico.egi.eu/indico/event/3378/ Webinar: EGI Applications On Demand Service]. It gives an overview of the service components, information details on how to access, available services provided via the framework. An introductory webinar (video recording) can be found at: http://connect.ct.infn.it/p8w34jegx9s/


==== User Manual ====
'''User Manual'''


*https://wiki.egi.eu/wiki/AoD is the wiki page of the service providing information details on how to access the service.
*
https://wiki.egi.eu/wiki/AoD is the wiki page of the service providing information details on how to access the service.


==== Publication ====
'''Publication'''


*[https://documents.egi.eu/document/3132 EGI Applications on Demand Service][pdf]. (2017), Gergely Sipos, Giuseppe La Rocca. In the proceeding of [http://iwsg2017.psnc.pl/ IWSG'17 conference]
*[https://documents.egi.eu/document/3132 EGI Applications on Demand Service][pdf]. (2017), Gergely Sipos, Giuseppe La Rocca. In the proceeding of [http://iwsg2017.psnc.pl/ IWSG'17 conference]
Line 61: Line 66:
The [https://www.egi.eu/about/newsletters/introducing-the-egi-datahub-prototype/ EGI DataHub] is a service prototype designed to make data discoverable and available in an easy way across all EGI federated resources.  
The [https://www.egi.eu/about/newsletters/introducing-the-egi-datahub-prototype/ EGI DataHub] is a service prototype designed to make data discoverable and available in an easy way across all EGI federated resources.  


==== Training Materials ====
'''Training Materials'''


*[https://www.digitalinfrastructures.eu/sites/default/files/EGIDataHub-tutorial-20160930_v3_0.pptx The EGI Open Data Platform and DataHub][ppt]. This training is given by Diego Scardaci during [https://www.digitalinfrastructures.eu/content/egi-datahub-and-open-data-platform-0 DI4R 2016]. It introduces what is the EGI DataHub, together with 4 hand-on exercise.
*[https://www.digitalinfrastructures.eu/sites/default/files/EGIDataHub-tutorial-20160930_v3_0.pptx The EGI Open Data Platform and DataHub][ppt]. This training is given by Diego Scardaci during [https://www.digitalinfrastructures.eu/content/egi-datahub-and-open-data-platform-0 DI4R 2016]. It introduces what is the EGI DataHub, together with 4 hand-on exercise.
Line 67: Line 72:
*[https://indico.egi.eu/indico/event/2969/contribution/0/material/slides/0.pptx The EGI Open Data Platform] [ppt]. This training is given by Lukasz Dutka (CYFRONET) and Matthew Viljoen (EGI.eu), 17 May 2016, during the webinar [https://indico.egi.eu/indico/event/2969/ The EGI Open Data Platform - Towards Scientific Data Hubs]. The topics cover OneData, EGI DataHub and EGI Open Data Platform (ODP). A video recording of this webinar can be found at http://connect.ct.infn.it/p68qfn1h2km/
*[https://indico.egi.eu/indico/event/2969/contribution/0/material/slides/0.pptx The EGI Open Data Platform] [ppt]. This training is given by Lukasz Dutka (CYFRONET) and Matthew Viljoen (EGI.eu), 17 May 2016, during the webinar [https://indico.egi.eu/indico/event/2969/ The EGI Open Data Platform - Towards Scientific Data Hubs]. The topics cover OneData, EGI DataHub and EGI Open Data Platform (ODP). A video recording of this webinar can be found at http://connect.ct.infn.it/p68qfn1h2km/


==== User Manuals ====
'''User Manuals'''
 
*
[https://onedata.org/#/home/documentation/doc/user_guide.html User Guide] provides information for end users


*[https://onedata.org/#/home/documentation/doc/user_guide.html User Guide] provides information for end users
*[https://onedata.org/#/home/documentation/doc/admin_guide.html Admin Guide] provides information for administrators
*[https://onedata.org/#/home/documentation/doc/admin_guide.html Admin Guide] provides information for administrators


==== Publication ====
'''Publication'''


*[https://www.egi.eu/wp-content/uploads/2016/08/EGI_Open_Data_Platform_and_EGI_Datahub.pdf Towards European Open Science Commons: The EGI Open Data Platform and The EGI DataHub], (2016), Matthew Viljoena, Lukasz Dutka, Bartosz Kryza, Yin Chen, [http://www.holacloud.eu/cloud-forward-conference-2016/ Cloud Forward 2018], Madrid, Oct. 2016.
*
[https://www.egi.eu/wp-content/uploads/2016/08/EGI_Open_Data_Platform_and_EGI_Datahub.pdf Towards European Open Science Commons: The EGI Open Data Platform and The EGI DataHub], (2016), Matthew Viljoena, Lukasz Dutka, Bartosz Kryza, Yin Chen, [http://www.holacloud.eu/cloud-forward-conference-2016/ Cloud Forward 2018], Madrid, Oct. 2016.


== Check-In  ==
== Check-In  ==
Line 80: Line 88:
[https://www.egi.eu/services/check-in/ EGI Check-In] is a proxy service that operates as a central hub to connect federated Identity Providers (IdPs) with EGI service providers. Check-in allows users to select their preferred IdP so that they can access and use EGI services in a uniform and easy way.  
[https://www.egi.eu/services/check-in/ EGI Check-In] is a proxy service that operates as a central hub to connect federated Identity Providers (IdPs) with EGI service providers. Check-in allows users to select their preferred IdP so that they can access and use EGI services in a uniform and easy way.  


==== Training Material ====
'''Training Material'''
 
*
[https://indico.egi.eu/indico/event/4086/material/slides/2.pptx Community AAI with Check-In] [ppt]. This training is given by Nicolas Liampotis, 15 Jun 2018, during [https://indico.egi.eu/indico/event/4086/ EOSC-hub tech-talk: AAI webinar]. This training introduces Check-in in a nutshell, users and groups management, and service options.


*[https://indico.egi.eu/indico/event/4086/material/slides/2.pptx Community AAI with Check-In] [ppt]. This training is given by Nicolas Liampotis, 15 Jun 2018, during [https://indico.egi.eu/indico/event/4086/ EOSC-hub tech-talk: AAI webinar]. This training introduces Check-in in a nutshell, users and groups management, and service options.
'''User Manuals'''


==== User Manuals ====
*
[https://wiki.egi.eu/wiki/AAI_usage_guide AAI usage guide]: A wiki page contains information about registering an account with the EGI AAI in order to access a variety of EGI tools and services using the same account.


*[https://wiki.egi.eu/wiki/AAI_usage_guide AAI usage guide]: A wiki page contains information about registering an account with the EGI AAI in order to access a variety of EGI tools and services using the same account.
*[https://wiki.egi.eu/wiki/AAI_guide_for_IdPs AAI guide for IdPs]: A wiki page contains information about integrating your identity provider with the EGI AAI Proxy in order to allow users in your community to access EGI tools and services.  
*[https://wiki.egi.eu/wiki/AAI_guide_for_IdPs AAI guide for IdPs]: A wiki page contains information about integrating your identity provider with the EGI AAI Proxy in order to allow users in your community to access EGI tools and services.  
*[https://wiki.egi.eu/wiki/AAI_guide_for_SPs AAI guide_for SPs]: A wiki page contains information about connecting services to the EGI AAI Check-in service in order to allow user login through Check-in and to receive users' attributes.
*[https://wiki.egi.eu/wiki/AAI_guide_for_SPs AAI guide_for SPs]: A wiki page contains information about connecting services to the EGI AAI Check-in service in order to allow user login through Check-in and to receive users' attributes.


==== Additional Information ====
'''Additional Information'''


*[https://www.egi.eu/wp-content/uploads/2017/09/Check-in.pdf EGI Check-In Service]: provide general information about the service.
*
[https://www.egi.eu/wp-content/uploads/2017/09/Check-in.pdf EGI Check-In Service]: provide general information about the service.


== EGI Jupyter Notebook  ==
== EGI Jupyter Notebook  ==
Line 98: Line 110:
[https://wiki.egi.eu/wiki/EGI_Jupyter EGI Jupyter Notebook] is an 'as a Service' environment based on the Jupyter technology, offering a browser-based, scalable tool for interactive data analysis. The EGI Notebooks environment provides users with notebooks where they can combine text, mathematics, computations and rich media output. EGI Notebooks is a multi-user service and can scale to multiple servers based on the EGI cloud service.  
[https://wiki.egi.eu/wiki/EGI_Jupyter EGI Jupyter Notebook] is an 'as a Service' environment based on the Jupyter technology, offering a browser-based, scalable tool for interactive data analysis. The EGI Notebooks environment provides users with notebooks where they can combine text, mathematics, computations and rich media output. EGI Notebooks is a multi-user service and can scale to multiple servers based on the EGI cloud service.  


==== Training Materials ====
'''Training Material'''
 
*
[https://documents.egi.eu/secure/ShowDocument?docid=3341 The EGI Jupyter Notebook] This training is delivered by Giuseppe La Rocca, 10 Jul 2018, during [http://www.envriplus.eu/2018/05/08/international-summer-school-data-management-in-environmental-earth-science-infrastructures-theory-practice-july-9-july-13/ ENVRIplus International Summer School]


*[https://documents.egi.eu/secure/ShowDocument?docid=3341 The EGI Jupyter Notebook] This training is delivered by Giuseppe La Rocca, 10 Jul 2018, during [http://www.envriplus.eu/2018/05/08/international-summer-school-data-management-in-environmental-earth-science-infrastructures-theory-practice-july-9-july-13/ ENVRIplus International Summer School]


== Online Storage  ==
== Online Storage  ==
Line 106: Line 120:
[https://www.egi.eu/services/online-storage/ Online Storage] allows you to store data in a reliable and high-quality environment and share it across distributed teams. Your data can be accessed through different standard protocols and can be replicated across different providers to increase fault-tolerance. Online Storage gives you complete control over the data you share and with whom  
[https://www.egi.eu/services/online-storage/ Online Storage] allows you to store data in a reliable and high-quality environment and share it across distributed teams. Your data can be accessed through different standard protocols and can be replicated across different providers to increase fault-tolerance. Online Storage gives you complete control over the data you share and with whom  


==== Training Material ====
'''Training Material'''


*[https://documents.egi.eu/secure/ShowDocument?docid=3359 EGI Online Storage] [ppt]. This information is provided by Vincenzo Spinoso, 17 Sep 2018. It gives an overview of the service, service components, access interfaces, and examples of usage.
*
[https://documents.egi.eu/secure/ShowDocument?docid=3359 EGI Online Storage] [ppt]. This information is provided by Vincenzo Spinoso, 17 Sep 2018. It gives an overview of the service, service components, access interfaces, and examples of usage.


==== User Manuals ====
'''User Manual'''


*https://wiki.egi.eu/wiki/Online_Storage : A wiki page provides detailed information about the service.
*https://wiki.egi.eu/wiki/Online_Storage : A wiki page provides detailed information about the service.


== Workload Manager ==
== Workload Manager ==


[https://www.egi.eu/services/workload-manager/ EGI Workload Manager] (also known as DIRAC4EGI) is a service is provided to the EGI community as a workload management service used to distribute the users' computing tasks among the available resources both HTC and cloud.  
[https://www.egi.eu/services/workload-manager/ EGI Workload Manager] (also known as DIRAC4EGI) is a service is provided to the EGI community as a workload management service used to distribute the users' computing tasks among the available resources both HTC and cloud.  


==== Training Material ====
'''Training Material'''


*[https://indico.egi.eu/indico/event/2978/contribution/0/material/slides/1.pdf DIRAC system for EGI communities] This training is provided by Andrei Tsaregorodtsev (CNRS), 07 Jun 2016, during [https://indico.egi.eu/indico/event/2978/ Webinar: DIRAC system for EGI communities]. It introduces the DIRAC setup for EGI, Workload manager, File Catalogue and web portal interface. The recording of the webinar is available at: http://connect.ct.infn.it/p3tj8sthvf0/
*
[https://indico.egi.eu/indico/event/2978/contribution/0/material/slides/1.pdf DIRAC system for EGI communities] This training is provided by Andrei Tsaregorodtsev (CNRS), 07 Jun 2016, during [https://indico.egi.eu/indico/event/2978/ Webinar: DIRAC system for EGI communities]. It introduces the DIRAC setup for EGI, Workload manager, File Catalogue and web portal interface. The recording of the webinar is available at: http://connect.ct.infn.it/p3tj8sthvf0/


==== User Manuals ====
'''User Manual'''


*https://wiki.egi.eu/wiki/Workload_Manager: A wiki page provide detailed information about service and guide for getting access.
*https://wiki.egi.eu/wiki/Workload_Manager: A wiki page provide detailed information about service and guide for getting access.
Line 128: Line 144:
= 3rd party services (not part of the EGI Service Catalogue)  =
= 3rd party services (not part of the EGI Service Catalogue)  =


== EGI VMOps dashboard ==
== EGI VMOps dashboard ==


The EGI Application Database (AppDB) has recently evolved its functionalities from its currently catalogue of applications and virtual machines (VMs) to include a Graphical User Interface (GUI) allowing authorized users to perform basic VM management operations.  
The EGI Application Database (AppDB) has recently evolved its functionalities from its currently catalogue of applications and virtual machines (VMs) to include a Graphical User Interface (GUI) allowing authorized users to perform basic VM management operations.  


==== Training Material ====
'''Training Material'''
 
*
[https://indico.egi.eu/indico/event/3522/ AppDB VMOps Dashboard: A graphical portal to manage applications and services on federated cloud infrastructures]. This training is given by William Vassilis Karageorgos (GRNET), 24 Nov 2017, during [https://indico.egi.eu/indico/event/3522/ Webinar: AppDB VMOps Dashboard: A graphical portal to manage applications and services on federated cloud infrastructures]. It introduces the key capabilities of the VMOps dashboard, particularly:


*[https://indico.egi.eu/indico/event/3522/ AppDB VMOps Dashboard: A graphical portal to manage applications and services on federated cloud infrastructures]. This training is given by William Vassilis Karageorgos (GRNET), 24 Nov 2017, during [https://indico.egi.eu/indico/event/3522/ Webinar: AppDB VMOps Dashboard: A graphical portal to manage applications and services on federated cloud infrastructures]. It introduces the key capabilities of the VMOps dashboard, particularly:
**A wizard-like builder that guides users through the selection of virtual machine images, virtual appliances, cloud resources and contextualization scripts to deploy complex applications or services in federated clouds.  
**A wizard-like builder that guides users through the selection of virtual machine images, virtual appliances, cloud resources and contextualization scripts to deploy complex applications or services in federated clouds.  
**Graphical interfaces and tools to monitor and manage your applications/services in federated clouds, independently of underlying cloud technologies.  
**Graphical interfaces and tools to monitor and manage your applications/services in federated clouds, independently of underlying cloud technologies.  
Line 143: Line 161:
The Webinar was recorded. You can watch the recording at: http://connect.ct.infn.it/p5cty7tor87/  
The Webinar was recorded. You can watch the recording at: http://connect.ct.infn.it/p5cty7tor87/  


==== Additional Information ====
'''Additional Information'''


*https://wiki.egi.eu/wiki/Federated_Cloud_AppDB_VMOps_Dashboard
*
https://wiki.egi.eu/wiki/Federated_Cloud_AppDB_VMOps_Dashboard


== WS-PGRADE  ==
== WS-PGRADE  ==
Line 151: Line 170:
WS-PGRADE is an open source, Liferay-based web portal environment for the design and execution of compute and data intensive applications on grid and cloud platforms. Applications can be defined in the form of data-driven workflows in WS-PGRADE. Steps within a workflow can be executable applications, Virtual Machine images, or online (Web) Services. These computational steps are connected through data-dependencies within the workflow: The output of one computation can serve as input of the next one. WS-PRADE resolves data dependencies and manages the computational operations on the infrastructure resources. The back-end system of WS-PGRADE is compatible with various academic and commercial grid and cloud systems, including federated clouds and gLite clusters of EGI. Moreover, scientific communities can connect workflow-specific Web interfaces to WS-PGRADE to simplify the execution of workflows for researchers. This is enabled by the 'Remote API' component of WS-PGRADE, a programming interface that exposes the workflow registry and the workflow manager service to third party clients. During the webinar the main capabilities of WS-PGRADE will be presented and demonstrated. The event will also provide opportunity to discuss scientific applications that would benefit from the use of WS-PGRADE to access EGI cloud and grid resources.  
WS-PGRADE is an open source, Liferay-based web portal environment for the design and execution of compute and data intensive applications on grid and cloud platforms. Applications can be defined in the form of data-driven workflows in WS-PGRADE. Steps within a workflow can be executable applications, Virtual Machine images, or online (Web) Services. These computational steps are connected through data-dependencies within the workflow: The output of one computation can serve as input of the next one. WS-PRADE resolves data dependencies and manages the computational operations on the infrastructure resources. The back-end system of WS-PGRADE is compatible with various academic and commercial grid and cloud systems, including federated clouds and gLite clusters of EGI. Moreover, scientific communities can connect workflow-specific Web interfaces to WS-PGRADE to simplify the execution of workflows for researchers. This is enabled by the 'Remote API' component of WS-PGRADE, a programming interface that exposes the workflow registry and the workflow manager service to third party clients. During the webinar the main capabilities of WS-PGRADE will be presented and demonstrated. The event will also provide opportunity to discuss scientific applications that would benefit from the use of WS-PGRADE to access EGI cloud and grid resources.  


==== Training Material ====
'''Training Material'''


*[https://indico.egi.eu/indico/event/2977/contribution/1/material/slides/0.pdf Workflow applications on EGI with WS-PGRADE] This training is given by Peter Kacsuk (MTA SZTAKI), 26 May 2016, during [https://indico.egi.eu/indico/event/2977/ Webinar: Workflow applications on EGI with WS-PGRADE]. A recording of the training can be found at http://connect.ct.infn.it/p6gvxozwgg4/
*
[https://indico.egi.eu/indico/event/2977/contribution/1/material/slides/0.pdf Workflow applications on EGI with WS-PGRADE] This training is given by Peter Kacsuk (MTA SZTAKI), 26 May 2016, during [https://indico.egi.eu/indico/event/2977/ Webinar: Workflow applications on EGI with WS-PGRADE]. A recording of the training can be found at http://connect.ct.infn.it/p6gvxozwgg4/


==== Additional Information ====
'''Additional Information'''


*http://wspgrade.hu
*
http://wspgrade.hu


== Hadoop on the EGI Federated Cloud ==
== Hadoop on the EGI Federated Cloud ==


With the rapid increase of data volumes in scientific computations, the importance of utilising parallel and distributed computing paradigms in data processing is becoming more and more important. Hadoop is an open source implementation of the MapReduce framework supporting processing large datasets in parallel and on multiple nodes in a reliable and fault-tolerant manner. Scientific workflow systems and science gateways are high level environments to facilitate the development, orchestration and execution of complex experiments from a user-friendly graphical user interface. Integrating MapReduce/Hadoop with such workflow systems and science gateways enables scientists to conduct complex data intensive experiments utilising the power of the MapReduce paradigm from the convenience provided by science gateway frameworks.  
With the rapid increase of data volumes in scientific computations, the importance of utilising parallel and distributed computing paradigms in data processing is becoming more and more important. Hadoop is an open source implementation of the MapReduce framework supporting processing large datasets in parallel and on multiple nodes in a reliable and fault-tolerant manner. Scientific workflow systems and science gateways are high level environments to facilitate the development, orchestration and execution of complex experiments from a user-friendly graphical user interface. Integrating MapReduce/Hadoop with such workflow systems and science gateways enables scientists to conduct complex data intensive experiments utilising the power of the MapReduce paradigm from the convenience provided by science gateway frameworks.  


==== Training Material ====
'''Training Material'''


*[https://indico.egi.eu/indico/event/2931/material/slides/0.ppt Hadoop on the EGI Federated Cloud]. This training is given by Tamas Kiss (University of Westminster, London, UK), and Carlos Blanco (University of Cantabria). 17 March 2016, during [https://indico.egi.eu/indico/event/2931/ Webinar: Hadoop on the EGI Federated Cloud]. This presentation and demonstration illustrates how easily Hadoop clusters can be deployed on EGI FedCloud resources, Hadoop applications can be executed on these clusters, and finally resources can be released after execution. Users of the EGI FedCloud WS-PGRADE gateway can import and parameterise pre-prepared workflows for the above tasks published in a public workflow repository. Users can set the type/flavour and number of desired nodes in the Hadoop cluster, select the target EGI FedCloud site, and define the Hadoop executable and the desired data source and destination. All three functionality (create Hadoop cluster, execute Hadoop job, destroy Hadoop cluster) can be executed as a standalone job or can be combined into more complex workflows automating different user scenarios.
*
[https://indico.egi.eu/indico/event/2931/material/slides/0.ppt Hadoop on the EGI Federated Cloud]. This training is given by Tamas Kiss (University of Westminster, London, UK), and Carlos Blanco (University of Cantabria). 17 March 2016, during [https://indico.egi.eu/indico/event/2931/ Webinar: Hadoop on the EGI Federated Cloud]. This presentation and demonstration illustrates how easily Hadoop clusters can be deployed on EGI FedCloud resources, Hadoop applications can be executed on these clusters, and finally resources can be released after execution. Users of the EGI FedCloud WS-PGRADE gateway can import and parameterise pre-prepared workflows for the above tasks published in a public workflow repository. Users can set the type/flavour and number of desired nodes in the Hadoop cluster, select the target EGI FedCloud site, and define the Hadoop executable and the desired data source and destination. All three functionality (create Hadoop cluster, execute Hadoop job, destroy Hadoop cluster) can be executed as a standalone job or can be combined into more complex workflows automating different user scenarios.


==== User Manual ====
'''User Manual'''


*[https://indico.egi.eu/indico/event/2931/material/0/0.docx Hadoop-EGI user manual v1.3]
*
[https://indico.egi.eu/indico/event/2931/material/0/0.docx Hadoop-EGI user manual v1.3]

Revision as of 17:16, 3 December 2018

This wiki page provides training materials and service documentations that help end users better understand EGI (and third-party's) services.

EGI Services

Cloud Computing

The Federated Cloud is a standards-based, open cloud system that federates institutional clouds to offer a scalable computing platform for data and/or compute driven applications and services in research and science.

Training Materials

  • Introduction and command line usage of the EGI Federated Cloud [ppt]. This training material is provided by Gergely Sipos, 05 Oct 2016. The goals of the training are to help users to learn the concept of IaaS Cloud Computing, conceptual model of the EGI federated Cloud, to obtain skills in using the standard interface of the EGI federated cloud and become an active user. Hand-on exercises are included.

User Manual

https://wiki.egi.eu/wiki/Federated_Cloud_user_support provides detailed information about the service and guideline of access the service

Publications

Cloud Container Compute

The Cloud Container Compute offers guaranteed computational resources in a secure and isolated environment with standard API access, without the overhead of managing the operating system.

Training Material

Training Material. [ppt]. This training is given by Enol Fernandez, 17 Jul 2018, during EOSC-hub tech-talk: Cloud, containers, orchestration. It introduces the service architecture and interfaces, options and attributes.

User Manual

https://wiki.egi.eu/wiki/Federated_Cloud_Containers: provides information about the docker image and configurations details.

Application on Demand

Application on Demand service allows user-friendly access to a portfolio of applications and application hosting frameworks (Science Gateways, VREs) that are configured to use the dedicated pool of cloud computing and HTC clusters from EGI. The service also allows users to run their own simulation/analysis models with custom input data.

Training Material

Overview of the EGI Applications on Demand (AoD) service. [ppt]. This training is provided by Gergely Sipos and Giuseppe La Rocca, 13 Jul 2017, during Webinar: EGI Applications On Demand Service. It gives an overview of the service components, information details on how to access, available services provided via the framework. An introductory webinar (video recording) can be found at: http://connect.ct.infn.it/p8w34jegx9s/

User Manual

https://wiki.egi.eu/wiki/AoD is the wiki page of the service providing information details on how to access the service.

Publication

DataHub

The EGI DataHub is a service prototype designed to make data discoverable and available in an easy way across all EGI federated resources.

Training Materials

User Manuals

User Guide provides information for end users

Publication

Towards European Open Science Commons: The EGI Open Data Platform and The EGI DataHub, (2016), Matthew Viljoena, Lukasz Dutka, Bartosz Kryza, Yin Chen, Cloud Forward 2018, Madrid, Oct. 2016.

Check-In

EGI Check-In is a proxy service that operates as a central hub to connect federated Identity Providers (IdPs) with EGI service providers. Check-in allows users to select their preferred IdP so that they can access and use EGI services in a uniform and easy way.

Training Material

Community AAI with Check-In [ppt]. This training is given by Nicolas Liampotis, 15 Jun 2018, during EOSC-hub tech-talk: AAI webinar. This training introduces Check-in in a nutshell, users and groups management, and service options.

User Manuals

AAI usage guide: A wiki page contains information about registering an account with the EGI AAI in order to access a variety of EGI tools and services using the same account.

  • AAI guide for IdPs: A wiki page contains information about integrating your identity provider with the EGI AAI Proxy in order to allow users in your community to access EGI tools and services.
  • AAI guide_for SPs: A wiki page contains information about connecting services to the EGI AAI Check-in service in order to allow user login through Check-in and to receive users' attributes.

Additional Information

EGI Check-In Service: provide general information about the service.

EGI Jupyter Notebook

EGI Jupyter Notebook is an 'as a Service' environment based on the Jupyter technology, offering a browser-based, scalable tool for interactive data analysis. The EGI Notebooks environment provides users with notebooks where they can combine text, mathematics, computations and rich media output. EGI Notebooks is a multi-user service and can scale to multiple servers based on the EGI cloud service.

Training Material

The EGI Jupyter Notebook This training is delivered by Giuseppe La Rocca, 10 Jul 2018, during ENVRIplus International Summer School


Online Storage

Online Storage allows you to store data in a reliable and high-quality environment and share it across distributed teams. Your data can be accessed through different standard protocols and can be replicated across different providers to increase fault-tolerance. Online Storage gives you complete control over the data you share and with whom

Training Material

EGI Online Storage [ppt]. This information is provided by Vincenzo Spinoso, 17 Sep 2018. It gives an overview of the service, service components, access interfaces, and examples of usage.

User Manual

Workload Manager

EGI Workload Manager (also known as DIRAC4EGI) is a service is provided to the EGI community as a workload management service used to distribute the users' computing tasks among the available resources both HTC and cloud.

Training Material

DIRAC system for EGI communities This training is provided by Andrei Tsaregorodtsev (CNRS), 07 Jun 2016, during Webinar: DIRAC system for EGI communities. It introduces the DIRAC setup for EGI, Workload manager, File Catalogue and web portal interface. The recording of the webinar is available at: http://connect.ct.infn.it/p3tj8sthvf0/

User Manual

3rd party services (not part of the EGI Service Catalogue)

EGI VMOps dashboard

The EGI Application Database (AppDB) has recently evolved its functionalities from its currently catalogue of applications and virtual machines (VMs) to include a Graphical User Interface (GUI) allowing authorized users to perform basic VM management operations.

Training Material

AppDB VMOps Dashboard: A graphical portal to manage applications and services on federated cloud infrastructures. This training is given by William Vassilis Karageorgos (GRNET), 24 Nov 2017, during Webinar: AppDB VMOps Dashboard: A graphical portal to manage applications and services on federated cloud infrastructures. It introduces the key capabilities of the VMOps dashboard, particularly:

    • A wizard-like builder that guides users through the selection of virtual machine images, virtual appliances, cloud resources and contextualization scripts to deploy complex applications or services in federated clouds.
    • Graphical interfaces and tools to monitor and manage your applications/services in federated clouds, independently of underlying cloud technologies.
    • A scalable architecture composed of a generic front-end and several, technology-specific back-ends for load balancing, and with a RESTful API to integrate with 3rd party services.

The EGI Applications Database is developed and operated by the Institute of Accelerating Systems and Applications (IASA) on behalf of the EGI Foundation as an EGI Core Service.

The Webinar was recorded. You can watch the recording at: http://connect.ct.infn.it/p5cty7tor87/

Additional Information

https://wiki.egi.eu/wiki/Federated_Cloud_AppDB_VMOps_Dashboard

WS-PGRADE

WS-PGRADE is an open source, Liferay-based web portal environment for the design and execution of compute and data intensive applications on grid and cloud platforms. Applications can be defined in the form of data-driven workflows in WS-PGRADE. Steps within a workflow can be executable applications, Virtual Machine images, or online (Web) Services. These computational steps are connected through data-dependencies within the workflow: The output of one computation can serve as input of the next one. WS-PRADE resolves data dependencies and manages the computational operations on the infrastructure resources. The back-end system of WS-PGRADE is compatible with various academic and commercial grid and cloud systems, including federated clouds and gLite clusters of EGI. Moreover, scientific communities can connect workflow-specific Web interfaces to WS-PGRADE to simplify the execution of workflows for researchers. This is enabled by the 'Remote API' component of WS-PGRADE, a programming interface that exposes the workflow registry and the workflow manager service to third party clients. During the webinar the main capabilities of WS-PGRADE will be presented and demonstrated. The event will also provide opportunity to discuss scientific applications that would benefit from the use of WS-PGRADE to access EGI cloud and grid resources.

Training Material

Workflow applications on EGI with WS-PGRADE This training is given by Peter Kacsuk (MTA SZTAKI), 26 May 2016, during Webinar: Workflow applications on EGI with WS-PGRADE. A recording of the training can be found at http://connect.ct.infn.it/p6gvxozwgg4/

Additional Information

http://wspgrade.hu

Hadoop on the EGI Federated Cloud

With the rapid increase of data volumes in scientific computations, the importance of utilising parallel and distributed computing paradigms in data processing is becoming more and more important. Hadoop is an open source implementation of the MapReduce framework supporting processing large datasets in parallel and on multiple nodes in a reliable and fault-tolerant manner. Scientific workflow systems and science gateways are high level environments to facilitate the development, orchestration and execution of complex experiments from a user-friendly graphical user interface. Integrating MapReduce/Hadoop with such workflow systems and science gateways enables scientists to conduct complex data intensive experiments utilising the power of the MapReduce paradigm from the convenience provided by science gateway frameworks.

Training Material

Hadoop on the EGI Federated Cloud. This training is given by Tamas Kiss (University of Westminster, London, UK), and Carlos Blanco (University of Cantabria). 17 March 2016, during Webinar: Hadoop on the EGI Federated Cloud. This presentation and demonstration illustrates how easily Hadoop clusters can be deployed on EGI FedCloud resources, Hadoop applications can be executed on these clusters, and finally resources can be released after execution. Users of the EGI FedCloud WS-PGRADE gateway can import and parameterise pre-prepared workflows for the above tasks published in a public workflow repository. Users can set the type/flavour and number of desired nodes in the Hadoop cluster, select the target EGI FedCloud site, and define the Hadoop executable and the desired data source and destination. All three functionality (create Hadoop cluster, execute Hadoop job, destroy Hadoop cluster) can be executed as a standalone job or can be combined into more complex workflows automating different user scenarios.

User Manual

Hadoop-EGI user manual v1.3