Difference between revisions of "ServiceGuides"

From EGIWiki
Jump to: navigation, search
(Workload Manager)
Line 1: Line 1:
This wiki page provides training materials and service documentations that help end users better understand EGI (and third-party's) services.  
+
This wiki page provides training materials and service documentations that help end users better understand EGI (and third-party's) services. You can ask questions and discuss a particular service at: https://community.egi.eu/  
You can ask questions and discuss a particular service at: https://community.egi.eu/
 
  
 
= EGI Services  =
 
= EGI Services  =
Line 20: Line 19:
 
*[https://www.digitalinfrastructures.eu/sites/default/files/DI4R_fedcloud-jocci-V6_0.pptx The EGI Federated Cloud for application developers (API access)], [ppt]. This training is jointly given by Giuseppe La Rocca, Enol Fernandez del Castillo, and Boris Parak, 28 Sep 2016, during [https://www.digitalinfrastructures.eu/content/egi-federated-cloud-developers DI4R 2016]. The topics covers EGI and EGI Federated Cloud, Access the training infrastructure. It also includes hand-on exercises.
 
*[https://www.digitalinfrastructures.eu/sites/default/files/DI4R_fedcloud-jocci-V6_0.pptx The EGI Federated Cloud for application developers (API access)], [ppt]. This training is jointly given by Giuseppe La Rocca, Enol Fernandez del Castillo, and Boris Parak, 28 Sep 2016, during [https://www.digitalinfrastructures.eu/content/egi-federated-cloud-developers DI4R 2016]. The topics covers EGI and EGI Federated Cloud, Access the training infrastructure. It also includes hand-on exercises.
  
'''User Manual'''
+
'''User Manual'''  
  
* https://wiki.egi.eu/wiki/Federated_Cloud_user_support provides detailed information about the service and guideline of access the service
+
*https://wiki.egi.eu/wiki/Federated_Cloud_user_support provides detailed information about the service and guideline of access the service
  
 
'''Publications'''  
 
'''Publications'''  
Line 34: Line 33:
 
[https://www.egi.eu/services/cloud-container/ The Cloud Container Compute] offers guaranteed computational resources in a secure and isolated environment with standard API access, without the overhead of managing the operating system.  
 
[https://www.egi.eu/services/cloud-container/ The Cloud Container Compute] offers guaranteed computational resources in a secure and isolated environment with standard API access, without the overhead of managing the operating system.  
  
'''Training Material'''
+
'''Training Material'''  
  
 
*[https://documents.egi.eu/public/RetrieveFile?docid=3353&version=1&filename=egi-container-compute-training-20180913.pptx Training Material]. [ppt]. This training is given by Enol Fernandez, 17 Jul 2018, during [https://indico.egi.eu/indico/event/4100/ EOSC-hub tech-talk: Cloud, containers, orchestration]. It introduces the service architecture and interfaces, options and attributes.
 
*[https://documents.egi.eu/public/RetrieveFile?docid=3353&version=1&filename=egi-container-compute-training-20180913.pptx Training Material]. [ppt]. This training is given by Enol Fernandez, 17 Jul 2018, during [https://indico.egi.eu/indico/event/4100/ EOSC-hub tech-talk: Cloud, containers, orchestration]. It introduces the service architecture and interfaces, options and attributes.
  
'''User Manual'''
+
'''User Manual'''  
  
 
*https://wiki.egi.eu/wiki/Federated_Cloud_Containers: provides information about the docker image and configurations details.
 
*https://wiki.egi.eu/wiki/Federated_Cloud_Containers: provides information about the docker image and configurations details.
  
 +
<br>
  
 
== Application on Demand  ==
 
== Application on Demand  ==
Line 47: Line 47:
 
[https://www.egi.eu/services/applications-on-demand/ Application on Demand] service allows user-friendly access to a portfolio of applications and application hosting frameworks (Science Gateways, VREs) that are configured to use the dedicated pool of cloud computing and HTC clusters from EGI. The service also allows users to run their own simulation/analysis models with custom input data.  
 
[https://www.egi.eu/services/applications-on-demand/ Application on Demand] service allows user-friendly access to a portfolio of applications and application hosting frameworks (Science Gateways, VREs) that are configured to use the dedicated pool of cloud computing and HTC clusters from EGI. The service also allows users to run their own simulation/analysis models with custom input data.  
  
'''Training Material'''
+
'''Training Material'''  
  
 
*[https://indico.egi.eu/indico/event/3378/contribution/0/material/slides/0.pptx Overview of the EGI Applications on Demand (AoD) service]. [ppt]. This training is provided by Gergely Sipos and Giuseppe La Rocca, 13 Jul 2017, during [https://indico.egi.eu/indico/event/3378/ Webinar: EGI Applications On Demand Service]. It gives an overview of the service components, information details on how to access, available services provided via the framework. An introductory webinar (video recording) can be found at: http://connect.ct.infn.it/p8w34jegx9s/
 
*[https://indico.egi.eu/indico/event/3378/contribution/0/material/slides/0.pptx Overview of the EGI Applications on Demand (AoD) service]. [ppt]. This training is provided by Gergely Sipos and Giuseppe La Rocca, 13 Jul 2017, during [https://indico.egi.eu/indico/event/3378/ Webinar: EGI Applications On Demand Service]. It gives an overview of the service components, information details on how to access, available services provided via the framework. An introductory webinar (video recording) can be found at: http://connect.ct.infn.it/p8w34jegx9s/
  
'''User Manual'''
+
'''User Manual'''  
  
 
*https://wiki.egi.eu/wiki/AoD is the wiki page of the service providing information details on how to access the service.
 
*https://wiki.egi.eu/wiki/AoD is the wiki page of the service providing information details on how to access the service.
Line 59: Line 59:
 
*[https://documents.egi.eu/document/3132 EGI Applications on Demand Service][pdf]. (2017), Gergely Sipos, Giuseppe La Rocca. In the proceeding of [http://iwsg2017.psnc.pl/ IWSG'17 conference]
 
*[https://documents.egi.eu/document/3132 EGI Applications on Demand Service][pdf]. (2017), Gergely Sipos, Giuseppe La Rocca. In the proceeding of [http://iwsg2017.psnc.pl/ IWSG'17 conference]
  
 +
<br>
  
 
== DataHub  ==
 
== DataHub  ==
Line 70: Line 71:
 
*[https://indico.egi.eu/indico/event/2969/contribution/0/material/slides/0.pptx The EGI Open Data Platform] [ppt]. This training is given by Lukasz Dutka (CYFRONET) and Matthew Viljoen (EGI.eu), 17 May 2016, during the webinar [https://indico.egi.eu/indico/event/2969/ The EGI Open Data Platform - Towards Scientific Data Hubs]. The topics cover OneData, EGI DataHub and EGI Open Data Platform (ODP). A video recording of this webinar can be found at http://connect.ct.infn.it/p68qfn1h2km/
 
*[https://indico.egi.eu/indico/event/2969/contribution/0/material/slides/0.pptx The EGI Open Data Platform] [ppt]. This training is given by Lukasz Dutka (CYFRONET) and Matthew Viljoen (EGI.eu), 17 May 2016, during the webinar [https://indico.egi.eu/indico/event/2969/ The EGI Open Data Platform - Towards Scientific Data Hubs]. The topics cover OneData, EGI DataHub and EGI Open Data Platform (ODP). A video recording of this webinar can be found at http://connect.ct.infn.it/p68qfn1h2km/
  
'''User Manuals'''
+
'''User Manuals'''  
  
 
*[https://onedata.org/#/home/documentation/doc/user_guide.html User Guide] provides information for end users
 
*[https://onedata.org/#/home/documentation/doc/user_guide.html User Guide] provides information for end users
Line 76: Line 77:
 
*[https://onedata.org/#/home/documentation/doc/admin_guide.html Admin Guide] provides information for administrators
 
*[https://onedata.org/#/home/documentation/doc/admin_guide.html Admin Guide] provides information for administrators
  
'''Publication'''
+
'''Publication'''  
  
 
*[https://www.egi.eu/wp-content/uploads/2016/08/EGI_Open_Data_Platform_and_EGI_Datahub.pdf Towards European Open Science Commons: The EGI Open Data Platform and The EGI DataHub], (2016), Matthew Viljoena, Lukasz Dutka, Bartosz Kryza, Yin Chen, [http://www.holacloud.eu/cloud-forward-conference-2016/ Cloud Forward 2018], Madrid, Oct. 2016.
 
*[https://www.egi.eu/wp-content/uploads/2016/08/EGI_Open_Data_Platform_and_EGI_Datahub.pdf Towards European Open Science Commons: The EGI Open Data Platform and The EGI DataHub], (2016), Matthew Viljoena, Lukasz Dutka, Bartosz Kryza, Yin Chen, [http://www.holacloud.eu/cloud-forward-conference-2016/ Cloud Forward 2018], Madrid, Oct. 2016.
  
'''Additional Materials'''
+
'''Additional Materials'''  
* [https://drive.google.com/drive/folders/1EJof8J5WizKjM5W5hAFSaKcagsvErGM2 DataHub google drive folder] provides documentation and presentation slides about the service
+
 
* [https://egi-datahub.readthedocs.io/en/latest/index.html EGI DataHub] is a documentation repository providing many technology details
+
*[https://drive.google.com/drive/folders/1EJof8J5WizKjM5W5hAFSaKcagsvErGM2 DataHub google drive folder] provides documentation and presentation slides about the service  
* [https://community.egi.eu/c/egi-services/datahub EGI community forum for DataHub discussion] includes many discussions
+
*[https://egi-datahub.readthedocs.io/en/latest/index.html EGI DataHub] is a documentation repository providing many technology details  
 +
*[https://community.egi.eu/c/egi-services/datahub EGI community forum for DataHub discussion] includes many discussions
  
 
== Check-In  ==
 
== Check-In  ==
Line 89: Line 91:
 
[https://www.egi.eu/services/check-in/ EGI Check-In] is a proxy service that operates as a central hub to connect federated Identity Providers (IdPs) with EGI service providers. Check-in allows users to select their preferred IdP so that they can access and use EGI services in a uniform and easy way.  
 
[https://www.egi.eu/services/check-in/ EGI Check-In] is a proxy service that operates as a central hub to connect federated Identity Providers (IdPs) with EGI service providers. Check-in allows users to select their preferred IdP so that they can access and use EGI services in a uniform and easy way.  
  
'''Training Material'''
+
'''Training Material'''  
  
 
*[https://indico.egi.eu/indico/event/4086/material/slides/2.pptx Community AAI with Check-In] [ppt]. This training is given by Nicolas Liampotis, 15 Jun 2018, during [https://indico.egi.eu/indico/event/4086/ EOSC-hub tech-talk: AAI webinar]. This training introduces Check-in in a nutshell, users and groups management, and service options.
 
*[https://indico.egi.eu/indico/event/4086/material/slides/2.pptx Community AAI with Check-In] [ppt]. This training is given by Nicolas Liampotis, 15 Jun 2018, during [https://indico.egi.eu/indico/event/4086/ EOSC-hub tech-talk: AAI webinar]. This training introduces Check-in in a nutshell, users and groups management, and service options.
  
'''User Manuals'''
+
'''User Manuals'''  
  
 
*[https://wiki.egi.eu/wiki/AAI_usage_guide AAI usage guide]: A wiki page contains information about registering an account with the EGI AAI in order to access a variety of EGI tools and services using the same account.
 
*[https://wiki.egi.eu/wiki/AAI_usage_guide AAI usage guide]: A wiki page contains information about registering an account with the EGI AAI in order to access a variety of EGI tools and services using the same account.
Line 100: Line 102:
 
*[https://wiki.egi.eu/wiki/AAI_guide_for_SPs AAI guide_for SPs]: A wiki page contains information about connecting services to the EGI AAI Check-in service in order to allow user login through Check-in and to receive users' attributes.
 
*[https://wiki.egi.eu/wiki/AAI_guide_for_SPs AAI guide_for SPs]: A wiki page contains information about connecting services to the EGI AAI Check-in service in order to allow user login through Check-in and to receive users' attributes.
  
'''Additional Information'''
+
'''Additional Information'''  
  
 
*[https://www.egi.eu/wp-content/uploads/2017/09/Check-in.pdf EGI Check-In Service]: provide general information about the service.
 
*[https://www.egi.eu/wp-content/uploads/2017/09/Check-in.pdf EGI Check-In Service]: provide general information about the service.
  
 +
== EGI Archive storage<br>  ==
 +
 +
'''User Manuals'''<br>
 +
 +
*[https://www.dcache.org/manuals/Book-2.7/cookbook/cb-clients-srm.shtml SRM manual]: the scripts command line used by the service.<br>
  
 
== EGI Jupyter Notebook  ==
 
== EGI Jupyter Notebook  ==
Line 109: Line 116:
 
[https://wiki.egi.eu/wiki/EGI_Jupyter EGI Jupyter Notebook] is an 'as a Service' environment based on the Jupyter technology, offering a browser-based, scalable tool for interactive data analysis. The EGI Notebooks environment provides users with notebooks where they can combine text, mathematics, computations and rich media output. EGI Notebooks is a multi-user service and can scale to multiple servers based on the EGI cloud service.  
 
[https://wiki.egi.eu/wiki/EGI_Jupyter EGI Jupyter Notebook] is an 'as a Service' environment based on the Jupyter technology, offering a browser-based, scalable tool for interactive data analysis. The EGI Notebooks environment provides users with notebooks where they can combine text, mathematics, computations and rich media output. EGI Notebooks is a multi-user service and can scale to multiple servers based on the EGI cloud service.  
  
'''User guide'''
+
'''User guide''' https://egi-notebooks.readthedocs.io/en/latest/  
https://egi-notebooks.readthedocs.io/en/latest/
+
 
 +
'''Training Material'''
  
'''Training Material'''
 
 
*[https://www.slideshare.net/EGI_Foundation/reproducible-open-science-with-egi-notebooks-binder-and-zenodo Turoial: Reproducible Open Science with EGI Notebooks, Binder and Zenodo]. This training is given by Enol Fernandez and Gergely Sipos during [https://indico4.twgrid.org/indico/event/8/program ISGC 2019 conference], 31 Mar - 5 Apr 2019, Taipei. The tutorial goes through the main features of the EGI Notebooks service and shows how to use it with Binder for Open Science. Hands-on exercises is provided including Python based on real applications from the environmental sciences domain. The integrated use of Notebooks with EGI’s DataHub service is also demonstrated. Communities or national e-infrastructures who would like to setup a similar service locally can learn about how to do this.  
 
*[https://www.slideshare.net/EGI_Foundation/reproducible-open-science-with-egi-notebooks-binder-and-zenodo Turoial: Reproducible Open Science with EGI Notebooks, Binder and Zenodo]. This training is given by Enol Fernandez and Gergely Sipos during [https://indico4.twgrid.org/indico/event/8/program ISGC 2019 conference], 31 Mar - 5 Apr 2019, Taipei. The tutorial goes through the main features of the EGI Notebooks service and shows how to use it with Binder for Open Science. Hands-on exercises is provided including Python based on real applications from the environmental sciences domain. The integrated use of Notebooks with EGI’s DataHub service is also demonstrated. Communities or national e-infrastructures who would like to setup a similar service locally can learn about how to do this.  
*[https://indico.egi.eu/indico/event/4537/ Webinar: The EGI Notebooks for interactive analysis of data using EGI storage and compute services]. This training is delivered by Giuseppe La Rocca and Enol Fernandez, 19 Mar 2019. It includes a presentation that gives an overview of the service and a live demo that gives a tour guide of the service. The recording is available: http://connect.ct.infn.it/p59n3ojbyqn
+
*[https://indico.egi.eu/indico/event/4537/ Webinar: The EGI Notebooks for interactive analysis of data using EGI storage and compute services]. This training is delivered by Giuseppe La Rocca and Enol Fernandez, 19 Mar 2019. It includes a presentation that gives an overview of the service and a live demo that gives a tour guide of the service. The recording is available: http://connect.ct.infn.it/p59n3ojbyqn  
 
*[https://documents.egi.eu/secure/ShowDocument?docid=3341 The EGI Jupyter Notebook] This training is delivered by Giuseppe La Rocca, 10 Jul 2018, during [http://www.envriplus.eu/2018/05/08/international-summer-school-data-management-in-environmental-earth-science-infrastructures-theory-practice-july-9-july-13/ ENVRIplus International Summer School]
 
*[https://documents.egi.eu/secure/ShowDocument?docid=3341 The EGI Jupyter Notebook] This training is delivered by Giuseppe La Rocca, 10 Jul 2018, during [http://www.envriplus.eu/2018/05/08/international-summer-school-data-management-in-environmental-earth-science-infrastructures-theory-practice-july-9-july-13/ ENVRIplus International Summer School]
  
Line 121: Line 128:
 
[https://www.egi.eu/services/online-storage/ Online Storage] allows you to store data in a reliable and high-quality environment and share it across distributed teams. Your data can be accessed through different standard protocols and can be replicated across different providers to increase fault-tolerance. Online Storage gives you complete control over the data you share and with whom  
 
[https://www.egi.eu/services/online-storage/ Online Storage] allows you to store data in a reliable and high-quality environment and share it across distributed teams. Your data can be accessed through different standard protocols and can be replicated across different providers to increase fault-tolerance. Online Storage gives you complete control over the data you share and with whom  
  
'''Training Material'''
+
'''Training Material'''  
  
 
*[https://documents.egi.eu/secure/ShowDocument?docid=3359 EGI Online Storage] [ppt]. This information is provided by Vincenzo Spinoso, 17 Sep 2018. It gives an overview of the service, service components, access interfaces, and examples of usage.
 
*[https://documents.egi.eu/secure/ShowDocument?docid=3359 EGI Online Storage] [ppt]. This information is provided by Vincenzo Spinoso, 17 Sep 2018. It gives an overview of the service, service components, access interfaces, and examples of usage.
  
'''User Manual'''
+
'''User Manual'''  
  
*https://wiki.egi.eu/wiki/Online_Storage : A wiki page provides detailed information about the service.
+
*https://wiki.egi.eu/wiki/Online_Storage&nbsp;: A wiki page provides detailed information about the service.
  
 
== Workload Manager  ==
 
== Workload Manager  ==
Line 133: Line 140:
 
[https://www.egi.eu/services/workload-manager/ EGI Workload Manager] (also known as DIRAC4EGI) is a service is provided to the EGI community as a workload management service used to distribute the users' computing tasks among the available resources both HTC and cloud.  
 
[https://www.egi.eu/services/workload-manager/ EGI Workload Manager] (also known as DIRAC4EGI) is a service is provided to the EGI community as a workload management service used to distribute the users' computing tasks among the available resources both HTC and cloud.  
  
'''Training Material'''
+
'''Training Material'''  
* [https://indico.cern.ch/event/756635/contributions/3382879/attachments/1843652/3023920/DIRAC_WMS_Resources.pdf Training materials] delivered by Andrei Tsaregorodtsev during [https://indico.cern.ch/event/756635/contributions/ the 9th DIRAC users' meeting], May 2019, London, providing updated information about the service.
+
 
* Trainings on DIRAC workload and data management system is delivered by Andrei Tsaregorodtsev (CNRS, CPPM) during [https://events.ego-gw.it/indico/conferenceOtherViews.py?view=standard&confId=77 Virgo Computing School], 8th Nov 2018, Cascina(Pisa), Italy. The topics covers the following aspects:
+
*[https://indico.cern.ch/event/756635/contributions/3382879/attachments/1843652/3023920/DIRAC_WMS_Resources.pdf Training materials] delivered by Andrei Tsaregorodtsev during [https://indico.cern.ch/event/756635/contributions/ the 9th DIRAC users' meeting], May 2019, London, providing updated information about the service.  
** [https://events.ego-gw.it/indico/getFile.py/access?contribId=4&resId=7&materialId=slides&confId=77 Distributed Infrastructure for Scientific Application] is an overview of DIRAC framework
+
*Trainings on DIRAC workload and data management system is delivered by Andrei Tsaregorodtsev (CNRS, CPPM) during [https://events.ego-gw.it/indico/conferenceOtherViews.py?view=standard&confId=77 Virgo Computing School], 8th Nov 2018, Cascina(Pisa), Italy. The topics covers the following aspects:  
** [https://events.ego-gw.it/indico/getFile.py/access?contribId=4&resId=0&materialId=slides&confId=77 Getting started] introduces the training plan and setups
+
**[https://events.ego-gw.it/indico/getFile.py/access?contribId=4&resId=7&materialId=slides&confId=77 Distributed Infrastructure for Scientific Application] is an overview of DIRAC framework  
** [https://events.ego-gw.it/indico/getFile.py/access?contribId=4&resId=1&materialId=slides&confId=77 Client installation] introduces DIRAC client software installation
+
**[https://events.ego-gw.it/indico/getFile.py/access?contribId=4&resId=0&materialId=slides&confId=77 Getting started] introduces the training plan and setups  
** [https://events.ego-gw.it/indico/getFile.py/access?contribId=4&resId=2&materialId=slides&confId=77 Job Management] introduces how to submit jobs via DIRAC web portal and command lines
+
**[https://events.ego-gw.it/indico/getFile.py/access?contribId=4&resId=1&materialId=slides&confId=77 Client installation] introduces DIRAC client software installation  
** [https://events.ego-gw.it/indico/getFile.py/access?contribId=4&resId=3&materialId=slides&confId=77 Mandelbrot quest] is a hand-on exercise to use DIRAC tool to explore the mandelbrot algorithm and images
+
**[https://events.ego-gw.it/indico/getFile.py/access?contribId=4&resId=2&materialId=slides&confId=77 Job Management] introduces how to submit jobs via DIRAC web portal and command lines  
** [https://events.ego-gw.it/indico/getFile.py/access?contribId=4&resId=4&materialId=slides&confId=77 Managing Data] introduces DIRAC Data Management system
+
**[https://events.ego-gw.it/indico/getFile.py/access?contribId=4&resId=3&materialId=slides&confId=77 Mandelbrot quest] is a hand-on exercise to use DIRAC tool to explore the mandelbrot algorithm and images  
** [https://events.ego-gw.it/indico/getFile.py/access?contribId=4&resId=5&materialId=slides&confId=77 Bulk job submission] introduces how to submit multiple jobs
+
**[https://events.ego-gw.it/indico/getFile.py/access?contribId=4&resId=4&materialId=slides&confId=77 Managing Data] introduces DIRAC Data Management system  
** [https://events.ego-gw.it/indico/getFile.py/access?contribId=4&resId=6&materialId=slides&confId=77 Job with input and output data] introduces how to input and output data from jobs  
+
**[https://events.ego-gw.it/indico/getFile.py/access?contribId=4&resId=5&materialId=slides&confId=77 Bulk job submission] introduces how to submit multiple jobs  
 +
**[https://events.ego-gw.it/indico/getFile.py/access?contribId=4&resId=6&materialId=slides&confId=77 Job with input and output data] introduces how to input and output data from jobs  
 
*[https://indico.egi.eu/indico/event/2978/contribution/0/material/slides/1.pdf DIRAC system for EGI communities] This training is provided by Andrei Tsaregorodtsev (CNRS), 07 Jun 2016, during [https://indico.egi.eu/indico/event/2978/ Webinar: DIRAC system for EGI communities]. It introduces the DIRAC setup for EGI, Workload manager, File Catalogue and web portal interface. The recording of the webinar is available at: http://connect.ct.infn.it/p3tj8sthvf0/
 
*[https://indico.egi.eu/indico/event/2978/contribution/0/material/slides/1.pdf DIRAC system for EGI communities] This training is provided by Andrei Tsaregorodtsev (CNRS), 07 Jun 2016, during [https://indico.egi.eu/indico/event/2978/ Webinar: DIRAC system for EGI communities]. It introduces the DIRAC setup for EGI, Workload manager, File Catalogue and web portal interface. The recording of the webinar is available at: http://connect.ct.infn.it/p3tj8sthvf0/
  
'''User Manual'''
+
'''User Manual'''  
  
 
*https://wiki.egi.eu/wiki/Workload_Manager: A wiki page provide detailed information about service and guide for getting access.
 
*https://wiki.egi.eu/wiki/Workload_Manager: A wiki page provide detailed information about service and guide for getting access.
Line 152: Line 160:
 
== Data Transfer  ==
 
== Data Transfer  ==
  
Data Transfer allows you to move any type of data files asynchronously from one place to another. The service includes dedicated interfaces to display statistics of on-going transfers and manage network resources.
+
Data Transfer allows you to move any type of data files asynchronously from one place to another. The service includes dedicated interfaces to display statistics of on-going transfers and manage network resources.  
  
'''Training Material'''
+
'''Training Material'''  
* [https://documents.egi.eu/secure/RetrieveFile?docid=3453&version=1&filename=Taining-%20EGI%20Data%20transfer%20Service%20%20V4.pptx EGI Data Transfer Service] gives an overview introduction of the service, provide information for users to get started with the service.
 
  
'''User Manual'''
+
*[https://documents.egi.eu/secure/RetrieveFile?docid=3453&version=1&filename=Taining-%20EGI%20Data%20transfer%20Service%20%20V4.pptx EGI Data Transfer Service] gives an overview introduction of the service, provide information for users to get started with the service.
* [[EGI Data Transfer]] wiki is the place to learn more about the service
 
* https://webfts.cern.ch/ (See Virtual Tour on the page)
 
  
==EGI Marketplace ==
+
'''User Manual'''
The EGI Marketplace provides an online platform for exposing and promoting services to new scientific and market segments. The Marketplace provides all the necessary functionalities for bringing together offering and demand to make research happen. All services are provided by the EGI Federation and partner institutions.
 
  
'''Training Material'''
+
*[[EGI Data Transfer]] wiki is the place to learn more about the service
* A [https://youtu.be/s848e3RpP5w YouTube video] is given by Björn Backeberg, on 6th Dec 2018, that step-by-step guides users to work through [https://www.egi.eu/internal-services/marketplace/ EGI Marketplace], introducing how to search and place an order of a service.
+
*https://webfts.cern.ch/ (See Virtual Tour on the page)
 +
 
 +
== EGI Marketplace  ==
 +
 
 +
The EGI Marketplace provides an online platform for exposing and promoting services to new scientific and market segments. The Marketplace provides all the necessary functionalities for bringing together offering and demand to make research happen. All services are provided by the EGI Federation and partner institutions.
 +
 
 +
'''Training Material'''  
 +
 
 +
*A [https://youtu.be/s848e3RpP5w YouTube video] is given by Björn Backeberg, on 6th Dec 2018, that step-by-step guides users to work through [https://www.egi.eu/internal-services/marketplace/ EGI Marketplace], introducing how to search and place an order of a service.
  
 
= 3rd party services (not part of the EGI Service Catalogue)  =
 
= 3rd party services (not part of the EGI Service Catalogue)  =
Line 173: Line 185:
 
The EGI Application Database (AppDB) has recently evolved its functionalities from its currently catalogue of applications and virtual machines (VMs) to include a Graphical User Interface (GUI) allowing authorized users to perform basic VM management operations.  
 
The EGI Application Database (AppDB) has recently evolved its functionalities from its currently catalogue of applications and virtual machines (VMs) to include a Graphical User Interface (GUI) allowing authorized users to perform basic VM management operations.  
  
'''Training Material'''
+
'''Training Material'''  
  
 
*[https://indico.egi.eu/indico/event/3522/ AppDB VMOps Dashboard: A graphical portal to manage applications and services on federated cloud infrastructures]. This training is given by William Vassilis Karageorgos (GRNET), 24 Nov 2017, during [https://indico.egi.eu/indico/event/3522/ Webinar: AppDB VMOps Dashboard: A graphical portal to manage applications and services on federated cloud infrastructures]. It introduces the key capabilities of the VMOps dashboard, particularly:
 
*[https://indico.egi.eu/indico/event/3522/ AppDB VMOps Dashboard: A graphical portal to manage applications and services on federated cloud infrastructures]. This training is given by William Vassilis Karageorgos (GRNET), 24 Nov 2017, during [https://indico.egi.eu/indico/event/3522/ Webinar: AppDB VMOps Dashboard: A graphical portal to manage applications and services on federated cloud infrastructures]. It introduces the key capabilities of the VMOps dashboard, particularly:
Line 185: Line 197:
 
The Webinar was recorded. You can watch the recording at: http://connect.ct.infn.it/p5cty7tor87/  
 
The Webinar was recorded. You can watch the recording at: http://connect.ct.infn.it/p5cty7tor87/  
  
'''Additional Information'''
+
'''Additional Information'''  
  
 
*https://wiki.egi.eu/wiki/Federated_Cloud_AppDB_VMOps_Dashboard
 
*https://wiki.egi.eu/wiki/Federated_Cloud_AppDB_VMOps_Dashboard
  
 +
<br>
  
 
== WS-PGRADE  ==
 
== WS-PGRADE  ==
Line 194: Line 207:
 
WS-PGRADE is an open source, Liferay-based web portal environment for the design and execution of compute and data intensive applications on grid and cloud platforms. Applications can be defined in the form of data-driven workflows in WS-PGRADE. Steps within a workflow can be executable applications, Virtual Machine images, or online (Web) Services. These computational steps are connected through data-dependencies within the workflow: The output of one computation can serve as input of the next one. WS-PRADE resolves data dependencies and manages the computational operations on the infrastructure resources. The back-end system of WS-PGRADE is compatible with various academic and commercial grid and cloud systems, including federated clouds and gLite clusters of EGI. Moreover, scientific communities can connect workflow-specific Web interfaces to WS-PGRADE to simplify the execution of workflows for researchers. This is enabled by the 'Remote API' component of WS-PGRADE, a programming interface that exposes the workflow registry and the workflow manager service to third party clients. During the webinar the main capabilities of WS-PGRADE will be presented and demonstrated. The event will also provide opportunity to discuss scientific applications that would benefit from the use of WS-PGRADE to access EGI cloud and grid resources.  
 
WS-PGRADE is an open source, Liferay-based web portal environment for the design and execution of compute and data intensive applications on grid and cloud platforms. Applications can be defined in the form of data-driven workflows in WS-PGRADE. Steps within a workflow can be executable applications, Virtual Machine images, or online (Web) Services. These computational steps are connected through data-dependencies within the workflow: The output of one computation can serve as input of the next one. WS-PRADE resolves data dependencies and manages the computational operations on the infrastructure resources. The back-end system of WS-PGRADE is compatible with various academic and commercial grid and cloud systems, including federated clouds and gLite clusters of EGI. Moreover, scientific communities can connect workflow-specific Web interfaces to WS-PGRADE to simplify the execution of workflows for researchers. This is enabled by the 'Remote API' component of WS-PGRADE, a programming interface that exposes the workflow registry and the workflow manager service to third party clients. During the webinar the main capabilities of WS-PGRADE will be presented and demonstrated. The event will also provide opportunity to discuss scientific applications that would benefit from the use of WS-PGRADE to access EGI cloud and grid resources.  
  
'''Training Material'''
+
'''Training Material'''  
  
 
*[https://indico.egi.eu/indico/event/2977/contribution/1/material/slides/0.pdf Workflow applications on EGI with WS-PGRADE] This training is given by Peter Kacsuk (MTA SZTAKI), 26 May 2016, during [https://indico.egi.eu/indico/event/2977/ Webinar: Workflow applications on EGI with WS-PGRADE]. A recording of the training can be found at http://connect.ct.infn.it/p6gvxozwgg4/
 
*[https://indico.egi.eu/indico/event/2977/contribution/1/material/slides/0.pdf Workflow applications on EGI with WS-PGRADE] This training is given by Peter Kacsuk (MTA SZTAKI), 26 May 2016, during [https://indico.egi.eu/indico/event/2977/ Webinar: Workflow applications on EGI with WS-PGRADE]. A recording of the training can be found at http://connect.ct.infn.it/p6gvxozwgg4/
  
'''Additional Information'''
+
'''Additional Information'''  
  
 
*http://wspgrade.hu
 
*http://wspgrade.hu
  
 +
<br>
  
 
== Hadoop on the EGI Federated Cloud  ==
 
== Hadoop on the EGI Federated Cloud  ==
Line 207: Line 221:
 
With the rapid increase of data volumes in scientific computations, the importance of utilising parallel and distributed computing paradigms in data processing is becoming more and more important. Hadoop is an open source implementation of the MapReduce framework supporting processing large datasets in parallel and on multiple nodes in a reliable and fault-tolerant manner. Scientific workflow systems and science gateways are high level environments to facilitate the development, orchestration and execution of complex experiments from a user-friendly graphical user interface. Integrating MapReduce/Hadoop with such workflow systems and science gateways enables scientists to conduct complex data intensive experiments utilising the power of the MapReduce paradigm from the convenience provided by science gateway frameworks.  
 
With the rapid increase of data volumes in scientific computations, the importance of utilising parallel and distributed computing paradigms in data processing is becoming more and more important. Hadoop is an open source implementation of the MapReduce framework supporting processing large datasets in parallel and on multiple nodes in a reliable and fault-tolerant manner. Scientific workflow systems and science gateways are high level environments to facilitate the development, orchestration and execution of complex experiments from a user-friendly graphical user interface. Integrating MapReduce/Hadoop with such workflow systems and science gateways enables scientists to conduct complex data intensive experiments utilising the power of the MapReduce paradigm from the convenience provided by science gateway frameworks.  
  
'''Training Material'''
+
'''Training Material'''  
  
 
*[https://indico.egi.eu/indico/event/2931/material/slides/0.ppt Hadoop on the EGI Federated Cloud]. This training is given by Tamas Kiss (University of Westminster, London, UK), and Carlos Blanco (University of Cantabria). 17 March 2016, during [https://indico.egi.eu/indico/event/2931/ Webinar: Hadoop on the EGI Federated Cloud]. This presentation and demonstration illustrates how easily Hadoop clusters can be deployed on EGI FedCloud resources, Hadoop applications can be executed on these clusters, and finally resources can be released after execution. Users of the EGI FedCloud WS-PGRADE gateway can import and parameterise pre-prepared workflows for the above tasks published in a public workflow repository. Users can set the type/flavour and number of desired nodes in the Hadoop cluster, select the target EGI FedCloud site, and define the Hadoop executable and the desired data source and destination. All three functionality (create Hadoop cluster, execute Hadoop job, destroy Hadoop cluster) can be executed as a standalone job or can be combined into more complex workflows automating different user scenarios.
 
*[https://indico.egi.eu/indico/event/2931/material/slides/0.ppt Hadoop on the EGI Federated Cloud]. This training is given by Tamas Kiss (University of Westminster, London, UK), and Carlos Blanco (University of Cantabria). 17 March 2016, during [https://indico.egi.eu/indico/event/2931/ Webinar: Hadoop on the EGI Federated Cloud]. This presentation and demonstration illustrates how easily Hadoop clusters can be deployed on EGI FedCloud resources, Hadoop applications can be executed on these clusters, and finally resources can be released after execution. Users of the EGI FedCloud WS-PGRADE gateway can import and parameterise pre-prepared workflows for the above tasks published in a public workflow repository. Users can set the type/flavour and number of desired nodes in the Hadoop cluster, select the target EGI FedCloud site, and define the Hadoop executable and the desired data source and destination. All three functionality (create Hadoop cluster, execute Hadoop job, destroy Hadoop cluster) can be executed as a standalone job or can be combined into more complex workflows automating different user scenarios.
  
'''User Manual'''
+
'''User Manual'''  
  
 
*[https://indico.egi.eu/indico/event/2931/material/0/0.docx Hadoop-EGI user manual v1.3]
 
*[https://indico.egi.eu/indico/event/2931/material/0/0.docx Hadoop-EGI user manual v1.3]
 +
 +
<br>

Revision as of 13:25, 11 September 2019

This wiki page provides training materials and service documentations that help end users better understand EGI (and third-party's) services. You can ask questions and discuss a particular service at: https://community.egi.eu/

EGI Services

Cloud Computing

The EGI Federated Cloud is a standards-based, open cloud system that federates institutional clouds to offer a scalable computing platform for data and/or compute driven applications and services in research and science.

Training Materials

  • Introduction and command line usage of the EGI Federated Cloud [ppt]. This training material is provided by Gergely Sipos, 05 Oct 2016. The goals of the training are to help users to learn the concept of IaaS Cloud Computing, conceptual model of the EGI federated Cloud, to obtain skills in using the standard interface of the EGI federated cloud and become an active user. Hand-on exercises are included.

User Manual

Publications

Cloud Container Compute

The Cloud Container Compute offers guaranteed computational resources in a secure and isolated environment with standard API access, without the overhead of managing the operating system.

Training Material

User Manual


Application on Demand

Application on Demand service allows user-friendly access to a portfolio of applications and application hosting frameworks (Science Gateways, VREs) that are configured to use the dedicated pool of cloud computing and HTC clusters from EGI. The service also allows users to run their own simulation/analysis models with custom input data.

Training Material

User Manual

Publication


DataHub

The EGI DataHub is a service prototype designed to make data discoverable and available in an easy way across all EGI federated resources.

Training Materials

User Manuals

Publication

Additional Materials

Check-In

EGI Check-In is a proxy service that operates as a central hub to connect federated Identity Providers (IdPs) with EGI service providers. Check-in allows users to select their preferred IdP so that they can access and use EGI services in a uniform and easy way.

Training Material

User Manuals

  • AAI usage guide: A wiki page contains information about registering an account with the EGI AAI in order to access a variety of EGI tools and services using the same account.
  • AAI guide for IdPs: A wiki page contains information about integrating your identity provider with the EGI AAI Proxy in order to allow users in your community to access EGI tools and services.
  • AAI guide_for SPs: A wiki page contains information about connecting services to the EGI AAI Check-in service in order to allow user login through Check-in and to receive users' attributes.

Additional Information

EGI Archive storage

User Manuals

  • SRM manual: the scripts command line used by the service.

EGI Jupyter Notebook

EGI Jupyter Notebook is an 'as a Service' environment based on the Jupyter technology, offering a browser-based, scalable tool for interactive data analysis. The EGI Notebooks environment provides users with notebooks where they can combine text, mathematics, computations and rich media output. EGI Notebooks is a multi-user service and can scale to multiple servers based on the EGI cloud service.

User guide https://egi-notebooks.readthedocs.io/en/latest/

Training Material

Online Storage

Online Storage allows you to store data in a reliable and high-quality environment and share it across distributed teams. Your data can be accessed through different standard protocols and can be replicated across different providers to increase fault-tolerance. Online Storage gives you complete control over the data you share and with whom

Training Material

  • EGI Online Storage [ppt]. This information is provided by Vincenzo Spinoso, 17 Sep 2018. It gives an overview of the service, service components, access interfaces, and examples of usage.

User Manual

Workload Manager

EGI Workload Manager (also known as DIRAC4EGI) is a service is provided to the EGI community as a workload management service used to distribute the users' computing tasks among the available resources both HTC and cloud.

Training Material

User Manual

Data Transfer

Data Transfer allows you to move any type of data files asynchronously from one place to another. The service includes dedicated interfaces to display statistics of on-going transfers and manage network resources.

Training Material

  • EGI Data Transfer Service gives an overview introduction of the service, provide information for users to get started with the service.

User Manual

EGI Marketplace

The EGI Marketplace provides an online platform for exposing and promoting services to new scientific and market segments. The Marketplace provides all the necessary functionalities for bringing together offering and demand to make research happen. All services are provided by the EGI Federation and partner institutions.

Training Material

  • A YouTube video is given by Björn Backeberg, on 6th Dec 2018, that step-by-step guides users to work through EGI Marketplace, introducing how to search and place an order of a service.

3rd party services (not part of the EGI Service Catalogue)

EGI VMOps dashboard

The EGI Application Database (AppDB) has recently evolved its functionalities from its currently catalogue of applications and virtual machines (VMs) to include a Graphical User Interface (GUI) allowing authorized users to perform basic VM management operations.

Training Material

    • A wizard-like builder that guides users through the selection of virtual machine images, virtual appliances, cloud resources and contextualization scripts to deploy complex applications or services in federated clouds.
    • Graphical interfaces and tools to monitor and manage your applications/services in federated clouds, independently of underlying cloud technologies.
    • A scalable architecture composed of a generic front-end and several, technology-specific back-ends for load balancing, and with a RESTful API to integrate with 3rd party services.

The EGI Applications Database is developed and operated by the Institute of Accelerating Systems and Applications (IASA) on behalf of the EGI Foundation as an EGI Core Service.

The Webinar was recorded. You can watch the recording at: http://connect.ct.infn.it/p5cty7tor87/

Additional Information


WS-PGRADE

WS-PGRADE is an open source, Liferay-based web portal environment for the design and execution of compute and data intensive applications on grid and cloud platforms. Applications can be defined in the form of data-driven workflows in WS-PGRADE. Steps within a workflow can be executable applications, Virtual Machine images, or online (Web) Services. These computational steps are connected through data-dependencies within the workflow: The output of one computation can serve as input of the next one. WS-PRADE resolves data dependencies and manages the computational operations on the infrastructure resources. The back-end system of WS-PGRADE is compatible with various academic and commercial grid and cloud systems, including federated clouds and gLite clusters of EGI. Moreover, scientific communities can connect workflow-specific Web interfaces to WS-PGRADE to simplify the execution of workflows for researchers. This is enabled by the 'Remote API' component of WS-PGRADE, a programming interface that exposes the workflow registry and the workflow manager service to third party clients. During the webinar the main capabilities of WS-PGRADE will be presented and demonstrated. The event will also provide opportunity to discuss scientific applications that would benefit from the use of WS-PGRADE to access EGI cloud and grid resources.

Training Material

Additional Information


Hadoop on the EGI Federated Cloud

With the rapid increase of data volumes in scientific computations, the importance of utilising parallel and distributed computing paradigms in data processing is becoming more and more important. Hadoop is an open source implementation of the MapReduce framework supporting processing large datasets in parallel and on multiple nodes in a reliable and fault-tolerant manner. Scientific workflow systems and science gateways are high level environments to facilitate the development, orchestration and execution of complex experiments from a user-friendly graphical user interface. Integrating MapReduce/Hadoop with such workflow systems and science gateways enables scientists to conduct complex data intensive experiments utilising the power of the MapReduce paradigm from the convenience provided by science gateway frameworks.

Training Material

  • Hadoop on the EGI Federated Cloud. This training is given by Tamas Kiss (University of Westminster, London, UK), and Carlos Blanco (University of Cantabria). 17 March 2016, during Webinar: Hadoop on the EGI Federated Cloud. This presentation and demonstration illustrates how easily Hadoop clusters can be deployed on EGI FedCloud resources, Hadoop applications can be executed on these clusters, and finally resources can be released after execution. Users of the EGI FedCloud WS-PGRADE gateway can import and parameterise pre-prepared workflows for the above tasks published in a public workflow repository. Users can set the type/flavour and number of desired nodes in the Hadoop cluster, select the target EGI FedCloud site, and define the Hadoop executable and the desired data source and destination. All three functionality (create Hadoop cluster, execute Hadoop job, destroy Hadoop cluster) can be executed as a standalone job or can be combined into more complex workflows automating different user scenarios.

User Manual