ServiceGuides

This wiki page provides training materials and service documentations that help end users better understand EGI (and third-party's) services.

EGI Services

Cloud Computing

The Federated Cloud is a standards-based, open cloud system that federates institutional clouds to offer a scalable computing platform for data and/or compute driven applications and services in research and science.

Training Materials

• Research Computational Infrastructure: Cloud infrastructures, [ppt]. This training was given by Giuseppe La Rocca & Gergely Sipos, 16-17 Aug 2018, during CODATA-RDA Research Data Science Summer Schoo 2018l. It provides up-to-dated information about EGI cloud computing and EGI services (AppDB, Jupyter Notebook), including demo and exercise. It also gives an overview about EOSC and EOSC-hub project.

• Research Computational Infrastructures: Cloud computing & the EGI Cloud [ppt]. This training was given by Giuseppe La Rocca & Gergely Sipos, 10-21 Jul 2017, during CODATA-RDA Research Data Science Summer School 2017. It covers topics including EGI Cloud, contextualisation and data management. Exercises are included.

• Using the EGI Fed-Cloud for Data Analysis, [pdf]. This training was given by Giuseppe La Rocca, 03-07 Jul 2017, during the EUDAT Summer School, FORTH, Heraklion, Crete, Greece. It gives an overview on EGI services with focuses on Cloud Computing, covering topics such as block storage and object storage. It also includes topics how to link EUDAT services with EGI FedCloud.

• Introduction and command line usage of the EGI Federated Cloud [ppt]. This training material is provided by Gergely Sipos, 05 Oct 2016. The goals of the training are to help users to learn the concept of IaaS Cloud Computing, conceptual model of the EGI federated Cloud, to obtain skills in using the standard interface of the EGI federated cloud and become an active user. Hand-on exercises are included.

• The EGI Federated Cloud for application developers (API access), [ppt]. This training is jointly given by Giuseppe La Rocca, Enol Fernandez del Castillo, and Boris Parak, 28 Sep 2016, during DI4R 2016. The topics covers EGI and EGI Federated Cloud, Access the training infrastructure. It also includes hand-on exercises.

User Manual

• https://wiki.egi.eu/wiki/Federated_Cloud_user_support provides detailed information about the service and guideline of access the service

Publications

• The EGI Federated Cloud e-Infrastructure (2015), Enol Fernández-del-Castillo, Diego Scardaci, Álvaro López García, in the proceeding HOLACONF-Cloud Forward: From Distributed to Complete Computing, 6-8 October 2015, Scuola Normale Superiore, Pisa. doi: 10.1016/j.procs.2015.09.235

• The user support programme and the training infrastructure of the EGI Federated Cloud (2015), Fernandez, E., Scardaci, D., Sipos, G., Chen, Y., Wallom, D, in the proceeding of High Performance Computing & Simulation (HPCS) 2015, Amsterdam, the Netherlands. doi: 10.1109/HPCSim.2015.7237016

Cloud Container Compute

The Cloud Container Compute offers guaranteed computational resources in a secure and isolated environment with standard API access, without the overhead of managing the operating system.

Training Material

• Training Material. [ppt]. This training is given by Enol Fernandez, 17 Jul 2018, during EOSC-hub tech-talk: Cloud, containers, orchestration. It introduces the service architecture and interfaces, options and attributes.

User Manual

• https://wiki.egi.eu/wiki/Federated_Cloud_Containers: provides information about the docker image and configurations details.

Application on Demand

Application on Demand service allows user-friendly access to a portfolio of applications and application hosting frameworks (Science Gateways, VREs) that are configured to use the dedicated pool of cloud computing and HTC clusters from EGI. The service also allows users to run their own simulation/analysis models with custom input data.

Training Material

• Overview of the EGI Applications on Demand (AoD) service. [ppt]. This training is provided by Gergely Sipos and Giuseppe La Rocca, 13 Jul 2017, during Webinar: EGI Applications On Demand Service. It gives an overview of the service components, information details on how to access, available services provided via the framework. An introductory webinar (video recording) can be found at: http://connect.ct.infn.it/p8w34jegx9s/

User Manual

• https://wiki.egi.eu/wiki/AoD is the wiki page of the service providing information details on how to access the service.

Publication

• EGI Applications on Demand Service[pdf]. (2017), Gergely Sipos, Giuseppe La Rocca. In the proceeding of IWSG'17 conference

DataHub

The EGI DataHub is a service prototype designed to make data discoverable and available in an easy way across all EGI federated resources.

Training Materials

• The EGI Open Data Platform and DataHub[ppt]. This training is given by Diego Scardaci during DI4R 2016. It introduces what is the EGI DataHub, together with 4 hand-on exercise.

• The EGI Open Data Platform [ppt]. This training is given by Lukasz Dutka (CYFRONET) and Matthew Viljoen (EGI.eu), 17 May 2016, during the webinar The EGI Open Data Platform - Towards Scientific Data Hubs. The topics cover OneData, EGI DataHub and EGI Open Data Platform (ODP). A video recording of this webinar can be found at http://connect.ct.infn.it/p68qfn1h2km/

User Manuals

• User Guide provides information for end users
• Admin Guide provides information for administrators

Publication

• Towards European Open Science Commons: The EGI Open Data Platform and The EGI DataHub, (2016), Matthew Viljoena, Lukasz Dutka, Bartosz Kryza, Yin Chen, Cloud Forward 2018, Madrid, Oct. 2016.

Check-In

EGI Check-In is a proxy service that operates as a central hub to connect federated Identity Providers (IdPs) with EGI service providers. Check-in allows users to select their preferred IdP so that they can access and use EGI services in a uniform and easy way.

Training Material

• Community AAI with Check-In [ppt]. This training is given by Nicolas Liampotis, 15 Jun 2018, during EOSC-hub tech-talk: AAI webinar. This training introduces Check-in in a nutshell, users and groups management, and service options.

User Manuals

• AAI usage guide: A wiki page contains information about registering an account with the EGI AAI in order to access a variety of EGI tools and services using the same account.
• AAI guide for IdPs: A wiki page contains information about integrating your identity provider with the EGI AAI Proxy in order to allow users in your community to access EGI tools and services.
• AAI guide_for SPs: A wiki page contains information about connecting services to the EGI AAI Check-in service in order to allow user login through Check-in and to receive users' attributes.

Additional Information

• EGI Check-In Service: provide general information about the service.

EGI Jupyter Notebook

EGI Jupyter Notebook is an 'as a Service' environment based on the Jupyter technology, offering a browser-based, scalable tool for interactive data analysis. The EGI Notebooks environment provides users with notebooks where they can combine text, mathematics, computations and rich media output. EGI Notebooks is a multi-user service and can scale to multiple servers based on the EGI cloud service.

Training Materials

• The EGI Jupyter Notebook This training is delivered by Giuseppe La Rocca, 10 Jul 2018, during ENVRIplus International Summer School

Online Storage

Online Storage allows you to store data in a reliable and high-quality environment and share it across distributed teams. Your data can be accessed through different standard protocols and can be replicated across different providers to increase fault-tolerance. Online Storage gives you complete control over the data you share and with whom

Training Material

• EGI Online Storage [ppt]. This information is provided by Vincenzo Spinoso, 17 Sep 2018. It gives an overview of the service, service components, access interfaces, and examples of usage.

User Manuals

• https://wiki.egi.eu/wiki/Online_Storage : A wiki page provides detailed information about the service.

Workload Manager

EGI Workload Manager (also known as DIRAC4EGI) is a service is provided to the EGI community as a workload management service used to distribute the users' computing tasks among the available resources both HTC and cloud.

Training Material

• DIRAC system for EGI communities This training is provided by Andrei Tsaregorodtsev (CNRS), 07 Jun 2016, during Webinar: DIRAC system for EGI communities. It introduces the DIRAC setup for EGI, Workload manager, File Catalogue and web portal interface. The recording of the webinar is available at: http://connect.ct.infn.it/p3tj8sthvf0/

User Manuals

• https://wiki.egi.eu/wiki/Workload_Manager: A wiki page provide detailed information about service and guide for getting access.

3rd party services (not part of the EGI Service Catalogue)

EGI VMOps dashboard

The EGI Application Database (AppDB) has recently evolved its functionalities from its currently catalogue of applications and virtual machines (VMs) to include a Graphical User Interface (GUI) allowing authorized users to perform basic VM management operations.

Training Material

• AppDB VMOps Dashboard: A graphical portal to manage applications and services on federated cloud infrastructures. This training is given by William Vassilis Karageorgos (GRNET), 24 Nov 2017, during Webinar: AppDB VMOps Dashboard: A graphical portal to manage applications and services on federated cloud infrastructures. It introduces the key capabilities of the VMOps dashboard, particularly:
  • A wizard-like builder that guides users through the selection of virtual machine images, virtual appliances, cloud resources and contextualization scripts to deploy complex applications or services in federated clouds.
  • Graphical interfaces and tools to monitor and manage your applications/services in federated clouds, independently of underlying cloud technologies.
  • A scalable architecture composed of a generic front-end and several, technology-specific back-ends for load balancing, and with a RESTful API to integrate with 3rd party services.

The EGI Applications Database is developed and operated by the Institute of Accelerating Systems and Applications (IASA) on behalf of the EGI Foundation as an EGI Core Service.

The Webinar was recorded. You can watch the recording at: http://connect.ct.infn.it/p5cty7tor87/

Additional Information

• https://wiki.egi.eu/wiki/Federated_Cloud_AppDB_VMOps_Dashboard

WS-PGRADE

WS-PGRADE is an open source, Liferay-based web portal environment for the design and execution of compute and data intensive applications on grid and cloud platforms. Applications can be defined in the form of data-driven workflows in WS-PGRADE. Steps within a workflow can be executable applications, Virtual Machine images, or online (Web) Services. These computational steps are connected through data-dependencies within the workflow: The output of one computation can serve as input of the next one. WS-PRADE resolves data dependencies and manages the computational operations on the infrastructure resources. The back-end system of WS-PGRADE is compatible with various academic and commercial grid and cloud systems, including federated clouds and gLite clusters of EGI. Moreover, scientific communities can connect workflow-specific Web interfaces to WS-PGRADE to simplify the execution of workflows for researchers. This is enabled by the 'Remote API' component of WS-PGRADE, a programming interface that exposes the workflow registry and the workflow manager service to third party clients. During the webinar the main capabilities of WS-PGRADE will be presented and demonstrated. The event will also provide opportunity to discuss scientific applications that would benefit from the use of WS-PGRADE to access EGI cloud and grid resources.

Training Material

• Workflow applications on EGI with WS-PGRADE This training is given by Peter Kacsuk (MTA SZTAKI), 26 May 2016, during Webinar: Workflow applications on EGI with WS-PGRADE. A recording of the training can be found at http://connect.ct.infn.it/p6gvxozwgg4/

Additional Information

• http://wspgrade.hu

Hadoop on the EGI Federated Cloud

With the rapid increase of data volumes in scientific computations, the importance of utilising parallel and distributed computing paradigms in data processing is becoming more and more important. Hadoop is an open source implementation of the MapReduce framework supporting processing large datasets in parallel and on multiple nodes in a reliable and fault-tolerant manner. Scientific workflow systems and science gateways are high level environments to facilitate the development, orchestration and execution of complex experiments from a user-friendly graphical user interface. Integrating MapReduce/Hadoop with such workflow systems and science gateways enables scientists to conduct complex data intensive experiments utilising the power of the MapReduce paradigm from the convenience provided by science gateway frameworks.

Training Material

• Hadoop on the EGI Federated Cloud. This training is given by Tamas Kiss (University of Westminster, London, UK), and Carlos Blanco (University of Cantabria). 17 March 2016, during Webinar: Hadoop on the EGI Federated Cloud. This presentation and demonstration illustrates how easily Hadoop clusters can be deployed on EGI FedCloud resources, Hadoop applications can be executed on these clusters, and finally resources can be released after execution. Users of the EGI FedCloud WS-PGRADE gateway can import and parameterise pre-prepared workflows for the above tasks published in a public workflow repository. Users can set the type/flavour and number of desired nodes in the Hadoop cluster, select the target EGI FedCloud site, and define the Hadoop executable and the desired data source and destination. All three functionality (create Hadoop cluster, execute Hadoop job, destroy Hadoop cluster) can be executed as a standalone job or can be combined into more complex workflows automating different user scenarios.

User Manual

• Hadoop-EGI user manual v1.3