This wiki page provides training materials and service documentations that help end users better understand EGI (and third-party's) services. You can ask questions and discuss a particular service at: https://community.egi.eu/
The EGI Federated Cloud is a standards-based, open cloud system that federates institutional clouds to offer a scalable computing platform for data and/or compute driven applications and services in research and science.
- Research Computational Infrastructure: Cloud infrastructures, [ppt]. This training was given by Giuseppe La Rocca & Gergely Sipos, 16-17 Aug 2018, during CODATA-RDA Research Data Science Summer Schoo 2018l. It provides up-to-dated information about EGI cloud computing and EGI services (AppDB, Jupyter Notebook), including demo and exercise. It also gives an overview about EOSC and EOSC-hub project.
- Research Computational Infrastructures: Cloud computing & the EGI Cloud [ppt]. This training was given by Giuseppe La Rocca & Gergely Sipos, 10-21 Jul 2017, during CODATA-RDA Research Data Science Summer School 2017. It covers topics including EGI Cloud, contextualisation and data management. Exercises are included.
- Using the EGI Fed-Cloud for Data Analysis, [pdf]. This training was given by Giuseppe La Rocca, 03-07 Jul 2017, during the EUDAT Summer School, FORTH, Heraklion, Crete, Greece. It gives an overview on EGI services with focuses on Cloud Computing, covering topics such as block storage and object storage. It also includes topics how to link EUDAT services with EGI FedCloud.
- Introduction and command line usage of the EGI Federated Cloud [ppt]. This training material is provided by Gergely Sipos, 05 Oct 2016. The goals of the training are to help users to learn the concept of IaaS Cloud Computing, conceptual model of the EGI federated Cloud, to obtain skills in using the standard interface of the EGI federated cloud and become an active user. Hand-on exercises are included.
- The EGI Federated Cloud for application developers (API access), [ppt]. This training is jointly given by Giuseppe La Rocca, Enol Fernandez del Castillo, and Boris Parak, 28 Sep 2016, during DI4R 2016. The topics covers EGI and EGI Federated Cloud, Access the training infrastructure. It also includes hand-on exercises.
- https://wiki.egi.eu/wiki/Federated_Cloud_user_support provides detailed information about the service and guideline of access the service
- The EGI Federated Cloud e-Infrastructure (2015), Enol Fernández-del-Castillo, Diego Scardaci, Álvaro López García, in the proceeding HOLACONF-Cloud Forward: From Distributed to Complete Computing, 6-8 October 2015, Scuola Normale Superiore, Pisa. doi: 10.1016/j.procs.2015.09.235
- The user support programme and the training infrastructure of the EGI Federated Cloud (2015), Fernandez, E., Scardaci, D., Sipos, G., Chen, Y., Wallom, D, in the proceeding of High Performance Computing & Simulation (HPCS) 2015, Amsterdam, the Netherlands. doi: 10.1109/HPCSim.2015.7237016
Cloud Container Compute
The Cloud Container Compute offers guaranteed computational resources in a secure and isolated environment with standard API access, without the overhead of managing the operating system.
- Training Material. [ppt]. This training is given by Enol Fernandez, 17 Jul 2018, during EOSC-hub tech-talk: Cloud, containers, orchestration. It introduces the service architecture and interfaces, options and attributes.
- https://wiki.egi.eu/wiki/Federated_Cloud_Containers: provides information about the docker image and configurations details.
Application on Demand
Application on Demand service allows user-friendly access to a portfolio of applications and application hosting frameworks (Science Gateways, VREs) that are configured to use the dedicated pool of cloud computing and HTC clusters from EGI. The service also allows users to run their own simulation/analysis models with custom input data.
- Overview of the EGI Applications on Demand (AoD) service. [ppt]. This training is provided by Gergely Sipos and Giuseppe La Rocca, 13 Jul 2017, during Webinar: EGI Applications On Demand Service. It gives an overview of the service components, information details on how to access, available services provided via the framework. An introductory webinar (video recording) can be found at: http://connect.ct.infn.it/p8w34jegx9s/
- https://wiki.egi.eu/wiki/AoD is the wiki page of the service providing information details on how to access the service.
- EGI Applications on Demand Service[pdf]. (2017), Gergely Sipos, Giuseppe La Rocca. In the proceeding of IWSG'17 conference
The EGI DataHub is a service prototype designed to make data discoverable and available in an easy way across all EGI federated resources.
- The EGI Open Data Platform and DataHub[ppt]. This training is given by Diego Scardaci during DI4R 2016. It introduces what is the EGI DataHub, together with 4 hand-on exercise.
- The EGI Open Data Platform [ppt]. This training is given by Lukasz Dutka (CYFRONET) and Matthew Viljoen (EGI.eu), 17 May 2016, during the webinar The EGI Open Data Platform - Towards Scientific Data Hubs. The topics cover OneData, EGI DataHub and EGI Open Data Platform (ODP). A video recording of this webinar can be found at http://connect.ct.infn.it/p68qfn1h2km/
- User Guide provides information for end users
- Admin Guide provides information for administrators
- Towards European Open Science Commons: The EGI Open Data Platform and The EGI DataHub, (2016), Matthew Viljoena, Lukasz Dutka, Bartosz Kryza, Yin Chen, Cloud Forward 2018, Madrid, Oct. 2016.
- DataHub google drive folder provides documentation and presentation slides about the service
- EGI DataHub is a documentation repository providing many technology details
- EGI community forum for DataHub discussion includes many discussions
EGI Check-In is a proxy service that operates as a central hub to connect federated Identity Providers (IdPs) with EGI service providers. Check-in allows users to select their preferred IdP so that they can access and use EGI services in a uniform and easy way.
- Community AAI with Check-In [ppt]. This training is given by Nicolas Liampotis, 15 Jun 2018, during EOSC-hub tech-talk: AAI webinar. This training introduces Check-in in a nutshell, users and groups management, and service options.
- AAI usage guide: A wiki page contains information about registering an account with the EGI AAI in order to access a variety of EGI tools and services using the same account.
- AAI guide for IdPs: A wiki page contains information about integrating your identity provider with the EGI AAI Proxy in order to allow users in your community to access EGI tools and services.
- AAI guide_for SPs: A wiki page contains information about connecting services to the EGI AAI Check-in service in order to allow user login through Check-in and to receive users' attributes.
- EGI Check-In Service: provide general information about the service.
EGI Archive storage
- SRM manual: the scripts command line used by the service.
EGI Jupyter Notebook
EGI Jupyter Notebook is an 'as a Service' environment based on the Jupyter technology, offering a browser-based, scalable tool for interactive data analysis. The EGI Notebooks environment provides users with notebooks where they can combine text, mathematics, computations and rich media output. EGI Notebooks is a multi-user service and can scale to multiple servers based on the EGI cloud service.
- Turoial: Reproducible Open Science with EGI Notebooks, Binder and Zenodo. This training is given by Enol Fernandez and Gergely Sipos during ISGC 2019 conference, 31 Mar - 5 Apr 2019, Taipei. The tutorial goes through the main features of the EGI Notebooks service and shows how to use it with Binder for Open Science. Hands-on exercises is provided including Python based on real applications from the environmental sciences domain. The integrated use of Notebooks with EGI’s DataHub service is also demonstrated. Communities or national e-infrastructures who would like to setup a similar service locally can learn about how to do this.
- Webinar: The EGI Notebooks for interactive analysis of data using EGI storage and compute services. This training is delivered by Giuseppe La Rocca and Enol Fernandez, 19 Mar 2019. It includes a presentation that gives an overview of the service and a live demo that gives a tour guide of the service. The recording is available: http://connect.ct.infn.it/p59n3ojbyqn
- The EGI Jupyter Notebook This training is delivered by Giuseppe La Rocca, 10 Jul 2018, during ENVRIplus International Summer School
Online Storage allows you to store data in a reliable and high-quality environment and share it across distributed teams. Your data can be accessed through different standard protocols and can be replicated across different providers to increase fault-tolerance. Online Storage gives you complete control over the data you share and with whom
- EGI Online Storage [ppt]. This information is provided by Vincenzo Spinoso, 17 Sep 2018. It gives an overview of the service, service components, access interfaces, and examples of usage.
- https://wiki.egi.eu/wiki/Online_Storage : A wiki page provides detailed information about the service.
EGI Workload Manager (also known as DIRAC4EGI) is a service is provided to the EGI community as a workload management service used to distribute the users' computing tasks among the available resources both HTC and cloud.
- Training materials delivered by Andrei Tsaregorodtsev during the 9th DIRAC users' meeting, May 2019, London, providing updated information about the service.
- Trainings on DIRAC workload and data management system is delivered by Andrei Tsaregorodtsev (CNRS, CPPM) during Virgo Computing School, 8th Nov 2018, Cascina(Pisa), Italy. The topics covers the following aspects:
- Distributed Infrastructure for Scientific Application is an overview of DIRAC framework
- Getting started introduces the training plan and setups
- Client installation introduces DIRAC client software installation
- Job Management introduces how to submit jobs via DIRAC web portal and command lines
- Mandelbrot quest is a hand-on exercise to use DIRAC tool to explore the mandelbrot algorithm and images
- Managing Data introduces DIRAC Data Management system
- Bulk job submission introduces how to submit multiple jobs
- Job with input and output data introduces how to input and output data from jobs
- DIRAC system for EGI communities This training is provided by Andrei Tsaregorodtsev (CNRS), 07 Jun 2016, during Webinar: DIRAC system for EGI communities. It introduces the DIRAC setup for EGI, Workload manager, File Catalogue and web portal interface. The recording of the webinar is available at: http://connect.ct.infn.it/p3tj8sthvf0/
- https://wiki.egi.eu/wiki/Workload_Manager: A wiki page provide detailed information about service and guide for getting access.
Data Transfer allows you to move any type of data files asynchronously from one place to another. The service includes dedicated interfaces to display statistics of on-going transfers and manage network resources.
- EGI Data Transfer Service gives an overview introduction of the service, provide information for users to get started with the service.
- EGI Data Transfer wiki is the place to learn more about the service
- https://webfts.cern.ch/ (See Virtual Tour on the page)
The EGI Marketplace provides an online platform for exposing and promoting services to new scientific and market segments. The Marketplace provides all the necessary functionalities for bringing together offering and demand to make research happen. All services are provided by the EGI Federation and partner institutions.
- A YouTube video is given by Björn Backeberg, on 6th Dec 2018, that step-by-step guides users to work through EGI Marketplace, introducing how to search and place an order of a service.
3rd party services (not part of the EGI Service Catalogue)
EGI VMOps dashboard
The EGI Application Database (AppDB) has recently evolved its functionalities from its currently catalogue of applications and virtual machines (VMs) to include a Graphical User Interface (GUI) allowing authorized users to perform basic VM management operations.
- AppDB VMOps Dashboard: A graphical portal to manage applications and services on federated cloud infrastructures. This training is given by William Vassilis Karageorgos (GRNET), 24 Nov 2017, during Webinar: AppDB VMOps Dashboard: A graphical portal to manage applications and services on federated cloud infrastructures. It introduces the key capabilities of the VMOps dashboard, particularly:
- A wizard-like builder that guides users through the selection of virtual machine images, virtual appliances, cloud resources and contextualization scripts to deploy complex applications or services in federated clouds.
- Graphical interfaces and tools to monitor and manage your applications/services in federated clouds, independently of underlying cloud technologies.
- A scalable architecture composed of a generic front-end and several, technology-specific back-ends for load balancing, and with a RESTful API to integrate with 3rd party services.
The EGI Applications Database is developed and operated by the Institute of Accelerating Systems and Applications (IASA) on behalf of the EGI Foundation as an EGI Core Service.
The Webinar was recorded. You can watch the recording at: http://connect.ct.infn.it/p5cty7tor87/
WS-PGRADE is an open source, Liferay-based web portal environment for the design and execution of compute and data intensive applications on grid and cloud platforms. Applications can be defined in the form of data-driven workflows in WS-PGRADE. Steps within a workflow can be executable applications, Virtual Machine images, or online (Web) Services. These computational steps are connected through data-dependencies within the workflow: The output of one computation can serve as input of the next one. WS-PRADE resolves data dependencies and manages the computational operations on the infrastructure resources. The back-end system of WS-PGRADE is compatible with various academic and commercial grid and cloud systems, including federated clouds and gLite clusters of EGI. Moreover, scientific communities can connect workflow-specific Web interfaces to WS-PGRADE to simplify the execution of workflows for researchers. This is enabled by the 'Remote API' component of WS-PGRADE, a programming interface that exposes the workflow registry and the workflow manager service to third party clients. During the webinar the main capabilities of WS-PGRADE will be presented and demonstrated. The event will also provide opportunity to discuss scientific applications that would benefit from the use of WS-PGRADE to access EGI cloud and grid resources.
- Workflow applications on EGI with WS-PGRADE This training is given by Peter Kacsuk (MTA SZTAKI), 26 May 2016, during Webinar: Workflow applications on EGI with WS-PGRADE. A recording of the training can be found at http://connect.ct.infn.it/p6gvxozwgg4/
Hadoop on the EGI Federated Cloud
With the rapid increase of data volumes in scientific computations, the importance of utilising parallel and distributed computing paradigms in data processing is becoming more and more important. Hadoop is an open source implementation of the MapReduce framework supporting processing large datasets in parallel and on multiple nodes in a reliable and fault-tolerant manner. Scientific workflow systems and science gateways are high level environments to facilitate the development, orchestration and execution of complex experiments from a user-friendly graphical user interface. Integrating MapReduce/Hadoop with such workflow systems and science gateways enables scientists to conduct complex data intensive experiments utilising the power of the MapReduce paradigm from the convenience provided by science gateway frameworks.
- Hadoop on the EGI Federated Cloud. This training is given by Tamas Kiss (University of Westminster, London, UK), and Carlos Blanco (University of Cantabria). 17 March 2016, during Webinar: Hadoop on the EGI Federated Cloud. This presentation and demonstration illustrates how easily Hadoop clusters can be deployed on EGI FedCloud resources, Hadoop applications can be executed on these clusters, and finally resources can be released after execution. Users of the EGI FedCloud WS-PGRADE gateway can import and parameterise pre-prepared workflows for the above tasks published in a public workflow repository. Users can set the type/flavour and number of desired nodes in the Hadoop cluster, select the target EGI FedCloud site, and define the Hadoop executable and the desired data source and destination. All three functionality (create Hadoop cluster, execute Hadoop job, destroy Hadoop cluster) can be executed as a standalone job or can be combined into more complex workflows automating different user scenarios.