Alert.png The wiki is deprecated and due to be decommissioned by the end of September 2022.
The content is being migrated to other supports, new updates will be ignored and lost.
If needed you can get in touch with EGI SDIS team using operations @ egi.eu.

Difference between revisions of "Training infrastructure"

From EGIWiki
Jump to navigation Jump to search
 
(37 intermediate revisions by 4 users not shown)
Line 3: Line 3:
= Introduction to the EGI training infrastructure =
= Introduction to the EGI training infrastructure =


During 2015 the EGI community has established a cloud based e-infrastructure for training, under the coordination of the EGI-Engage project. This training infrastructure is hosted as a dedicated resource pool, a Virtual Organisation (VO), on the EGI Federated Cloud infrastructure. The infrastructure provides resources and services for face-to-face events, online training courses (Webinars, MOOCs) or self-paced learning modules.  
During 2015, under the coordination of the EGI-Engage project, the EGI community established a cloud based e-infrastructure specifically for training. The training infrastructure exists as a resource pool - a so called Virtual Organisation (VO) - within the [Federated_Cloud_user_support|EGI Federated Cloud infrastructure]. The training infrastructure provides ‘Infrastructure as a Service’ clouds (IaaS), i.e. physical resources (CPU, storage), as well as access services (login, application catalogue and application management portal) for face-to-face events, online training courses or self-paced learning modules.
The training infrastructure is integrated with the EGI authentication-authorisation service (CheckIn) that allows trainers to generate short-living user accounts for training participants. Such accounts can identify students individually, and for a limited lifetime - typically few hours or days, depending on the length of the training event - allow them to interact with the training infrastructure sites and services.


The training VO is integrated with the ‘per-user proxy factory’ solution of EGI to generate personal, but short-living proxy certificates in an easy way, for trainers and trainees. Such proxy certificates can identify students individually, and have limited lifetime - typically few hours or days, depending on the length of the training event.
The infrastructure currently includes enough capacity to scale up to class-room size audiences, approximately up to 100 participants.


The infrastructure can be extended with customised training environments on-demand. These environments can be implemented and deployed on the infrastructure in the form of Virtual Machine images.
Introductory slides and poster of the Training Infrastructure: https://indico.egi.eu/indico/contributionDisplay.py?contribId=122&confId=2544
(These were prepared for a Demo at the EGI Community Forum event in Bari, 2015 November. Outdated in some parts.)


Introductory slides and poster of the Training Infrastructure: https://indico.egi.eu/indico/contributionDisplay.py?contribId=122&confId=2544
= Usage models  =
(These were prepared for a Demo at the EGI Community Forum event in Bari, 2015 November)
 
The training infrastructure is suitable for two types of courses:


'''Do you want to deploy a custom training environment on this infrastructure? Please email support@egi.eu.'''
# Cloud computing courses: Such courses teach students about IaaS clouds and the way Virtual Machine images, storage blocks and other types of ‘low level’ resources are managed. For such courses In this operational mode the infrastructure the trainer does not have to deploy applications or online services on the infrastructure in advance of the course. The applications/services will be deployed by the students themselves as training exercises. Such courses typically target programmers or other rather technical members of scientific communities or projects.
# Scientific courses: Such courses teach scientists or programmers about a specific software that’s relevant for their work. For example a specific gene sequence analysis software suite, an earthquake visualisation tool, a data processing pipeline. In this operational mode the the trainer deploys the domain specific application/tool on the training infrastructure before the training and the students interact directly with those applications/tools without even knowing where those are deployed and running. Depending on how computationally or data heavy the student exercises are, multiple students can share a single software deployment, or each student can have his/her own dedicated deployment. The configuration can be controlled by the teacher when the setup is deployed. have dedicated training environments, and the community can benefit from the easy deployment, predictability and repeatability of courses.


= Usage models  =
In both operational case the deployment of applications/tools/services can happen in the form of ‘Virtual Machine images’ (VMIs), and block storages - the latter basically behaving like a virtual USB drive that can be attached/detached to VMIs to provide data and storage space for applications.


The goals and possible usage modes of the EGI training infrastructure are:  
VMIs can be deployed on the infrastructure through the EGI Applications Marketplace (https://appdb.egi.eu/browse/cloud), a growing catalogue that includes both basic applications (e.g. latest version of clean Linux deployments) and more specialised applications (e.g. Jupyter Notebook). A specific section within the catalogue is reserved for training-related application VMIs (https://appdb.egi.eu/store/vo/training.egi.eu).


#Provide an infrastructure that demonstrates services of the EGI federated cloud. In this operational mode the infrastructure can accommodate courses that focus on the usage of the EGI cloud services themselves. Such courses typically target programmers or other technical members of scientific communities or projects.  
The Applications Marketplace also includes a VMI management dashboard (https://dashboard.appdb.egi.eu), offering a Web interface for both trainers and students to deploy and manage VMIs from the Marketplace on the training infrastructure.
#Offer a baseline cloud infrastructure for training courses about scientific software and services. In this operational mode the representatives of the community (the trainers) deploy custom Virtual Machine images on the training infrastructure before the training, and these images offer the training environment for the students. Because of the cloud-based operational model the students can have dedicated training environments, and the community can benefit from the easy deployment, predictability and repeatability of courses.


= Available resources  =
= Available resources  =
Line 29: Line 32:
'''Available clouds:'''  
'''Available clouds:'''  


*CESNET OpenNebula cloud site (Czech Republic)  
*CESNET-MCC - OpenStack cloud site (Czech Republic)  
**64 vCPUs, 110 GB of RAM, 1 TB of persistent storage  
**64 vCPUs, 110 GB of RAM, 1 TB of persistent storage  
**Contact: Miroslav Ruda <ruda@ics.muni.cz>, Boris Parak <parak@cesnet.cz>
**Contact: Miroslav Ruda <ruda@ics.muni.cz>, Boris Parak <parak@cesnet.cz>


*INFN-CATANIA-STACK OpenStack cloud site (Italy)  
*INFN-CATANIA-STACK - OpenStack cloud site (Italy)  
**20 vCPUs, 50 GB of RAM, 1TB of storage volume, 10 public IPs
**20 vCPUs, 50 GB of RAM, 1TB of storage volume  
**Contact: roberto.barbera@ct.infn.it, sg-licence@ct.infn.it
**Contact: roberto.barbera@ct.infn.it, giuseppe.platania@ct.infn.it, sg-licence@ct.infn.it
 
*IISAS-GPUCloud - OpenStack cloud site (Slovakia)
**12 vCPU cores, 2 GPU and 48GB of RAM
*** In case of special request, these resources can be doubled but must be reserved in advance
**Contact: viet.ui@savba.sk, cloud-admin@savba.sk


'''Clouds under maintenance:'''
*IISAS-FedCloud - OpenStack cloud site (Slovakia)
**16 vCPU cores with 1.6GB of RAM per core
**Contact: viet.ui@savba.sk, cloud-admin@savba.sk


*UKIM OpenNebula cloud site (Macedonia)  
* IFCA-LCG2 OpenStack (Spain)
**48 vCPUs, 48 GB of RAM, 48 public IP addresses  
**20 vCPU cores, 50GB of RAM in total, 1TB of block storage, 1 public IP address
**Contact: Boro Jakimovski <boro.jakimovski@finki.ukim.mk>
**Additional IP addresses can be provided, upon request, for the specific training event
**Contact: aloga@ifca.unican.es
 
'''Cloud not anymore available:'''


*BIFI OpenStack cloud site (Spain)  
*BIFI OpenStack cloud site (Spain)  
**100 vCPUs, 200 GB of RAM, 1TB for storage volumes, 50 public IP addresses  
**100 vCPUs, 200 GB of RAM, 1TB for storage volumes, 50 public IP addresses  
**Contact: Ruben Valles <rvalles@bifi.es>
**Contact: Ruben Valles <rvalles@bifi.es>
*CETA-CIEMAT OpenStack cloud site (Spain)
**20 physical CPU cores (or virtual cores), 40 GB of RAM, 5.4TB storage shared among all the VOs (CDMI is not provided), 20 public IPs
**Contact: miguelangel.diaz@externos.ciemat.es, admin.iaas@ceta-ciemat.es


'''Documentation:'''
'''Documentation:'''


* User manual (for trainers): The training infrastructure is one of the Virtual Organisations of the EGI Federated cloud. The Federated Cloud user guide is applicable for usage: https://wiki.egi.eu/wiki/Federated_Cloud_user_support
* [https://documents.egi.eu/document/2768 Operation Level Agreements with the sites]
* [https://documents.egi.eu/document/2768 Operation Level Agreements with the sites]
*VO sites and endorsed Virtual Machine images in AppDB: https://appdb.egi.eu/store/vo/training.egi.eu  
*VO sites and endorsed Virtual Machine images in AppDB: https://appdb.egi.eu/store/vo/training.egi.eu  
Line 81: Line 91:
* EGI training during ENVRIplus week in Prague (14-18 November 2016). Training materials from the event: https://documents.egi.eu/document/3005
* EGI training during ENVRIplus week in Prague (14-18 November 2016). Training materials from the event: https://documents.egi.eu/document/3005
* MEDGENET-Workshop INAB, Thessaloniki, Greece (15 December 2016). Training materials from the event: http://tinyurl.com/medgenet
* MEDGENET-Workshop INAB, Thessaloniki, Greece (15 December 2016). Training materials from the event: http://tinyurl.com/medgenet
* Cloud Tutorial at EUDAT summer School, Heraklion, Crete (03-07/07/2017). Training material from the event: https://www.eudat.eu/eudat-summer-school
* CODATA-RDA Research Data Science Summer School, Trieste (21 July 2017). Training materials from the event: https://documents.egi.eu/document/3168
* CODATA-RDA Research Data Science Summer School, Trieste (21 July 2017). Training materials from the event: https://documents.egi.eu/document/3168
* Scipion tutorial on Cloud. Training materials from the event: http://i2pc.es/instruct-course-on-image-processing-for-electron-microscopy-in-the-cloud-madrid-january-17-19-2018/
* International Summer School “Data Management in Environmental & Earth Science Infrastructures: Theory & Practice”, Lecce (09-13 July 2019). Training info and material from the event: https://documents.egi.eu/public/ShowDocument?docid=3341
* CODATA-RDA Research Data Science Summer School, Trieste (17 Aug 2018). Training materials from the event: https://documents.egi.eu/public/ShowDocument?docid=3349
* NGSchool 2018. Training info and materials from the event: https://ngschool.eu/
* 3rd Int'l Summer School on Data Science (SSDS 2018). Training info and materials from the event: https://sites.google.com/site/ssdatascience2018/
* Training for PhD students at the University of Genoa (04 June 2019). Training info and materials from the event: http://dottorato.dicca.unige.it/eng/schede_corsi_2019/eScience%20new%20information%20technologies%20for%20research.pdf
* Introduction to Jupyter and Open Science - Training, Yerevan (27 September 2019). Training materials from the event: http://go.egi.eu/eapec2019
* HPC graduate class (August - November 2019) at UNICAM (Brazil), 15 meetings
* NGSChool 2019 (October, 24-31). Agenda page: https://ngschool.eu/ngschool2019/
* Open Science with Jupyter, Zenodo and Binder (tutorial), 04 Dec. 2019. Training info and materials from the event: http://go.egi.eu/bigdata2019
* Hercules European School (April 2020). Agenda page: http://hercules-school.eu/
= Advanced Usage =
To configure an HTC clusters on top of IaaS infastructure, please check the following
[[Deploy HTC clusters on top of IaaS infrastructure | Deploy HTC clusters on top of IaaS infrastructure ]] guide.


= Booking the infrastructure =
= Booking the infrastructure =


Do you want to book the infrastructure for a course? Please send a request through https://www.egi.eu/services/training-infrastructure/
The infrastructure currently includes enough capacity to scale up to class-room size audiences, approximately up to 100 participants. Do you want to book the infrastructure for a course? Please send a request through https://www.egi.eu/services/training-infrastructure/


[[Category:Community_Engagement]]
[[Category:Community_Engagement]]

Latest revision as of 07:36, 8 July 2020

Engagement overview Community requirements Community events Training EGI Webinars Documentations



Introduction to the EGI training infrastructure

During 2015, under the coordination of the EGI-Engage project, the EGI community established a cloud based e-infrastructure specifically for training. The training infrastructure exists as a resource pool - a so called Virtual Organisation (VO) - within the [Federated_Cloud_user_support|EGI Federated Cloud infrastructure]. The training infrastructure provides ‘Infrastructure as a Service’ clouds (IaaS), i.e. physical resources (CPU, storage), as well as access services (login, application catalogue and application management portal) for face-to-face events, online training courses or self-paced learning modules. The training infrastructure is integrated with the EGI authentication-authorisation service (CheckIn) that allows trainers to generate short-living user accounts for training participants. Such accounts can identify students individually, and for a limited lifetime - typically few hours or days, depending on the length of the training event - allow them to interact with the training infrastructure sites and services.

The infrastructure currently includes enough capacity to scale up to class-room size audiences, approximately up to 100 participants.

Introductory slides and poster of the Training Infrastructure: https://indico.egi.eu/indico/contributionDisplay.py?contribId=122&confId=2544 (These were prepared for a Demo at the EGI Community Forum event in Bari, 2015 November. Outdated in some parts.)

Usage models

The training infrastructure is suitable for two types of courses:

  1. Cloud computing courses: Such courses teach students about IaaS clouds and the way Virtual Machine images, storage blocks and other types of ‘low level’ resources are managed. For such courses In this operational mode the infrastructure the trainer does not have to deploy applications or online services on the infrastructure in advance of the course. The applications/services will be deployed by the students themselves as training exercises. Such courses typically target programmers or other rather technical members of scientific communities or projects.
  2. Scientific courses: Such courses teach scientists or programmers about a specific software that’s relevant for their work. For example a specific gene sequence analysis software suite, an earthquake visualisation tool, a data processing pipeline. In this operational mode the the trainer deploys the domain specific application/tool on the training infrastructure before the training and the students interact directly with those applications/tools without even knowing where those are deployed and running. Depending on how computationally or data heavy the student exercises are, multiple students can share a single software deployment, or each student can have his/her own dedicated deployment. The configuration can be controlled by the teacher when the setup is deployed. have dedicated training environments, and the community can benefit from the easy deployment, predictability and repeatability of courses.

In both operational case the deployment of applications/tools/services can happen in the form of ‘Virtual Machine images’ (VMIs), and block storages - the latter basically behaving like a virtual USB drive that can be attached/detached to VMIs to provide data and storage space for applications.

VMIs can be deployed on the infrastructure through the EGI Applications Marketplace (https://appdb.egi.eu/browse/cloud), a growing catalogue that includes both basic applications (e.g. latest version of clean Linux deployments) and more specialised applications (e.g. Jupyter Notebook). A specific section within the catalogue is reserved for training-related application VMIs (https://appdb.egi.eu/store/vo/training.egi.eu).

The Applications Marketplace also includes a VMI management dashboard (https://dashboard.appdb.egi.eu), offering a Web interface for both trainers and students to deploy and manage VMIs from the Marketplace on the training infrastructure.

Available resources

The infrastructure currently federates the following resources:

Sites through the training.egi.eu Virtual Organisation (VO):

Available clouds:

  • CESNET-MCC - OpenStack cloud site (Czech Republic)
    • 64 vCPUs, 110 GB of RAM, 1 TB of persistent storage
    • Contact: Miroslav Ruda <ruda@ics.muni.cz>, Boris Parak <parak@cesnet.cz>
  • INFN-CATANIA-STACK - OpenStack cloud site (Italy)
    • 20 vCPUs, 50 GB of RAM, 1TB of storage volume
    • Contact: roberto.barbera@ct.infn.it, giuseppe.platania@ct.infn.it, sg-licence@ct.infn.it
  • IISAS-GPUCloud - OpenStack cloud site (Slovakia)
    • 12 vCPU cores, 2 GPU and 48GB of RAM
      • In case of special request, these resources can be doubled but must be reserved in advance
    • Contact: viet.ui@savba.sk, cloud-admin@savba.sk
  • IISAS-FedCloud - OpenStack cloud site (Slovakia)
    • 16 vCPU cores with 1.6GB of RAM per core
    • Contact: viet.ui@savba.sk, cloud-admin@savba.sk
  • IFCA-LCG2 OpenStack (Spain)
    • 20 vCPU cores, 50GB of RAM in total, 1TB of block storage, 1 public IP address
    • Additional IP addresses can be provided, upon request, for the specific training event
    • Contact: aloga@ifca.unican.es

Cloud not anymore available:

  • BIFI OpenStack cloud site (Spain)
    • 100 vCPUs, 200 GB of RAM, 1TB for storage volumes, 50 public IP addresses
    • Contact: Ruben Valles <rvalles@bifi.es>

Documentation:


Do you want to join as a resource provider? Please email support@egi.eu.

Modules and examples

The training infrastructure was used in July 2015 for two face-to-face training courses that trained programmers about the use of the EGI Federated Cloud (so the first operational model above). The focus of these courses were on the use of the rOCCI client interface to interact with the EGI cloud (instantiate, delete, access VMs). During these courses the training infrastructure included three cloud sites (CESNET, BIFI, UKIM) and hosted approx 25 student per event. Before the event the trainers prepared a basic VM that included the rOCCI client (with the training VO pre-configured in it), one user account for each student with a short-term certificate copied under each (with 24h lifetime). One instance of the VM was started on the infrastructure by the trainers before the event and the student were provided with login details for the user accounts. After logging in the students could use the OCCI commands of the rOCCI client with their own short-term certificates and could manage virtual machines on the 3 cloud sites of the training infrastructure. The presentations and hands-on materials from the teo events are available here:

Advanced Usage

To configure an HTC clusters on top of IaaS infastructure, please check the following Deploy HTC clusters on top of IaaS infrastructure guide.

Booking the infrastructure

The infrastructure currently includes enough capacity to scale up to class-room size audiences, approximately up to 100 participants. Do you want to book the infrastructure for a course? Please send a request through https://www.egi.eu/services/training-infrastructure/