Alert.png The wiki is deprecated and due to be decommissioned by the end of September 2022.
The content is being migrated to other supports, new updates will be ignored and lost.
If needed you can get in touch with EGI SDIS team using operations @ egi.eu.

Difference between revisions of "User:Enolfc"

From EGIWiki
Jump to navigation Jump to search
Line 4: Line 4:


{{Fedcloud_Menu}} {{TOC_right}}  
{{Fedcloud_Menu}} {{TOC_right}}  
<br> <big>Technical consultancy and support is available via the [mailto:support@egi.eu EGI.eu Support Team]</big>


== Introduction  ==
== Introduction  ==


<br> <big>Technical consultancy and support is available via the [mailto:support@egi.eu EGI.eu Support Team]</big>
The EGI Cloud federation is a hybrid, Infrastructure as a Service cloud composed by public, community and private cloud providers. These providers are federated with the use of the 'EGI Core Infrastructure Platform', offering a scalable compute and storage infrastructure for scientific applications, services and data- and compute-intensive workloads.  


<br> The EGI Cloud federation is an hybrid, Infrastructure as a Service cloud composed by public, community and private cloud. The clouds are federated with the use of the 'EGI Core Infrastructure Platform', offering a scalable compute and storage infrastructure for scientific applications, services, workloads. The main capabilities of the system are:
=== Main features ===


;Elastic computing infrastructure.
;Elastic computing infrastructure
:Execute compute and data intensive workloads (both batch and interactive), host long-running services (e.g. web servers, databases or applications servers), or create disposable testing and development environments in VMs and containers. Scale your application or service within a single provider, or across multiple providers of the federation (within providers of your virtual organisation). Select VM configurations (CPU, memory, disk) and ready-to-deploy application VMs that best fit your needs.
:Execute compute and data intensive workloads (both batch and interactive), host long-running services (e.g. web servers, databases or applications servers), or create disposable testing and development environments in VMs and containers. Scale your application or service within a single provider, or across multiple providers of the federation (within providers of your virtual organisation). Select VM configurations (CPU, memory, disk) and ready-to-deploy application VMs that best fit your needs.


Line 23: Line 26:
:Run docker applications on the EGI resources; Use one of the already integrated PaaS and SaaS solution; Follow our user guides to deploy Hadoop, Docker Swarm, to access Object Storage and many more...
:Run docker applications on the EGI resources; Use one of the already integrated PaaS and SaaS solution; Follow our user guides to deploy Hadoop, Docker Swarm, to access Object Storage and many more...


=== Usage models and examples ===
The flexibility of the Infrastructure as a Service EGI cloud can benefit various use cases and usage models. Besides serving compute/data intensive analysis workflows, Web services and interactive applications can be also integrated with and hosted on this infrastructure. Contextualisation and other deployment features can help application operators fine tune services in the cloud, meeting software (OS and software packages), hardware (number of cores, amount of RAM, etc.) and other types of needs (e.g. orchestration, scalability).
Since the opening of the EGI Federated Cloud, the following typical usage models have emerged:
*'''Service hosting''': the EGI Federated Cloud can be used to hosts any IT service as web servers, databases, etc. Cloud features, as elasticity, can help users to provide better performance and reliable services.
** Example: [https://www.egi.eu/use-cases/scientific-applications-tools/nbis-toolkit/ NBIS Web services], [https://www.egi.eu/news/peachnote-in-unison-with-egi/ Peachnote analysis platform]
*'''Compute and data intensive''': applications needing considerable amount of resources in term of computation and/or memory and/or intensive I/O. Ad-hoc computing environments can be created in the FedCloud sites also to satisfy very hard HW resource requirements.
** Example: [https://www.egi.eu/news/new-egi-use-case-a-close-look-at-the-amatrice-earthquake/ VERCE platform], [https://www.egi.eu/use-cases/research-stories/the-genetics-of-salmonella-infections/ The Genetics of Salmonella Infections], [https://www.egi.eu/use-cases/research-stories/new-viruses-implicated-in-fatal-snake-disease/ The Chipster Platform]
*'''Datasets repository''': the EGI Federated Cloud can be used to store and manage large datasets exploiting the big amount of disk storage available in the Federation.
*'''Disposable and testing environments''': environments for training or testing new developments.
** Example: [[Training_infrastructure|Events conducted on the cloud-based EGI Training Infrastructure]]
<!--
=== Success Stories ===
Some links
-->
<!-- == Current Users and Communities  ==
<!-- == Current Users and Communities  ==


Line 30: Line 53:
-->
-->


== Access modes  ==
== Getting started  ==
 
=== Access to the resources ===


EGI Infrastructure as a Service (IaaS) Cloud Resources can be accessed through 'Virtual Organizations' (VOs). A VO is a grouping of IaaS cloud provider from the EGI federation, who allocate capacity for a specific user group. Users with similar interest/requirements can join or form a VO to gather resources from EGI cloud providers - typically for a given project, experiment or use case. There are generic VOs too, for example the 'fedcloud.egi.eu' VO, which is open for any user who wants to experiment with the EGI Federated Cloud. You have to join a VO before you can interact with EGI IaaS cloud resources, while higher level services (PaaS, SaaS) do not always require VO membership.
EGI Infrastructure as a Service (IaaS) Cloud Resources can be accessed through '''''Virtual Organizations''''' (VOs). A VO is a grouping of IaaS cloud provider from the EGI federation, who allocate capacity for a specific user group. Users with similar interest/requirements can join or form a VO to gather resources from EGI cloud providers - typically for a given project, experiment or use case. There are generic VOs too, for example the <code>fedcloud.egi.eu</code> VO, which is open for any user who wants to experiment with the EGI Federated Cloud. You have to join a VO before you can interact with EGI IaaS cloud resources, while higher level services (PaaS, SaaS) do not always require VO membership.


VO membership is controlled in EGI by X.509 certificates. To access the EGI IaaS cloud you need to:  
VO membership is controlled in EGI by X.509 certificates. To access the EGI IaaS cloud you need to:  


# [http://www.egi.eu/how-to/get_a_certificate.html Obtain a personal X.509 access certificate] from a recognised Certification Authority (unless you have one already).
# [http://www.egi.eu/how-to/get_a_certificate.html Obtain a personal X.509 access certificate] from a recognised Certification Authority (unless you have one already).
# Join an existing VO, or form a VO if none of the existing ones suit your purpose"
# Join an existing VO, or form a VO if none of the existing ones suit your purpose:
## The [https://perun.metacentrum.cz/cert/registrar/?vo=fedcloud.egi.eu fedcloud.egi.eu] Virtual Organisation serves as a test ground for users to try the EGI cloud and to prototype and validate applications. It can be used for up to 6 month by any new user.  
## The [https://perun.metacentrum.cz/cert/registrar/?vo=fedcloud.egi.eu fedcloud.egi.eu] Virtual Organisation serves as a test ground for users to try the EGI cloud and to prototype and validate applications. It can be used for up to 6 month by any new user.  
## You can search for and join other [http://operations-portal.egi.eu/vo/search established VOs]. (Filter for 'cloud' in the middleware column)
## You can search for and join other [http://operations-portal.egi.eu/vo/search established VOs]. (Filter for '''cloud''' in the middleware column)
 
Remarks:
* After the 6-month long membership in the fedcloud.egi.eu VO, you will need to move to a production VO, or establish a new VO.
* Pre-existing VOs of EGI can be also extended with cloud resources. Consult with your VO manager. 
* If none of the existing VOs matches your use case, then a new VO can be created. Please follow [[PROC14_VO_Registration|VO Registration procedure]]. We can invite sites from the infrastructure to support your VO.
 
=== Cloud interfaces ===


IaaS cloud resources can expose two types of interfaces towards users (one or the other or both - depending on the cloud provider):  
IaaS cloud resources on EGI can expose two types of interfaces towards users (one or the other or both - depending on the cloud provider):  
* '''Open Standard interfaces''': OCCI ([http://occi-wg.org/ Open Cloud Computing Interface]) to manage compute, blocks storage and network resources. This interface set are currently exposed by all of the OpenNebula and Synnefo cloud providers, and some of the OpenStack providers.
* '''Open Standard interfaces''': OCCI ([http://occi-wg.org/ Open Cloud Computing Interface]) to manage compute, blocks storage and network resources. This interface set are currently exposed by all of the OpenNebula and Synnefo cloud providers, and some of the OpenStack providers.
* '''OpenStack interfaces''': The native OpenStack interfaces (with X.509 authentication). These interfaces are currently exposed by all of the OpenStack-based EGI cloud providers.  
* '''OpenStack interfaces''': The native OpenStack interfaces (with X.509 authentication). These interfaces are currently exposed by all of the OpenStack-based EGI cloud providers.  
Line 55: Line 87:
| '''API level access'''  
| '''API level access'''  
| [[Federated_Cloud_APIs_and_SDKs#API|OCCI]]
| [[Federated_Cloud_APIs_and_SDKs#API|OCCI]]
| [[Federated_Cloud_APIs_and_SDKs#API_2|OpenStack Compute && Openstack Object Storage]]
| [[Federated_Cloud_APIs_and_SDKs#API_2|OpenStack Compute & Openstack Object Storage]]
|-
|-
| '''Command Line access'''
| '''Command Line access'''
Line 66: Line 98:
|}
|}


== Typical usage models with examples ==
=== Creating your first VM instance  ===


The flexibility of the Infrastructure as a Service EGI cloud can benefit various use cases and usage models. Besides serving compute/data intensive analysis workflows, Web services and interactive applications can be also integrated with and hosted on this infrastructure. Contextualisation and other deployment features can help application operators fine tune services in the cloud, meeting software (OS and software packages), hardware (number of cores, amount of RAM, etc.) and other types of needs (e.g. orchestration, scalability).
Since the opening of the EGI Federated Cloud, the following typical usage models have emerged:


*'''Service hosting''': the EGI Federated Cloud can be used to hosts any IT service as web servers, databases, etc. Cloud features, as elasticity, can help users to provide better performance and reliable services.
The basic user workflow for starting a Virtual Machine (VM) looks like this:
** Example: [https://www.egi.eu/use-cases/scientific-applications-tools/nbis-toolkit/ NBIS Web services], [https://www.egi.eu/news/peachnote-in-unison-with-egi/ Peachnote analysis platform]
*'''Compute and data intensive''': applications needing considerable amount of resources in term of computation and/or memory and/or intensive I/O. Ad-hoc computing environments can be created in the FedCloud sites also to satisfy very hard HW resource requirements.
** Example: [https://www.egi.eu/news/new-egi-use-case-a-close-look-at-the-amatrice-earthquake/ VERCE platform], [https://www.egi.eu/use-cases/research-stories/the-genetics-of-salmonella-infections/ The Genetics of Salmonella Infections], [https://www.egi.eu/use-cases/research-stories/new-viruses-implicated-in-fatal-snake-disease/ The Chipster Platform]
*'''Datasets repository''': the EGI Federated Cloud can be used to store and manage large datasets exploiting the big amount of disk storage available in the Federation.
*'''Disposable and testing environments''': environments for training or testing new developments.
** Example: [[Training_infrastructure|Events conducted on the cloud-based EGI Training Infrastructure]]


[[Image:Fedcloud.png|center|550px|Fedcloud.png]] <br>


A theoretical setup combining multiple usage models:
[[Image:FedCloudUsageModels.png|center|550px|FedCloudUsageModels.png]] <br>


== User guides ==
The '''IaaS Cloud user''' (or a user gateway that acts on behalf of the users) is responsible for managing the virtual infrastructure on top of the IaaS provider. He/she spins up Virtual Machines, Block Storage and Object Storage to create services that will be accessed by some '''service users''' (could be the very same IaaS user but also other VO members or unrelated users). The method of accessing the services vary greatly from one service to another. '''Cloud Developers''' in each community are responsible for creating new appliances that encapsulate the applications that the IaaS Cloud users can start. These appliances are managed in the AppDB with the help of the VO managers that curate which appliances are available to each community.


=== User authentication and Virtual Organisations ===
As an IaaS Cloud user you will handle VMs, follow these steps to start your first one:


EGI provides users with a single sign-on mechanism to access the services of the federated cloud. The single sign-on mechanism is based on X.509 certificates and Virtual Organisations (VOs). Before using EGI Federated Cloud IaaS service you will need to:
# Get your environment ready to launch VMs. The [[HOWTO11|command line client HOWTO]] describes how to install on your machine the client tool or how to get a docker image or VM image for VirtualBox that can be used to run the commands. You will need your certificate on your client.
 
# Browse the [https://appdb.egi.eu/browse/cloud Application Database Cloud Marketplace] the available Virtual Appliances. Virtual Appliances are the templates for the root volume of the running instances (Operating System and applications). EGI offers a set of basic images with minimal configuration that can get you started easily, but you can also find complete application stacks. See for example these images:  
# [http://www.egi.eu/how-to/get_a_certificate.html Obtain a personal X.509 access certificate] from a recognised Certification Authority (unless you have one already).
# Join an existing VO, or form a VO if none of the existing ones suit your purpose"
## The [https://perun.metacentrum.cz/cert/registrar/?vo=fedcloud.egi.eu fedcloud.egi.eu] Virtual Organisation serves as a test ground for users to try the EGI cloud and to prototype and validate applications. It can be used for up to 6 month by any new user.  
## You can search for and join other [http://operations-portal.egi.eu/vo/search established VOs]. (Filter for 'cloud' in the middleware column)
 
Remarks:
* After the 6-month long membership in the fedcloud.egi.eu VO, you will need to move to a production VO, or establish a new VO.
*Grid VOs of EGI can be also extended with cloud resources. Consult with the manager of the VO. 
*If none of the existing VOs matches your use case, then a new VO can be created. Please follow [[PROC14_VO_Registration|VO Registration procedure]]. We can invite sites from the infrastructure to support your VO.
 
=== Creating your first VM - Instantiate an exiting image ===
 
The basic user workflow for starting a Virtual Machine (VM) looks like this:
 
#Browse in the [https://appdb.egi.eu/browse/cloud Application Database Cloud Marketplace] the available Virtual Appliances. Virtual Appliances are the templates for the root volume of the running instances (Operating System and applications). EGI offers a set of basic images with minimal configuration that can get you started easily, but you can also find complete application stacks. See for example these images:  
#*[https://appdb.egi.eu/store/vappliance/egi.ubuntu.14.04 EGI Ubuntu 14], a basic [http://www.ubuntu.com/ Ubuntu 14.04] image  
#*[https://appdb.egi.eu/store/vappliance/egi.ubuntu.14.04 EGI Ubuntu 14], a basic [http://www.ubuntu.com/ Ubuntu 14.04] image  
#*[https://appdb.egi.eu/store/vappliance/egi.centos.6 EGI Centos 6], a basic [https://www.centos.org/ CentOS 6] image  
#*[https://appdb.egi.eu/store/vappliance/egi.centos.6 EGI Centos 6], a basic [https://www.centos.org/ CentOS 6] image  
#*[https://appdb.egi.eu/store/software/compss.framework COMPSs], a VM for using [[HOWTO14_How_to_use_COMPSs|COMPSs]] programming framework
#*[https://appdb.egi.eu/store/software/compss.framework COMPSs], a VM for using [[HOWTO14_How_to_use_COMPSs|COMPSs]] programming framework
#*[https://appdb.egi.eu/store/vappliance/biovel.portal BioVel Portal], for executing a http://www.biovel.eu/ biodiversity virtual e-Laboratory]
#*[https://appdb.egi.eu/store/vappliance/biovel.portal BioVel Portal], for executing a http://www.biovel.eu/ biodiversity virtual e-Laboratory]
#Read the description of the appliance for information on how to use it and connect to it once started.
#*You may need to have a ssh key in order to login into the VM, check [[FAQ10#How_can_I_inject_my_public_SSH_key_into_the_machine.3F|the FAQ on how to create one]]
#Get the IDs for starting the appliance at one of the sites supporting it in the "Availability and Usage" tab.  
#Get the IDs for starting the appliance at one of the sites supporting it in the "Availability and Usage" tab.  
#Get the [[HOWTO11|command line client]] to start the VM using the IDs from previous step
# Virtual Appliances in AppDB are ready to start on the EGI resources, but in order to use them, you will need to perform some '''''contextualisation'''''. Contextualisation is the process of customising the appliance when it is instantiated on the resources, e.g. entering some credentials to log into the VM.
#*Passwords are not allowed by default in EGI Federated Cloud for accessing VMs, you would likely need a ssh key to proceed. Check [[FAQ10#How_can_I_inject_my_public_SSH_key_into_the_machine.3F|the FAQ on how to create one]]
#*Most VAs in AppDB support cloud-init, the de-facto standard for contextualisation. Check the [https://cloudinit.readthedocs.io/en/latest/ documentation and examples].
# [[HOWTO11#Proxy_Generation|Create a VOMS proxy for getting access to the infrastructure]]
# Use the client to start the VM using the IDs obtained from AppDB.
#*Check the [[FAQ10#How_can_I_start_a_VM.3F|How can I start a VM?]] FAQ entry and the [[HOWTO11|command line client how to]] for more information.
#*Check the [[FAQ10#How_can_I_start_a_VM.3F|How can I start a VM?]] FAQ entry and the [[HOWTO11|command line client how to]] for more information.
#Connect to VM and start using it
# Now you can connect to the VM and start using it
#*You may need to allocate a public IP for the VM, check [[FAQ10#How_can_I_assign_a_public_IP_to_a_VM.3F|How can I assign a public IP to a VM]] in the Federated Cloud FAQ
#*You may need to allocate a public IP for the VM, check [[FAQ10#How_can_I_assign_a_public_IP_to_a_VM.3F|How can I assign a public IP to a VM]] in the Federated Cloud FAQ
#*Login using ssh: <code>ssh -i <your private key> <vm user>@<your vm ip></code>


Check out these [https://documents.egi.eu/document/2622 tutorial slides] for a practical overview of these steps.
Check out these [https://documents.egi.eu/document/2622 tutorial slides] for a practical overview of these steps.


=== Advanced topics  ===


==== API and SDKs access to Federated Cloud resources ====
=== API and SDKs access to Federated Cloud resources ===


Besides the command line client, there are several [[Federated Cloud APIs and SDKs|APIs and SDK ready to be used with the EGI Federated Cloud]].Preferred API for EGI federated cloud is OCCI, which can be accesses using using a command-line client (rOCCI), high-level tools or directly implementing the ‘Open Cloud Computing Interface’ (OCCI) into your environment. OpenStack Nova API is also available for OpenStack sites belonging to the Federation.
Besides the command line client, there are several [[Federated Cloud APIs and SDKs|APIs and SDK ready to be used with the EGI Federated Cloud]].Preferred API for EGI federated cloud is OCCI, which can be accesses using using a command-line client (rOCCI), high-level tools or directly implementing the ‘Open Cloud Computing Interface’ (OCCI) into your environment. OpenStack Nova API is also available for OpenStack sites belonging to the Federation.
Line 127: Line 136:
A crash course on how to use programming interfaces of the EGI Federated Cloud, and how these APIs can be used to integrate high-level systems with it is available [https://indico.egi.eu/indico/event/3113/ here]. Please check the [[EGI Federated Cloud jOCCI APIs| EGI Federated Cloud for developers]] guide for details on how to use them.
A crash course on how to use programming interfaces of the EGI Federated Cloud, and how these APIs can be used to integrate high-level systems with it is available [https://indico.egi.eu/indico/event/3113/ here]. Please check the [[EGI Federated Cloud jOCCI APIs| EGI Federated Cloud for developers]] guide for details on how to use them.


==== Storage ====
 
=== Creating custom appliances ===
 
== User guides ==
 
=== User authentication and Virtual Organisations ===
 
EGI provides users with a single sign-on mechanism to access the services of the federated cloud. The single sign-on mechanism is based on X.509 certificates and Virtual Organisations (VOs). Before using EGI Federated Cloud IaaS service you will need to:
 
# [http://www.egi.eu/how-to/get_a_certificate.html Obtain a personal X.509 access certificate] from a recognised Certification Authority (unless you have one already).
# Join an existing VO, or form a VO if none of the existing ones suit your purpose"
## The [https://perun.metacentrum.cz/cert/registrar/?vo=fedcloud.egi.eu fedcloud.egi.eu] Virtual Organisation serves as a test ground for users to try the EGI cloud and to prototype and validate applications. It can be used for up to 6 month by any new user.
## You can search for and join other [http://operations-portal.egi.eu/vo/search established VOs]. (Filter for 'cloud' in the middleware column)
 
===Creating custom VMs  ===
 
You can prepare fully customised Virtual Appliances and deploy them to the sites:
 
#Prepare Virtual Machine Images (VMIs) that encapsulate your application. See the [[HOWTO10|application porting tutorial]] for tips.
#Make the VMI available online, for example in the [http://appliance-repo.egi.eu/images/ EGI Appliance Repository]
#Register the VMIs as Virtual Appliances in the [http://appdb.egi.eu EGI Applications Database] (for howto please click [https://wiki.appdb.egi.eu/main:faq:how_to_register_a_virtual_appliance here])
#Once your VA is published (see howto [https://wiki.appdb.egi.eu/main:guides:guide_for_managing_virtual_appliance_versions_using_the_portal here]),inform the Manager of your VO through Applications Database about it (see howto [https://wiki.appdb.egi.eu/main:guides:notify_virtual_organization_representatives here]). He/she will include your images in the VO-wide image list, so these will be deployed on the Federated Cloud sites of your VO.
 
== Advanced topics  ==
=== Storage ===


Every instantiated VM has some disk space provided with it, if you need more storage or need to share data, you can use a cloud storage solution. There are two kind of services: Block Storage and Object Storage. Check the [[HOWTO09|EGI Federated Cloud Storage How To]] for more information.
Every instantiated VM has some disk space provided with it, if you need more storage or need to share data, you can use a cloud storage solution. There are two kind of services: Block Storage and Object Storage. Check the [[HOWTO09|EGI Federated Cloud Storage How To]] for more information.


==== Distributed data management with the EGI OpenData platform ====
=== Distributed data management with the EGI OpenData platform ===
The EGI OpenData platform is a solution allowing integration of various data repositories available in a distributed infrastructure, offering the capability to make data open, and link them to key open data catalogues following respective guidelines. The core enabling technology of OpenData platform is Onedata, a data management solution that allows a seamless and optimised access to data spread over a distributed infrastructure. Instructions on how to setup a OneData deployment in the EGI Federated Cloud are available [[EGI_Opendata_platform|here]].
The EGI OpenData platform is a solution allowing integration of various data repositories available in a distributed infrastructure, offering the capability to make data open, and link them to key open data catalogues following respective guidelines. The core enabling technology of OpenData platform is Onedata, a data management solution that allows a seamless and optimised access to data spread over a distributed infrastructure. Instructions on how to setup a OneData deployment in the EGI Federated Cloud are available [[EGI_Opendata_platform|here]].


==== Docker containers, Docker Swarm ====
=== Docker containers, Docker Swarm ===


You can [[Federated Cloud Containers|run your docker applications on EGI's Cloud]]. EGI also provides a [[Federated_Cloud_Containers#EGI_Federated_Cloud_clients_docker_image|docker image with the clients]] so you can test them easily.
You can [[Federated Cloud Containers|run your docker applications on EGI's Cloud]]. EGI also provides a [[Federated_Cloud_Containers#EGI_Federated_Cloud_clients_docker_image|docker image with the clients]] so you can test them easily.


==== High level tools: Orchestrators, Platforms/Software as a Service ====
=== High level tools: Orchestrators, Platforms/Software as a Service ===


Read ''[[Federated Cloud PaaS|this guidance]]'' about strategies of porting applications to the EGI Federated Cloud. The guide also includes references to high level user environments (orchestrators, Platform/Software as a Service) that can simplify the application integration and operation process for you. These environments offer high level abstractions and services on top of the baseline 'Infrastructure as a Service' cloud.
Read ''[[Federated Cloud PaaS|this guidance]]'' about strategies of porting applications to the EGI Federated Cloud. The guide also includes references to high level user environments (orchestrators, Platform/Software as a Service) that can simplify the application integration and operation process for you. These environments offer high level abstractions and services on top of the baseline 'Infrastructure as a Service' cloud.


==== Running Hadoop applications ====
=== Running Hadoop applications ===


Using a WS-PGRADE gateway that is connected to the EGI Federated cloud, it is possible to deploy Hadoop clusters on EGI Federated Cloud resources, to execute Hadoop applications on those clusters and finally to release resources after application execution. The concept is outlined on [https://indico.egi.eu/indico/event/2931/material/slides/ these PPT slides]. A user manual is [https://indico.egi.eu/indico/event/2931/material/0/ available here (v1.3)].
Using a WS-PGRADE gateway that is connected to the EGI Federated cloud, it is possible to deploy Hadoop clusters on EGI Federated Cloud resources, to execute Hadoop applications on those clusters and finally to release resources after application execution. The concept is outlined on [https://indico.egi.eu/indico/event/2931/material/slides/ these PPT slides]. A user manual is [https://indico.egi.eu/indico/event/2931/material/0/ available here (v1.3)].


==== GPGPUs (pre-production!) ====
=== GPGPUs (pre-production!) ===


'''PRE-PRODUCTION''' GPGPUs-enabled cloud resources are available in selected sites of the EGI Federated Cloud. Check the [[Federated_Cloud_GPGPU|GPGPU guide on FedCloud]] for details on how to access them.
'''PRE-PRODUCTION''' GPGPUs-enabled cloud resources are available in selected sites of the EGI Federated Cloud. Check the [[Federated_Cloud_GPGPU|GPGPU guide on FedCloud]] for details on how to access them.


==== Customized Virtual Appliances  ====
You can prepare fully customised Virtual Appliances and deploy them to the sites:
#Prepare Virtual Machine Images (VMIs) that encapsulate your application. See the [[HOWTO10|application porting tutorial]] for tips.
#Make the VMI available online, for example in the [http://appliance-repo.egi.eu/images/ EGI Appliance Repository]
#Register the VMIs as Virtual Appliances in the [http://appdb.egi.eu EGI Applications Database] (for howto please click [https://wiki.appdb.egi.eu/main:faq:how_to_register_a_virtual_appliance here])
#Once your VA is published (see howto [https://wiki.appdb.egi.eu/main:guides:guide_for_managing_virtual_appliance_versions_using_the_portal here]),inform the Manager of your VO through Applications Database about it (see howto [https://wiki.appdb.egi.eu/main:guides:notify_virtual_organization_representatives here]). He/she will include your images in the VO-wide image list, so these will be deployed on the Federated Cloud sites of your VO.


==== Running Galaxy Workflows with EC3 ====
=== Running Galaxy Workflows with EC3 ===


Using the [http://servproject.i3m.upv.es/ec3/ EC3] open-source software platform, users can deploy elastic clusters on demand and dynamically deploy complex scientific virtual computing infrastructures on top of Infrastructure as a Service Clouds. More details on how to use Galaxy workflows on the platform are described in this wiki. Please check the [[Galaxy workflows with EC3 | Galaxy workflows in EGI with EC3]] guide for further details.
Using the [http://servproject.i3m.upv.es/ec3/ EC3] open-source software platform, users can deploy elastic clusters on demand and dynamically deploy complex scientific virtual computing infrastructures on top of Infrastructure as a Service Clouds. More details on how to use Galaxy workflows on the platform are described in this wiki. Please check the [[Galaxy workflows with EC3 | Galaxy workflows in EGI with EC3]] guide for further details.


==== How SMEs Can Use EGI's Cloud for Computer-Aided Engineering (CAE) ====
=== How SMEs Can Use EGI's Cloud for Computer-Aided Engineering (CAE) ===
Please check how SMEs can use [[Computer-Aided Engineering (CAE) in the EGI Federated Cloud | Computer-Aided Engineering (CAE) in the EGI Federated Cloud]], with the example of OpenFOAM software containers.
Please check how SMEs can use [[Computer-Aided Engineering (CAE) in the EGI Federated Cloud | Computer-Aided Engineering (CAE) in the EGI Federated Cloud]], with the example of OpenFOAM software containers.



Revision as of 14:47, 20 January 2017

Enol Fernandez - EGI

https://www.egi.eu/sso/userDetail/enolfc

Overview For users For resource providers Infrastructure status Site-specific configuration Architecture





Technical consultancy and support is available via the EGI.eu Support Team


Introduction

The EGI Cloud federation is a hybrid, Infrastructure as a Service cloud composed by public, community and private cloud providers. These providers are federated with the use of the 'EGI Core Infrastructure Platform', offering a scalable compute and storage infrastructure for scientific applications, services and data- and compute-intensive workloads.

Main features

Elastic computing infrastructure
Execute compute and data intensive workloads (both batch and interactive), host long-running services (e.g. web servers, databases or applications servers), or create disposable testing and development environments in VMs and containers. Scale your application or service within a single provider, or across multiple providers of the federation (within providers of your virtual organisation). Select VM configurations (CPU, memory, disk) and ready-to-deploy application VMs that best fit your needs.
VM image sharing and distribution
Easily share and distribute customised VM images to multiple clouds via the open 'Applications Database' library of Virtual Appliances. Community curated VMs and VM appliances are securely and automatically replicated across the infrastructure. The EGI User Community Support Team provides generic, baseline VM images, user communities can offer more specialised VMs and applications.
Unified view of federation
The EGI Cloud provides: Single sing-on (SSO) for authentication and authorisation across all resource providers; Federated accounting with an integrated view of the the resource and service usage; Distributed information system for delivering a real-time view of the capabilities; and Federated monitoring to compute metrics for availability and reliability of the services.
Beyond VMs
Run docker applications on the EGI resources; Use one of the already integrated PaaS and SaaS solution; Follow our user guides to deploy Hadoop, Docker Swarm, to access Object Storage and many more...

Usage models and examples

The flexibility of the Infrastructure as a Service EGI cloud can benefit various use cases and usage models. Besides serving compute/data intensive analysis workflows, Web services and interactive applications can be also integrated with and hosted on this infrastructure. Contextualisation and other deployment features can help application operators fine tune services in the cloud, meeting software (OS and software packages), hardware (number of cores, amount of RAM, etc.) and other types of needs (e.g. orchestration, scalability).

Since the opening of the EGI Federated Cloud, the following typical usage models have emerged:

  • Service hosting: the EGI Federated Cloud can be used to hosts any IT service as web servers, databases, etc. Cloud features, as elasticity, can help users to provide better performance and reliable services.
  • Compute and data intensive: applications needing considerable amount of resources in term of computation and/or memory and/or intensive I/O. Ad-hoc computing environments can be created in the FedCloud sites also to satisfy very hard HW resource requirements.
  • Datasets repository: the EGI Federated Cloud can be used to store and manage large datasets exploiting the big amount of disk storage available in the Federation.
  • Disposable and testing environments: environments for training or testing new developments.


Getting started

Access to the resources

EGI Infrastructure as a Service (IaaS) Cloud Resources can be accessed through Virtual Organizations (VOs). A VO is a grouping of IaaS cloud provider from the EGI federation, who allocate capacity for a specific user group. Users with similar interest/requirements can join or form a VO to gather resources from EGI cloud providers - typically for a given project, experiment or use case. There are generic VOs too, for example the fedcloud.egi.eu VO, which is open for any user who wants to experiment with the EGI Federated Cloud. You have to join a VO before you can interact with EGI IaaS cloud resources, while higher level services (PaaS, SaaS) do not always require VO membership.

VO membership is controlled in EGI by X.509 certificates. To access the EGI IaaS cloud you need to:

  1. Obtain a personal X.509 access certificate from a recognised Certification Authority (unless you have one already).
  2. Join an existing VO, or form a VO if none of the existing ones suit your purpose:
    1. The fedcloud.egi.eu Virtual Organisation serves as a test ground for users to try the EGI cloud and to prototype and validate applications. It can be used for up to 6 month by any new user.
    2. You can search for and join other established VOs. (Filter for cloud in the middleware column)

Remarks:

  • After the 6-month long membership in the fedcloud.egi.eu VO, you will need to move to a production VO, or establish a new VO.
  • Pre-existing VOs of EGI can be also extended with cloud resources. Consult with your VO manager.
  • If none of the existing VOs matches your use case, then a new VO can be created. Please follow VO Registration procedure. We can invite sites from the infrastructure to support your VO.

Cloud interfaces

IaaS cloud resources on EGI can expose two types of interfaces towards users (one or the other or both - depending on the cloud provider):

  • Open Standard interfaces: OCCI (Open Cloud Computing Interface) to manage compute, blocks storage and network resources. This interface set are currently exposed by all of the OpenNebula and Synnefo cloud providers, and some of the OpenStack providers.
  • OpenStack interfaces: The native OpenStack interfaces (with X.509 authentication). These interfaces are currently exposed by all of the OpenStack-based EGI cloud providers.

The user can interact with IaaS cloud resources via programming APIs, command line interfaces or Web dashboards. The different access modes are summarized in the following table:

Open Standards interface OpenStack interface
API level access OCCI OpenStack Compute & Openstack Object Storage
Command Line access rOCCI-cli OpenStack CLI with VOMS authentication plugin
Web dashboard access AppDB VMOps Dashboard (in final test) OpenStack Horizon (production instance under deployment)

Creating your first VM instance

The basic user workflow for starting a Virtual Machine (VM) looks like this:

Fedcloud.png



The IaaS Cloud user (or a user gateway that acts on behalf of the users) is responsible for managing the virtual infrastructure on top of the IaaS provider. He/she spins up Virtual Machines, Block Storage and Object Storage to create services that will be accessed by some service users (could be the very same IaaS user but also other VO members or unrelated users). The method of accessing the services vary greatly from one service to another. Cloud Developers in each community are responsible for creating new appliances that encapsulate the applications that the IaaS Cloud users can start. These appliances are managed in the AppDB with the help of the VO managers that curate which appliances are available to each community.

As an IaaS Cloud user you will handle VMs, follow these steps to start your first one:

  1. Get your environment ready to launch VMs. The command line client HOWTO describes how to install on your machine the client tool or how to get a docker image or VM image for VirtualBox that can be used to run the commands. You will need your certificate on your client.
  2. Browse the Application Database Cloud Marketplace the available Virtual Appliances. Virtual Appliances are the templates for the root volume of the running instances (Operating System and applications). EGI offers a set of basic images with minimal configuration that can get you started easily, but you can also find complete application stacks. See for example these images:
  3. Get the IDs for starting the appliance at one of the sites supporting it in the "Availability and Usage" tab.
  4. Virtual Appliances in AppDB are ready to start on the EGI resources, but in order to use them, you will need to perform some contextualisation. Contextualisation is the process of customising the appliance when it is instantiated on the resources, e.g. entering some credentials to log into the VM.
    • Passwords are not allowed by default in EGI Federated Cloud for accessing VMs, you would likely need a ssh key to proceed. Check the FAQ on how to create one
    • Most VAs in AppDB support cloud-init, the de-facto standard for contextualisation. Check the documentation and examples.
  5. Create a VOMS proxy for getting access to the infrastructure
  6. Use the client to start the VM using the IDs obtained from AppDB.
  7. Now you can connect to the VM and start using it
    • You may need to allocate a public IP for the VM, check How can I assign a public IP to a VM in the Federated Cloud FAQ
    • Login using ssh: ssh -i <your private key> <vm user>@<your vm ip>

Check out these tutorial slides for a practical overview of these steps.


API and SDKs access to Federated Cloud resources

Besides the command line client, there are several APIs and SDK ready to be used with the EGI Federated Cloud.Preferred API for EGI federated cloud is OCCI, which can be accesses using using a command-line client (rOCCI), high-level tools or directly implementing the ‘Open Cloud Computing Interface’ (OCCI) into your environment. OpenStack Nova API is also available for OpenStack sites belonging to the Federation.

A crash course on how to use programming interfaces of the EGI Federated Cloud, and how these APIs can be used to integrate high-level systems with it is available here. Please check the EGI Federated Cloud for developers guide for details on how to use them.


Creating custom appliances

User guides

User authentication and Virtual Organisations

EGI provides users with a single sign-on mechanism to access the services of the federated cloud. The single sign-on mechanism is based on X.509 certificates and Virtual Organisations (VOs). Before using EGI Federated Cloud IaaS service you will need to:

  1. Obtain a personal X.509 access certificate from a recognised Certification Authority (unless you have one already).
  2. Join an existing VO, or form a VO if none of the existing ones suit your purpose"
    1. The fedcloud.egi.eu Virtual Organisation serves as a test ground for users to try the EGI cloud and to prototype and validate applications. It can be used for up to 6 month by any new user.
    2. You can search for and join other established VOs. (Filter for 'cloud' in the middleware column)

Creating custom VMs

You can prepare fully customised Virtual Appliances and deploy them to the sites:

  1. Prepare Virtual Machine Images (VMIs) that encapsulate your application. See the application porting tutorial for tips.
  2. Make the VMI available online, for example in the EGI Appliance Repository
  3. Register the VMIs as Virtual Appliances in the EGI Applications Database (for howto please click here)
  4. Once your VA is published (see howto here),inform the Manager of your VO through Applications Database about it (see howto here). He/she will include your images in the VO-wide image list, so these will be deployed on the Federated Cloud sites of your VO.

Advanced topics

Storage

Every instantiated VM has some disk space provided with it, if you need more storage or need to share data, you can use a cloud storage solution. There are two kind of services: Block Storage and Object Storage. Check the EGI Federated Cloud Storage How To for more information.

Distributed data management with the EGI OpenData platform

The EGI OpenData platform is a solution allowing integration of various data repositories available in a distributed infrastructure, offering the capability to make data open, and link them to key open data catalogues following respective guidelines. The core enabling technology of OpenData platform is Onedata, a data management solution that allows a seamless and optimised access to data spread over a distributed infrastructure. Instructions on how to setup a OneData deployment in the EGI Federated Cloud are available here.

Docker containers, Docker Swarm

You can run your docker applications on EGI's Cloud. EGI also provides a docker image with the clients so you can test them easily.

High level tools: Orchestrators, Platforms/Software as a Service

Read this guidance about strategies of porting applications to the EGI Federated Cloud. The guide also includes references to high level user environments (orchestrators, Platform/Software as a Service) that can simplify the application integration and operation process for you. These environments offer high level abstractions and services on top of the baseline 'Infrastructure as a Service' cloud.

Running Hadoop applications

Using a WS-PGRADE gateway that is connected to the EGI Federated cloud, it is possible to deploy Hadoop clusters on EGI Federated Cloud resources, to execute Hadoop applications on those clusters and finally to release resources after application execution. The concept is outlined on these PPT slides. A user manual is available here (v1.3).

GPGPUs (pre-production!)

PRE-PRODUCTION GPGPUs-enabled cloud resources are available in selected sites of the EGI Federated Cloud. Check the GPGPU guide on FedCloud for details on how to access them.


Running Galaxy Workflows with EC3

Using the EC3 open-source software platform, users can deploy elastic clusters on demand and dynamically deploy complex scientific virtual computing infrastructures on top of Infrastructure as a Service Clouds. More details on how to use Galaxy workflows on the platform are described in this wiki. Please check the Galaxy workflows in EGI with EC3 guide for further details.

How SMEs Can Use EGI's Cloud for Computer-Aided Engineering (CAE)

Please check how SMEs can use Computer-Aided Engineering (CAE) in the EGI Federated Cloud, with the example of OpenFOAM software containers.

Useful resources

User support

Technical support

Users' technical support is provided via the EGI support contact.

Helpdesk

Technical problems and questions relating to the use of the EGI Federated Cloud can be reported and dealt with through the EGI Helpdesk ticketing system.

Note: Please choose 'Federated cloud' in the 'Type of problem' field of the ticket submission form!

Feedback and open issues

A list of open-issue and feedbacks reported by the FedCloud users is available at this page.

Technical background

Cloud providers in the EGI Federated Cloud use hardware virtualization technologies to host software on their resources. The cloud management platforms that make this possible can vary from site to site, but they all enable the provisioning of virtualized computing, storage and networking resources, thus they empower scientific groups to setup and operate domain specific services, applications and simulations on these resources. Read more about the technology that drives the Federated Cloud.