Alert.png The wiki is deprecated and due to be decommissioned by the end of September 2022.
The content is being migrated to other supports, new updates will be ignored and lost.
If needed you can get in touch with EGI SDIS team using operations @ egi.eu.

Difference between revisions of "User:Enolfc"

From EGIWiki
Jump to navigation Jump to search
Line 1: Line 1:
Enol Fernandez - CSIC
Enol Fernandez - EGI
 
https://www.egi.eu/sso/userDetail/enolfc
https://www.egi.eu/sso/userDetail/enolfc


{{Fedcloud_Menu}} {{TOC_right}}


== Introduction  ==


{{Fedcloud_Menu}}
<br> <big>Technical consultancy and support is available via the [mailto:support@egi.eu EGI.eu Support Team]</big>
{{TOC_right}}


= EGI Cloud Federation =
<br> The EGI Cloud federation is an hybrid, Infrastructure as a Service cloud composed by public, community and private cloud. The clouds are federated with the use of the 'EGI Core Infrastructure Platform', offering a scalable compute and storage infrastructure for scientific applications, services, workloads. The main capabilities of the system are:


The EGI Federated Cloud is a multi-national cloud system that integrates institutional clouds into a scalable computing platform for data and/or compute driven applications and services. The initial architecture of the EGI Federated Cloud was defined in 2011-2012 and was fully implemented by May 2014. Currently, the federation is a collaboration that enables various types of cloud federations to serve diverse demands of researchers from both academia and industry. The EGI Federated Cloud brings together scientific communities, R&D projects, technology and resource providers to form a community that integrates and maintains a flexible solution portfolio that enables various types of cloud federations with IaaS, PaaS and SaaS capabilities. The collaboration is committed to the use of open source tools and services that are reusable across scientific disciplines. These tools and services form a flexible portfolio from which a scientific community can mix and match items to establish its own, customised cloud federation.  
;Elastic computing infrastructure.
:Execute compute and data intensive workloads (both batch and interactive), host long-running services (e.g. web servers, databases or applications servers), or create disposable testing and development environments in VMs and containers. Scale your application or service within a single provider, or across multiple providers of the federation (within providers of your virtual organisation). Select VM configurations (CPU, memory, disk) and ready-to-deploy application VMs that best fit your needs.


;VM image sharing and distribution
:Easily share and distribute customised VM images to multiple clouds via the open 'Applications Database' library of Virtual Appliances. Community curated VMs and VM appliances are securely and automatically replicated across the infrastructure. The EGI User Community Support Team provides generic, baseline VM images, user communities can offer more specialised VMs and applications.


The EGI Federated Cloud provides the services and technologies to create federation of clouds (community, private or public clouds) that operate according to the preferences, choices and constraints set by its members and users. The EGI Cloud Federations are modelled around the concept of an abstract Cloud Management stack subsystem that is integrated with components of the EGI Core Infrastructure and that provides a set of agreed uniform interfaces within the community it provides services to.
;Unified view of federation
:The EGI Cloud provides: Single sing-on (SSO) for authentication and authorisation across all resource providers; Federated accounting with an integrated view of the the resource and service usage; Distributed information system for delivering a real-time view of the capabilities; and Federated monitoring to compute metrics for availability and reliability of the services.


[[Image:Federated_Cloud_Model.png|thumb|center|600px|Federated Cloud Model]]
;Beyond VMs
:Run docker applications on the EGI resources; Use one of the already integrated PaaS and SaaS solution; Follow our user guides to deploy Hadoop, Docker Swarm, to access Object Storage and many more...


The EGI Cloud Federation (see Figure) is a hybrid cloud composed by public, community and private clouds, all supported by the EGI Core Infrastructure Platform services. The EGI Federated Cloud is composed by multiple “realms”, each realm having homogeneous cloud management interfaces and capabilities. A cloud realm is a subset of cloud providers exposing homogeneous cloud management interfaces and capabilities. The Open Standards Cloud Realm supports the usage of open standards for its interfaces and is completely integrated with the EGI Core Infrastructure Platform. A Community Platform provides community-specific data, tools and applications, which can be supported by one or more realms.
<!-- == Current Users and Communities  ==


== Services in cloud federations ==
EGI Federated Cloud already has a large use base, check the [[Federated Cloud Communities|FedCloud Users Communities]] page for more details.  
Despite the large diversity in the type of cloud realms, a relatively small number of identical building blocks (or federator services) can be identified in almost all of them. These services turn individual clouds into a federation. The table collects these common services to help architects identify topics they should focus on when designing a cloud federation. Technical details fro these are also available at [[Federated Cloud Technology]].


{| class="wikitable" style="margin: auto;"
[[Image:Fedclouduclogos.png|center|550px|Fedclouduclogos.png]]
-->
 
== Access modes  ==
 
EGI Infrastructure as a Service (IaaS) Cloud Resources can be accessed through 'Virtual Organizations' (VOs). A VO is a grouping of IaaS cloud provider from the EGI federation, who allocate capacity for a specific user group. Users with similar interest/requirements can join or form a VO to gather resources from EGI cloud providers - typically for a given project, experiment or use case. There are generic VOs too, for example the 'fedcloud.egi.eu' VO, which is open for any user who wants to experiment with the EGI Federated Cloud. You have to join a VO before you can interact with EGI IaaS cloud resources, while higher level services (PaaS, SaaS) do not always require VO membership.
 
VO membership is controlled in EGI by X.509 certificates. To access the EGI IaaS cloud you need to:
 
# [http://www.egi.eu/how-to/get_a_certificate.html Obtain a personal X.509 access certificate] from a recognised Certification Authority (unless you have one already).
# Join an existing VO, or form a VO if none of the existing ones suit your purpose"
## The [https://perun.metacentrum.cz/cert/registrar/?vo=fedcloud.egi.eu fedcloud.egi.eu] Virtual Organisation serves as a test ground for users to try the EGI cloud and to prototype and validate applications. It can be used for up to 6 month by any new user.
## You can search for and join other [http://operations-portal.egi.eu/vo/search established VOs]. (Filter for 'cloud' in the middleware column)
 
IaaS cloud resources can expose two types of interfaces towards users (one or the other or both - depending on the cloud provider):
* '''Open Standard interfaces''': OCCI ([http://occi-wg.org/ Open Cloud Computing Interface]) to manage compute, blocks storage and network resources. This interface set are currently exposed by all of the OpenNebula and Synnefo cloud providers, and some of the OpenStack providers.
* '''OpenStack interfaces''': The native OpenStack interfaces (with X.509 authentication). These interfaces are currently exposed by all of the OpenStack-based EGI cloud providers.
 
The user can interact with IaaS cloud resources via programming APIs, command line interfaces or Web dashboards. The different access modes are summarized in the following table:
 
{| cellspacing="5" cellpadding="5" border="0" class="wikitable"
|-
|-
! Federation Service
|
! Role within the federation
| '''Open Standards interface'''
! Existing technical solution in EGI
| '''OpenStack interface'''
|-
|-
! Service Registry
| '''API level access'''
| A registry where all the federated sites and services are registered and state their capabilities. The registry provides the ‘big picture view’ about the federation for both human users and online services (such as service monitors).
| [[Federated_Cloud_APIs_and_SDKs#API|OCCI]]
|GOCDB
| [[Federated_Cloud_APIs_and_SDKs#API_2|OpenStack Compute && Openstack Object Storage]]
|-
|-
!Information System
| '''Command Line access'''
|A database that provides real-time view about the actual capabilities and load of federation participants. Can be used by both human users and online services.
| [[Federated_Cloud_APIs_and_SDKs#CLI|rOCCI-cli]]
|BDII
| [[Federated_Cloud_APIs_and_SDKs#CLI_2|OpenStack CLI with VOMS authentication plugin]]
|-
|-
! Virtual Machine Image Catalogue
| '''Web dashboard access'''
| A catalogue of Virtual Machine Images (VMIs) that encapsulate those software configurations that is useful and relevant for the given community (typically pre-configured scientific models and algorithms).
| AppDB VMOps Dashboard (in final test)
|AppDB
| OpenStack Horizon (production instance under deployment)
|-
! Image replication mechanism
| A system that automatically replicates VMIs from the federation VMI catalogue to each of the member sites, as well as removes them when needed. Automated replication can ensure consistency of capabilities across sites and is very often coupled with a VMI vetting process to ensure that only properly working, and relevant VMIs are replicated to the cloud sites of the community.
| vmcatcher/vmcaster
|-
! Single sign-on for users
| Ensuring that users of the federation need to register for access only once before they can use the federated services. Single sign-on is increasingly implemented in the form of identity federations in both industry and academia.
| IGTF X509 proxies with VOMS extensions
|-
! Integrated view about resource/service usage
|A system that pulls together usage (accounting) information from the federated sites and services, integrates the data and presents them in such a way that both individual users and communities can monitor their own resource/service usage across the whole federation.
| Cloud Usage Record, APEL Accounting repository and portal
|-
! Integrated interfaces or user environments
|Having interfaces through which users and user applications can interact with the services offered by the various cloud providers. In case of an IaaS cloud federation these interfaces offer compute, storage and network management capabilities.
|OCCI API and OpenStack API
|-
!Availability Monitoring
| Use a shared system to monitor and collect availability and reliability statistics about the distributed cloud service providers and to retrieve this information programmatically.
| ARGO monitoring system
|-
! Federated service management tools
| A set of processes, policies, activities and supporting tools customized to the federated cloud.
| EGI federated service management
|}
|}


= EGI cloud realms =
== Typical usage models with examples ==
 
The flexibility of the Infrastructure as a Service EGI cloud can benefit various use cases and usage models. Besides serving compute/data intensive analysis workflows, Web services and interactive applications can be also integrated with and hosted on this infrastructure. Contextualisation and other deployment features can help application operators fine tune services in the cloud, meeting software (OS and software packages), hardware (number of cores, amount of RAM, etc.) and other types of needs (e.g. orchestration, scalability).
 
Since the opening of the EGI Federated Cloud, the following typical usage models have emerged:
 
*'''Service hosting''': the EGI Federated Cloud can be used to hosts any IT service as web servers, databases, etc. Cloud features, as elasticity, can help users to provide better performance and reliable services.
** Example: [https://www.egi.eu/use-cases/scientific-applications-tools/nbis-toolkit/ NBIS Web services], [https://www.egi.eu/news/peachnote-in-unison-with-egi/ Peachnote analysis platform]
*'''Compute and data intensive''': applications needing considerable amount of resources in term of computation and/or memory and/or intensive I/O. Ad-hoc computing environments can be created in the FedCloud sites also to satisfy very hard HW resource requirements.
** Example: [https://www.egi.eu/news/new-egi-use-case-a-close-look-at-the-amatrice-earthquake/ VERCE platform], [https://www.egi.eu/use-cases/research-stories/the-genetics-of-salmonella-infections/ The Genetics of Salmonella Infections], [https://www.egi.eu/use-cases/research-stories/new-viruses-implicated-in-fatal-snake-disease/ The Chipster Platform]
*'''Datasets repository''': the EGI Federated Cloud can be used to store and manage large datasets exploiting the big amount of disk storage available in the Federation.
*'''Disposable and testing environments''': environments for training or testing new developments.
** Example: [[Training_infrastructure|Events conducted on the cloud-based EGI Training Infrastructure]]
 
 
A theoretical setup combining multiple usage models:
[[Image:FedCloudUsageModels.png|center|550px|FedCloudUsageModels.png]] <br>
 
== User guides ==
 
=== User authentication and Virtual Organisations ===
 
EGI provides users with a single sign-on mechanism to access the services of the federated cloud. The single sign-on mechanism is based on X.509 certificates and Virtual Organisations (VOs). Before using EGI Federated Cloud IaaS service you will need to:
 
# [http://www.egi.eu/how-to/get_a_certificate.html Obtain a personal X.509 access certificate] from a recognised Certification Authority (unless you have one already).
# Join an existing VO, or form a VO if none of the existing ones suit your purpose"
## The [https://perun.metacentrum.cz/cert/registrar/?vo=fedcloud.egi.eu fedcloud.egi.eu] Virtual Organisation serves as a test ground for users to try the EGI cloud and to prototype and validate applications. It can be used for up to 6 month by any new user.
## You can search for and join other [http://operations-portal.egi.eu/vo/search established VOs]. (Filter for 'cloud' in the middleware column)
 
Remarks:
* After the 6-month long membership in the fedcloud.egi.eu VO, you will need to move to a production VO, or establish a new VO.
*Grid VOs of EGI can be also extended with cloud resources. Consult with the manager of the VO. 
*If none of the existing VOs matches your use case, then a new VO can be created. Please follow [[PROC14_VO_Registration|VO Registration procedure]]. We can invite sites from the infrastructure to support your VO.
 
=== Creating your first VM - Instantiate an exiting image ===
 
The basic user workflow for starting a Virtual Machine (VM) looks like this:
 
#Browse in the [https://appdb.egi.eu/browse/cloud Application Database Cloud Marketplace] the available Virtual Appliances. Virtual Appliances  are the templates for the root volume of the running instances (Operating System and applications). EGI offers a set of basic images with minimal configuration that can get you started easily, but you can also find complete application stacks. See for example these images:
#*[https://appdb.egi.eu/store/vappliance/egi.ubuntu.14.04 EGI Ubuntu 14], a basic [http://www.ubuntu.com/ Ubuntu 14.04] image
#*[https://appdb.egi.eu/store/vappliance/egi.centos.6 EGI Centos 6], a basic [https://www.centos.org/ CentOS 6] image
#*[https://appdb.egi.eu/store/software/compss.framework COMPSs], a VM for using [[HOWTO14_How_to_use_COMPSs|COMPSs]] programming framework
#*[https://appdb.egi.eu/store/vappliance/biovel.portal BioVel Portal], for executing a http://www.biovel.eu/ biodiversity virtual e-Laboratory]
#Read the description of the appliance for information on how to use it and connect to it once started.
#*You may need to have a ssh key in order to login into the VM, check [[FAQ10#How_can_I_inject_my_public_SSH_key_into_the_machine.3F|the FAQ on how to create one]]
#Get the IDs for starting the appliance at one of the sites supporting it in the "Availability and Usage" tab.
#Get the [[HOWTO11|command line client]] to start the VM using the IDs from previous step
#*Check the [[FAQ10#How_can_I_start_a_VM.3F|How can I start a VM?]] FAQ entry and the [[HOWTO11|command line client how to]] for more information.
#Connect to VM and start using it
#*You may need to allocate a public IP for the VM, check [[FAQ10#How_can_I_assign_a_public_IP_to_a_VM.3F|How can I assign a public IP to a VM]] in the Federated Cloud FAQ
 
Check out these [https://documents.egi.eu/document/2622 tutorial slides] for a practical overview of these steps.
 
=== Advanced topics  ===
 
==== API and SDKs access to Federated Cloud resources ====
 
Besides the command line client, there are several [[Federated Cloud APIs and SDKs|APIs and SDK ready to be used with the EGI Federated Cloud]].Preferred API for EGI federated cloud is OCCI, which can be accesses using using a command-line client (rOCCI), high-level tools or directly implementing the ‘Open Cloud Computing Interface’ (OCCI) into your environment. OpenStack Nova API is also available for OpenStack sites belonging to the Federation.
 
A crash course on how to use programming interfaces of the EGI Federated Cloud, and how these APIs can be used to integrate high-level systems with it is available [https://indico.egi.eu/indico/event/3113/ here]. Please check the [[EGI Federated Cloud jOCCI APIs| EGI Federated Cloud for developers]] guide for details on how to use them.
 
==== Storage ====
 
Every instantiated VM has some disk space provided with it, if you need more storage or need to share data, you can use a cloud storage solution. There are two kind of services: Block Storage and Object Storage. Check the [[HOWTO09|EGI Federated Cloud Storage How To]] for more information.
 
==== Distributed data management with the EGI OpenData platform ====
The EGI OpenData platform is a solution allowing integration of various data repositories available in a distributed infrastructure, offering the capability to make data open, and link them to key open data catalogues following respective guidelines. The core enabling technology of OpenData platform is Onedata, a data management solution that allows a seamless and optimised access to data spread over a distributed infrastructure. Instructions on how to setup a OneData deployment in the EGI Federated Cloud are available [[EGI_Opendata_platform|here]].
 
==== Docker containers, Docker Swarm ====
 
You can [[Federated Cloud Containers|run your docker applications on EGI's Cloud]]. EGI also provides a [[Federated_Cloud_Containers#EGI_Federated_Cloud_clients_docker_image|docker image with the clients]] so you can test them easily.
 
==== High level tools: Orchestrators, Platforms/Software as a Service ====
 
Read ''[[Federated Cloud PaaS|this guidance]]'' about strategies of porting applications to the EGI Federated Cloud. The guide also includes references to high level user environments (orchestrators, Platform/Software as a Service) that can simplify the application integration and operation process for you. These environments offer high level abstractions and services on top of the baseline 'Infrastructure as a Service' cloud.
 
==== Running Hadoop applications ====
 
Using a WS-PGRADE gateway that is connected to the EGI Federated cloud, it is possible to deploy Hadoop clusters on EGI Federated Cloud resources, to execute Hadoop applications on those clusters and finally to release resources after application execution. The concept is outlined on [https://indico.egi.eu/indico/event/2931/material/slides/ these PPT slides]. A user manual is [https://indico.egi.eu/indico/event/2931/material/0/ available here (v1.3)].
 
==== GPGPUs (pre-production!) ====
 
'''PRE-PRODUCTION''' GPGPUs-enabled cloud resources are available in selected sites of the EGI Federated Cloud. Check the [[Federated_Cloud_GPGPU|GPGPU guide on FedCloud]] for details on how to access them.
 
==== Customized Virtual Appliances  ====
 
You can prepare fully customised Virtual Appliances and deploy them to the sites:
 
#Prepare Virtual Machine Images (VMIs) that encapsulate your application. See the [[HOWTO10|application porting tutorial]] for tips.
#Make the VMI available online, for example in the [http://appliance-repo.egi.eu/images/ EGI Appliance Repository]
#Register the VMIs as Virtual Appliances in the [http://appdb.egi.eu EGI Applications Database] (for howto please click [https://wiki.appdb.egi.eu/main:faq:how_to_register_a_virtual_appliance here])
#Once your VA is published (see howto [https://wiki.appdb.egi.eu/main:guides:guide_for_managing_virtual_appliance_versions_using_the_portal here]),inform the Manager of your VO through Applications Database about it (see howto [https://wiki.appdb.egi.eu/main:guides:notify_virtual_organization_representatives here]). He/she will include your images in the VO-wide image list, so these will be deployed on the Federated Cloud sites of your VO.
 
==== Running Galaxy Workflows with EC3 ====
 
Using the [http://servproject.i3m.upv.es/ec3/ EC3] open-source software platform, users can deploy elastic clusters on demand and dynamically deploy complex scientific virtual computing infrastructures on top of Infrastructure as a Service Clouds. More details on how to use Galaxy workflows on the platform are described in this wiki. Please check the [[Galaxy workflows with EC3 | Galaxy workflows in EGI with EC3]] guide for further details.
 
==== How SMEs Can Use EGI's Cloud for Computer-Aided Engineering (CAE) ====
Please check how SMEs can use [[Computer-Aided Engineering (CAE) in the EGI Federated Cloud | Computer-Aided Engineering (CAE) in the EGI Federated Cloud]], with the example of OpenFOAM software containers.
 
== Useful resources  ==
 
*[[FAQ10|FedCloud FAQ page]]
*[http://www.egi.eu/how-to/get_a_certificate.html How to get a certificate (to access Federated Cloud resources)]
*[https://wiki.appdb.egi.eu/ How to use the Applications Database Cloud Marketplace]
*[[HOWTO10|Porting your application/web service to the EGI Federated Cloud]]
*[[Federated_Cloud_Operation#Current_Resource_Providers|List and details about certified cloud resource providers]]
*[[Cloud_SAM_tests|Monitor tests performed by EGI on cloud resources]]
**[https://cloudmon.egi.eu/nagios/cgi-bin/status.cgi?host=all&amp;sorttype=2&amp;sortoption=2 Current status of Federated Cloud resources (NAGIOS test results)]
**[https://cloudmon.egi.eu/nagios/cgi-bin/status.cgi?servicegroup=SERVICE_org.openstack.nova&amp;style=overview Current status of OpenStack interfaces in the EGI federated cloud]
**[http://argo.egi.eu/lavoisier/status_report-site?report=Cloud&accept=html Status of cloud sites in the EGI federated cloud in the ARGO system]
 
== User support  ==


The EGI Federated Cloud can support multiple cloud federations (community specific, private or public). Based on the EGI federation services and custom external solutions, any scientific community can create a federated cloud. Each community or e-infrastructure that wants to build a cloud federation decides the services required to support their computational needs. Because these cloud federations are largely built from tools and services of the same solution portfolio, they can maintain the portfolio together; they can share best practices, and can offer user support and training in a collaborative fashion.
=== Technical support ===


EGI currently operates two realms: the '''Open Standards Realm''' and the '''OpenStack Realm'''. Both are completely integrated with the EGI federator services described above but use different interfaces to offer the IaaS capabilities to the users: the Open Standards Realm uses OCCI standard (supported by providers with OpenNebula, OpenStack or Synnefo cloud management frameworks), while the OpenStack Realm uses OpenStack native Nova API (support limited to OpenStack providers).  This OpenStack Realm was introduced in the federation during November 2015 and most of the resource providers already in the Open Standards Realm using OpenStack have started to provide this API along with the existing OCCI interface.
Users' technical support is provided via the [mailto:usupport@egi.eu EGI support contact].  


{| class="wikitable" style="margin: auto;"
=== Helpdesk  ===
|-
 
! Service
Technical problems and questions relating to the use of the EGI Federated Cloud can be reported and dealt with through the [https://ggus.eu/ EGI Helpdesk ticketing system].
! Open Standards Realm
 
! OpenStack Realm
'''Note: '''Please choose 'Federated cloud' in the 'Type of problem' field of the ticket submission form!
|-
 
! Service Registry
=== Feedback and open issues  ===
| colspan="2" style="text-align: center;" | GOCDB
 
|-
A list of open-issue and feedbacks reported by the FedCloud users is available at [[Federated Cloud Users Feedback and open issues|this page]].
! Single sign-on
 
| colspan="2" style="text-align: center;" | X.509 proxies with VOMS extensions
== Technical background  ==
|-
 
! Accounting
Cloud providers in the EGI Federated Cloud use hardware virtualization technologies to host software on their resources. The cloud management platforms that make this possible can [[Federated_Cloud_Operation#Current_Resource_Providers|vary from site to site]], but they all enable the provisioning of virtualized computing, storage and networking resources, thus they empower scientific groups to setup and operate domain specific services, applications and simulations on these resources. Read more about the [[Federated Cloud Technology|technology that drives the Federated Cloud]].
| colspan="2" style="text-align: center;" | Cloud Usage Record
|-
! Information discovery
| colspan="2" style="text-align: center;"| BDII
|-
! VM Image catalogue
| colspan="2" style="text-align: center;" | AppDB
|-
! VM Image distribution
| colspan="2" style="text-align: center;" | vmcatcher/vmcaster
|-
! IaaS interface
| style="text-align: center;" | OCCI
| style="text-align: center;" | OpenStack Compute API
|-
! Monitoring
| ARGO (OCCI specific probes)
| ARGO (OpenStack specific probes)
|}

Revision as of 13:20, 20 January 2017

Enol Fernandez - EGI

https://www.egi.eu/sso/userDetail/enolfc

Overview For users For resource providers Infrastructure status Site-specific configuration Architecture




Introduction


Technical consultancy and support is available via the EGI.eu Support Team


The EGI Cloud federation is an hybrid, Infrastructure as a Service cloud composed by public, community and private cloud. The clouds are federated with the use of the 'EGI Core Infrastructure Platform', offering a scalable compute and storage infrastructure for scientific applications, services, workloads. The main capabilities of the system are:

Elastic computing infrastructure.
Execute compute and data intensive workloads (both batch and interactive), host long-running services (e.g. web servers, databases or applications servers), or create disposable testing and development environments in VMs and containers. Scale your application or service within a single provider, or across multiple providers of the federation (within providers of your virtual organisation). Select VM configurations (CPU, memory, disk) and ready-to-deploy application VMs that best fit your needs.
VM image sharing and distribution
Easily share and distribute customised VM images to multiple clouds via the open 'Applications Database' library of Virtual Appliances. Community curated VMs and VM appliances are securely and automatically replicated across the infrastructure. The EGI User Community Support Team provides generic, baseline VM images, user communities can offer more specialised VMs and applications.
Unified view of federation
The EGI Cloud provides: Single sing-on (SSO) for authentication and authorisation across all resource providers; Federated accounting with an integrated view of the the resource and service usage; Distributed information system for delivering a real-time view of the capabilities; and Federated monitoring to compute metrics for availability and reliability of the services.
Beyond VMs
Run docker applications on the EGI resources; Use one of the already integrated PaaS and SaaS solution; Follow our user guides to deploy Hadoop, Docker Swarm, to access Object Storage and many more...


Access modes

EGI Infrastructure as a Service (IaaS) Cloud Resources can be accessed through 'Virtual Organizations' (VOs). A VO is a grouping of IaaS cloud provider from the EGI federation, who allocate capacity for a specific user group. Users with similar interest/requirements can join or form a VO to gather resources from EGI cloud providers - typically for a given project, experiment or use case. There are generic VOs too, for example the 'fedcloud.egi.eu' VO, which is open for any user who wants to experiment with the EGI Federated Cloud. You have to join a VO before you can interact with EGI IaaS cloud resources, while higher level services (PaaS, SaaS) do not always require VO membership.

VO membership is controlled in EGI by X.509 certificates. To access the EGI IaaS cloud you need to:

  1. Obtain a personal X.509 access certificate from a recognised Certification Authority (unless you have one already).
  2. Join an existing VO, or form a VO if none of the existing ones suit your purpose"
    1. The fedcloud.egi.eu Virtual Organisation serves as a test ground for users to try the EGI cloud and to prototype and validate applications. It can be used for up to 6 month by any new user.
    2. You can search for and join other established VOs. (Filter for 'cloud' in the middleware column)

IaaS cloud resources can expose two types of interfaces towards users (one or the other or both - depending on the cloud provider):

  • Open Standard interfaces: OCCI (Open Cloud Computing Interface) to manage compute, blocks storage and network resources. This interface set are currently exposed by all of the OpenNebula and Synnefo cloud providers, and some of the OpenStack providers.
  • OpenStack interfaces: The native OpenStack interfaces (with X.509 authentication). These interfaces are currently exposed by all of the OpenStack-based EGI cloud providers.

The user can interact with IaaS cloud resources via programming APIs, command line interfaces or Web dashboards. The different access modes are summarized in the following table:

Open Standards interface OpenStack interface
API level access OCCI OpenStack Compute && Openstack Object Storage
Command Line access rOCCI-cli OpenStack CLI with VOMS authentication plugin
Web dashboard access AppDB VMOps Dashboard (in final test) OpenStack Horizon (production instance under deployment)

Typical usage models with examples

The flexibility of the Infrastructure as a Service EGI cloud can benefit various use cases and usage models. Besides serving compute/data intensive analysis workflows, Web services and interactive applications can be also integrated with and hosted on this infrastructure. Contextualisation and other deployment features can help application operators fine tune services in the cloud, meeting software (OS and software packages), hardware (number of cores, amount of RAM, etc.) and other types of needs (e.g. orchestration, scalability).

Since the opening of the EGI Federated Cloud, the following typical usage models have emerged:

  • Service hosting: the EGI Federated Cloud can be used to hosts any IT service as web servers, databases, etc. Cloud features, as elasticity, can help users to provide better performance and reliable services.
  • Compute and data intensive: applications needing considerable amount of resources in term of computation and/or memory and/or intensive I/O. Ad-hoc computing environments can be created in the FedCloud sites also to satisfy very hard HW resource requirements.
  • Datasets repository: the EGI Federated Cloud can be used to store and manage large datasets exploiting the big amount of disk storage available in the Federation.
  • Disposable and testing environments: environments for training or testing new developments.


A theoretical setup combining multiple usage models:

FedCloudUsageModels.png


User guides

User authentication and Virtual Organisations

EGI provides users with a single sign-on mechanism to access the services of the federated cloud. The single sign-on mechanism is based on X.509 certificates and Virtual Organisations (VOs). Before using EGI Federated Cloud IaaS service you will need to:

  1. Obtain a personal X.509 access certificate from a recognised Certification Authority (unless you have one already).
  2. Join an existing VO, or form a VO if none of the existing ones suit your purpose"
    1. The fedcloud.egi.eu Virtual Organisation serves as a test ground for users to try the EGI cloud and to prototype and validate applications. It can be used for up to 6 month by any new user.
    2. You can search for and join other established VOs. (Filter for 'cloud' in the middleware column)

Remarks:

  • After the 6-month long membership in the fedcloud.egi.eu VO, you will need to move to a production VO, or establish a new VO.
  • Grid VOs of EGI can be also extended with cloud resources. Consult with the manager of the VO.
  • If none of the existing VOs matches your use case, then a new VO can be created. Please follow VO Registration procedure. We can invite sites from the infrastructure to support your VO.

Creating your first VM - Instantiate an exiting image

The basic user workflow for starting a Virtual Machine (VM) looks like this:

  1. Browse in the Application Database Cloud Marketplace the available Virtual Appliances. Virtual Appliances are the templates for the root volume of the running instances (Operating System and applications). EGI offers a set of basic images with minimal configuration that can get you started easily, but you can also find complete application stacks. See for example these images:
  2. Read the description of the appliance for information on how to use it and connect to it once started.
  3. Get the IDs for starting the appliance at one of the sites supporting it in the "Availability and Usage" tab.
  4. Get the command line client to start the VM using the IDs from previous step
  5. Connect to VM and start using it

Check out these tutorial slides for a practical overview of these steps.

Advanced topics

API and SDKs access to Federated Cloud resources

Besides the command line client, there are several APIs and SDK ready to be used with the EGI Federated Cloud.Preferred API for EGI federated cloud is OCCI, which can be accesses using using a command-line client (rOCCI), high-level tools or directly implementing the ‘Open Cloud Computing Interface’ (OCCI) into your environment. OpenStack Nova API is also available for OpenStack sites belonging to the Federation.

A crash course on how to use programming interfaces of the EGI Federated Cloud, and how these APIs can be used to integrate high-level systems with it is available here. Please check the EGI Federated Cloud for developers guide for details on how to use them.

Storage

Every instantiated VM has some disk space provided with it, if you need more storage or need to share data, you can use a cloud storage solution. There are two kind of services: Block Storage and Object Storage. Check the EGI Federated Cloud Storage How To for more information.

Distributed data management with the EGI OpenData platform

The EGI OpenData platform is a solution allowing integration of various data repositories available in a distributed infrastructure, offering the capability to make data open, and link them to key open data catalogues following respective guidelines. The core enabling technology of OpenData platform is Onedata, a data management solution that allows a seamless and optimised access to data spread over a distributed infrastructure. Instructions on how to setup a OneData deployment in the EGI Federated Cloud are available here.

Docker containers, Docker Swarm

You can run your docker applications on EGI's Cloud. EGI also provides a docker image with the clients so you can test them easily.

High level tools: Orchestrators, Platforms/Software as a Service

Read this guidance about strategies of porting applications to the EGI Federated Cloud. The guide also includes references to high level user environments (orchestrators, Platform/Software as a Service) that can simplify the application integration and operation process for you. These environments offer high level abstractions and services on top of the baseline 'Infrastructure as a Service' cloud.

Running Hadoop applications

Using a WS-PGRADE gateway that is connected to the EGI Federated cloud, it is possible to deploy Hadoop clusters on EGI Federated Cloud resources, to execute Hadoop applications on those clusters and finally to release resources after application execution. The concept is outlined on these PPT slides. A user manual is available here (v1.3).

GPGPUs (pre-production!)

PRE-PRODUCTION GPGPUs-enabled cloud resources are available in selected sites of the EGI Federated Cloud. Check the GPGPU guide on FedCloud for details on how to access them.

Customized Virtual Appliances

You can prepare fully customised Virtual Appliances and deploy them to the sites:

  1. Prepare Virtual Machine Images (VMIs) that encapsulate your application. See the application porting tutorial for tips.
  2. Make the VMI available online, for example in the EGI Appliance Repository
  3. Register the VMIs as Virtual Appliances in the EGI Applications Database (for howto please click here)
  4. Once your VA is published (see howto here),inform the Manager of your VO through Applications Database about it (see howto here). He/she will include your images in the VO-wide image list, so these will be deployed on the Federated Cloud sites of your VO.

Running Galaxy Workflows with EC3

Using the EC3 open-source software platform, users can deploy elastic clusters on demand and dynamically deploy complex scientific virtual computing infrastructures on top of Infrastructure as a Service Clouds. More details on how to use Galaxy workflows on the platform are described in this wiki. Please check the Galaxy workflows in EGI with EC3 guide for further details.

How SMEs Can Use EGI's Cloud for Computer-Aided Engineering (CAE)

Please check how SMEs can use Computer-Aided Engineering (CAE) in the EGI Federated Cloud, with the example of OpenFOAM software containers.

Useful resources

User support

Technical support

Users' technical support is provided via the EGI support contact.

Helpdesk

Technical problems and questions relating to the use of the EGI Federated Cloud can be reported and dealt with through the EGI Helpdesk ticketing system.

Note: Please choose 'Federated cloud' in the 'Type of problem' field of the ticket submission form!

Feedback and open issues

A list of open-issue and feedbacks reported by the FedCloud users is available at this page.

Technical background

Cloud providers in the EGI Federated Cloud use hardware virtualization technologies to host software on their resources. The cloud management platforms that make this possible can vary from site to site, but they all enable the provisioning of virtualized computing, storage and networking resources, thus they empower scientific groups to setup and operate domain specific services, applications and simulations on these resources. Read more about the technology that drives the Federated Cloud.

Retrieved from "https://wiki.egi.eu/w/index.php?title=User:Enolfc&oldid=92659"