Alert.png The wiki is deprecated and due to be decommissioned by the end of September 2022.
The content is being migrated to other supports, new updates will be ignored and lost.
If needed you can get in touch with EGI SDIS team using operations @ egi.eu.

Difference between revisions of "Federated Cloud Technology"

From EGIWiki
Jump to navigation Jump to search
Line 5: Line 5:
= Introduction  =
= Introduction  =


The current high throughput model of grid computing has proven to be extremely powerful for a small number of different communities. These communities have thrived in the current grid environment but the uptake of e-infrastructure by other communities has been limited. EGI has therefore strategically decided to investigate how it could broaden the uptake of its infrastructure to support other research communities and application design models, that would not only be able to take advantage of the existing functionality and investment already made in EGI’s Core Infrastructure, but also support different research communities and their applications on the current production infrastructure than it was previously able to.  
The EGI Federated Cloud is shifting its model to become a wider collaboration to include several cloud federations (community, private or public clouds), each developing, operating and exploiting clouds according to the preferences, choices and constraints set by its members and users. The EGI Federated Cloud Collaboration acts as a meeting point for these federations to share technologies, services and operational practices among each in order e.g. to lower maintenance cost and to increase sustainability and the speed of innovation.
 
The utilization of Virtualization and Infrastructure as a Service (IaaS) cloud computing was a clear candidate to enable this transformation. It was also clear that with a number of different open source technologies already in use across a number of different resource providers, that it would not be possible to mandate a single software stack. Instead, following on from a number of different activities already on-going in Europe including SIENA1, an approach that required the utilization of open standards where available and, where not, methods that have broad acceptance in the e-infrastructure community were essential.  
 
This led to the current EGI Cloud Federation model, a model based on the needs of each community using the IaaS cloud as described below.  


= Federation Model  =
= Federation Model  =


The EGI Federated Cloud is a set of Resource Providers clusters targeting global or specific user communities: a Public Federated Cloud open for any researcher and various Community and Private Clouds accessible to one or more selected Virtual Organizations. Every cluster follows the model shown in the figure:  
The EGI Federated Cloud is a collaboration of clouds targeting global or specific user communities. Each cloud follows the model shown in the figure:  


[[Image:Federated_Cloud_Model.png|thumb|center|400px|Federated Cloud Model]]  
[[Image:Federated_Cloud_Model.png|thumb|center|400px|Federated Cloud Model]]  


The federation of IaaS Cloud resources in EGI is built upon the extensive autonomy of Resource Providers in terms of ownership of exposed resources. The federation model for distributed IaaS Cloud resources allows a lightweight aggregation of local Cloud resources into the EGI Cloud Infrastructure Platform (CLIP). At the heart of the federation are the locally deployed Cloud Management stacks. In compliance with the Cloud computing model, the EGI CLIP does not mandate deploying any particular or specific Cloud Management stack; it is the responsibility of the Resource Providers to investigate, identify and deploy the solution that fits best their individual needs for as long as the offered services implement the required interfaces and domain languages. These interfaces and domain languages, and the interoperability of their implementation with other solutions are the focus of the federation.  
The federation of IaaS Cloud resources in EGI is built upon the extensive autonomy of Resource Providers in terms of ownership of exposed resources. The federation model for distributed IaaS Cloud resources allows a lightweight aggregation of local Cloud resources into the EGI Cloud Infrastructure Platform (CLIP). At the heart of the federation are the locally deployed Cloud Management stacks.  


Every cluster of the federation defines the actual level of integration with the EGI Federated Operations Services, the EGI marketplace and their Federated AAI architecture and technologies. The IaaS services of the cluster must be offered with interfaces that assure the interoperability within the community and whenever possible open standards should be used. The EGI Federated Operations services include the following:
Every member of the collaboration defines the actual level of integration with the EGI Federated Operations Services, the EGI marketplace and their Federated AAI architecture and technologies. The IaaS services of the cluster should  be offered with interfaces that assure the interoperability within the community and whenever possible open standards should be used. The EGI Federated Operations services include the following:
* [[Federated Cloud Architecture#Information_discovery:_BDII|Information discovery (BDII)]]: The EGI Information Discovery allows users and tools to look for the services that provide the capabilities and the resources to run their activities. It's based on [https://www.ogf.org/documents/GFD.147.pdf OGF GLUE2] and uses LDAP as protocol.  
* [[Federated Cloud Architecture#Information_discovery:_BDII|Information discovery (BDII)]]: The EGI Information Discovery allows users and tools to look for the services that provide the capabilities and the resources to run their activities. It's based on [https://www.ogf.org/documents/GFD.147.pdf OGF GLUE2] and uses LDAP as protocol.  
* [[Federated Cloud Architecture#Accounting|Accounting]]: EGI collects all the CPU and storage information to the [[Accounting Repository]]. These data is also provided in a web-based [[Accounting Portal]] where users and VO managers have a detailed view of the resources consumed.
* [[Federated Cloud Architecture#Accounting|Accounting]]: EGI collects all the CPU and storage information to the [[Accounting Repository]]. These data is also provided in a web-based [[Accounting Portal]] where users and VO managers have a detailed view of the resources consumed.
Line 27: Line 23:
The EGI MarketPlace provides [[Federated Cloud Architecture#VM_Image_management|VM Image management]] implemented in the [[AppDB|EGI Applications Database]]. AppDB provides a central registration point for virtual appliances which can be endorsed by VO managers and automatically and securely distributed to any resource provider subscribed to the VO virtual appliance lists.
The EGI MarketPlace provides [[Federated Cloud Architecture#VM_Image_management|VM Image management]] implemented in the [[AppDB|EGI Applications Database]]. AppDB provides a central registration point for virtual appliances which can be endorsed by VO managers and automatically and securely distributed to any resource provider subscribed to the VO virtual appliance lists.


The EGI Cloud Federation includes a key member, the EGI Public Cloud, that is open to any research community and  is completely integrated with the EGI production infrastructure.
The table below summarizes the differences between the public and community federated clouds:
The table below summarizes the differences between the public and community federated clouds:
{| class="wikitable" style="margin: auto;"
{| class="wikitable" style="margin: auto;"
Line 53: Line 50:
== Public Federated Cloud  ==
== Public Federated Cloud  ==


The following figure depicts the concrete model for the public federated cloud, open to any research community, which is completely integrated with all EGI Core Services and EGI Marketplace, uses open standards (if available) for all the public APIs and uses the current AAI schema of EGI based on X.509 proxies with VOMS extensions.  
The following figure depicts the specific model for the public federated cloud, open to any research community, which is completely integrated with all EGI Core Services and EGI Marketplace, uses open standards (if available) for all the public APIs and uses the current AAI schema of EGI based on X.509 proxies with VOMS extensions.  


[[Image:Public_Federated_Cloud_Model.png|thumb|center|800px|Public Federated Cloud Architecture]]  
[[Image:Public_Federated_Cloud_Model.png|thumb|center|800px|Public Federated Cloud Architecture]]  
This federation does not mandate deploying any particular or specific Cloud Management stack; it is the responsibility of the Resource Providers to investigate, identify and deploy the solution that fits best their individual needs for as long as the offered services implement the required interfaces and domain languages. These interfaces and domain languages, and the interoperability of their implementation with other solutions are the focus of the federation.


Resource providers in the public Federated Cloud must offer one or both of the following IaaS cloud capabilities by implementing these interfaces:  
Resource providers in the public Federated Cloud must offer one or both of the following IaaS cloud capabilities by implementing these interfaces:  
Line 114: Line 113:
== Community Federated Cloud ==
== Community Federated Cloud ==


A community cloud is accessible to a selected group of users or Virtual Organizations. These clouds have a looser federation model hence the level of integration with the EGI services depends on the needs of the community it serves. Community clouds may also choose the interfaces to access the IaaS capabilities as long as the selected interfaces assure the interoperability within the federated resources. The standards used in the Public Federated Cloud are recommended if there are no requirements that prevent their usage.
A community cloud is accessible to a selected group of users or Virtual Organizations. These clouds may have a looser federation model hence the level of integration with the EGI services depends on the needs of the community it serves. Community clouds may also choose the interfaces to access the IaaS capabilities as long as the selected interfaces assure the interoperability within the federated resources. The standards used in the Public Federated Cloud are recommended if there are no requirements that prevent their usage.


Community clouds may profit the existing developments for the integration of Cloud Management stacks into the public federated cloud documented in the [[Federated Cloud resource providers support]] page.
Community clouds may profit the existing developments for the integration of Cloud Management stacks into the public federated cloud documented in the [[Federated Cloud resource providers support]] page.


[[Category:Federated_Cloud]]
[[Category:Federated_Cloud]]

Revision as of 18:31, 24 June 2015

Overview For users For resource providers Infrastructure status Site-specific configuration Architecture




Baustelle.png This page is under construction.


Introduction

The EGI Federated Cloud is shifting its model to become a wider collaboration to include several cloud federations (community, private or public clouds), each developing, operating and exploiting clouds according to the preferences, choices and constraints set by its members and users. The EGI Federated Cloud Collaboration acts as a meeting point for these federations to share technologies, services and operational practices among each in order e.g. to lower maintenance cost and to increase sustainability and the speed of innovation.

Federation Model

The EGI Federated Cloud is a collaboration of clouds targeting global or specific user communities. Each cloud follows the model shown in the figure:

Federated Cloud Model

The federation of IaaS Cloud resources in EGI is built upon the extensive autonomy of Resource Providers in terms of ownership of exposed resources. The federation model for distributed IaaS Cloud resources allows a lightweight aggregation of local Cloud resources into the EGI Cloud Infrastructure Platform (CLIP). At the heart of the federation are the locally deployed Cloud Management stacks.

Every member of the collaboration defines the actual level of integration with the EGI Federated Operations Services, the EGI marketplace and their Federated AAI architecture and technologies. The IaaS services of the cluster should be offered with interfaces that assure the interoperability within the community and whenever possible open standards should be used. The EGI Federated Operations services include the following:

  • Information discovery (BDII): The EGI Information Discovery allows users and tools to look for the services that provide the capabilities and the resources to run their activities. It's based on OGF GLUE2 and uses LDAP as protocol.
  • Accounting: EGI collects all the CPU and storage information to the Accounting Repository. These data is also provided in a web-based Accounting Portal where users and VO managers have a detailed view of the resources consumed.
  • Monitoring (SAM): Resources in the infrastructure are monitored via the Service Availability Monitoring (SAM) system that periodically executes tests on the provided services. The results of the tests are used to calculate Availability and Reliability of the resources.
  • Central service registry (GOCDB): GOCDB contains general information about the sites participating to the production infrastructure. It collects information about different entities such as the Operations Centres, the Resource Centres, service endpoints and the contact information and roles of people responsible for operations at different levels.

The EGI MarketPlace provides VM Image management implemented in the EGI Applications Database. AppDB provides a central registration point for virtual appliances which can be endorsed by VO managers and automatically and securely distributed to any resource provider subscribed to the VO virtual appliance lists.

The EGI Cloud Federation includes a key member, the EGI Public Cloud, that is open to any research community and is completely integrated with the EGI production infrastructure. The table below summarizes the differences between the public and community federated clouds:

Public Cloud Community Cloud
Users Open to any researcher community Targets specific community (or set of communities)
IaaS interfaces Open standard interfaces for providing the IaaS capabilities Community chosen interfaces, standards are recommended
VM Image Management Fully integrated Optional integration
EGI services integration Tight integration with EGI core services (information system, monitoring, accounting, GOCDB) Looser federation profile, may only use a subset of the EGI core services

Public Federated Cloud

The following figure depicts the specific model for the public federated cloud, open to any research community, which is completely integrated with all EGI Core Services and EGI Marketplace, uses open standards (if available) for all the public APIs and uses the current AAI schema of EGI based on X.509 proxies with VOMS extensions.

Public Federated Cloud Architecture

This federation does not mandate deploying any particular or specific Cloud Management stack; it is the responsibility of the Resource Providers to investigate, identify and deploy the solution that fits best their individual needs for as long as the offered services implement the required interfaces and domain languages. These interfaces and domain languages, and the interoperability of their implementation with other solutions are the focus of the federation.

Resource providers in the public Federated Cloud must offer one or both of the following IaaS cloud capabilities by implementing these interfaces:

Moreover, to federate a resource provider in the public EGI Federated Cloud, it must integrate with the EGI core services and interfaces listed below:

The EGI Federated Clouds Task gives Resource Providers a platform to share their implementation solutions for a commonly deployed specific Cloud Management stack (e.g. OpenNebula and OpenStack). The Federated Cloud resource providers support page is dedicated to the documentation of the steps necessary to integrate a local deployment of a given Cloud Management stack into the EGI Cloud federation. The table below summarizes the current integration level of the participating Cloud Management stacks in the federation:

Cloud Mgmt. Stack Fed. AAI Info. Discovery Monitoring Accounting VM Img. Mgmt. OCCI CDMI
OpenStack Yes Yes Yes Yes Yes Yes In progress
OpenNebula Yes Yes Yes Yes Yes Yes N/A
Synnefo Yes Yes Yes Yes Yes Yes Yes

Community Federated Cloud

A community cloud is accessible to a selected group of users or Virtual Organizations. These clouds may have a looser federation model hence the level of integration with the EGI services depends on the needs of the community it serves. Community clouds may also choose the interfaces to access the IaaS capabilities as long as the selected interfaces assure the interoperability within the federated resources. The standards used in the Public Federated Cloud are recommended if there are no requirements that prevent their usage.

Community clouds may profit the existing developments for the integration of Cloud Management stacks into the public federated cloud documented in the Federated Cloud resource providers support page.