Alert.png The wiki is deprecated and due to be decommissioned by the end of September 2022.
The content is being migrated to other supports, new updates will be ignored and lost.
If needed you can get in touch with EGI SDIS team using operations @ egi.eu.

Difference between revisions of "Federated Cloud Technology"

From EGIWiki
Jump to navigation Jump to search
Line 1: Line 1:
{{Fedcloud_Menu}} {{TOC_right}}  
{{Fedcloud_Menu}} {{TOC_right}}  


<br>
{{Under_construction}}


= Introduction  =
= Introduction  =
Line 13: Line 13:
= Federation Model  =
= Federation Model  =


The EGI Federated Cloud is a set of clusters targetting global or specific user communities: a public federated cloud open for any researchers and community clouds accesible to one or more selected Virtual Organisations. Every cluster follows the model shown in the figure  
The EGI Federated Cloud is a set of clusters targeting global or specific user communities: a public federated cloud open for any researchers and various community clouds accessible to one or more selected Virtual Organizations. Every cluster follows the model shown in the figure:


[[Image:Federated_Cloud_Model.png|thumb|center|400px|Federated Cloud Model]]


'''FIGURE'''
The federation of IaaS Cloud resources in EGI is built upon the extensive autonomy of Resource Providers in terms of ownership of exposed resources. The federation model for distributed IaaS Cloud resources allows a lightweight aggregation of local Cloud resources into the EGI Cloud Infrastructure Platform (CLIP). At the heart of the federation are the locally deployed Cloud Management stacks. In compliance with the Cloud computing model, the EGI CLIP does not mandate deploying any particular or specific Cloud Management stack; it is the responsibility of the Resource Providers to investigate, identify and deploy the solution that fits best their individual needs for as long as the offered services implement the required interfaces and domain languages. These interfaces and domain languages, and the interoperability of their implementation with other solutions are the focus of the federation.


Every cluster of the federation, defines the actual level of integration with the EGI Federated Operations Services, the EGI marketplace and their Federated AAI architecture and technologies. The IaaS services of the cluster must be offered with interfaces that assure the interoperability within the community and whenever possible open standards should be used.


The federation of IaaS Cloud resources in EGI is built upon the extensive autonomy of Resource Providers in terms of ownership of exposed resources. The federation model for distributed IaaS Cloud resources allows a lightweight aggregation of local Cloud resources into the EGI Cloud Infrastructure Platform (CLIP). At the heart of the federation are the locally deployed Cloud Management stacks. In compliance with the Cloud computing model, the EGI CLIP does not mandate deploying any particular or specific Cloud Management stack; it is the responsibility of the Resource Providers to investigate, identify and deploy the solution that fits best their individual needs for as long as the offered services implement the required interfaces and domain languages. These interfaces and domain languages, and the interoperability of their implementation with other solutions are the focus of the federation.
'''describe what's inside each box of the figure!'''


Every cluster of the federation, defines the actual level of integration with the EGI Federated Opetations Services, the EGI marketplace and their Federated AAI architecture and technologies. The IaaS services of the cluster must be offered with interfaces that assure the interoperability within the community and whenever possible open standards should be used.


== Public Federated Cloud  ==
== Public Federated Cloud  ==
Line 27: Line 28:
The following figure depicts the concrete model for the public federated cloud, open to any research community, which is completely integrated with all EGI Core Services and EGI Marketplace, uses open standards (if available) for all the public APIs and uses the current AAI schema of EGI based on X.509 proxies with VOMS extensions.  
The following figure depicts the concrete model for the public federated cloud, open to any research community, which is completely integrated with all EGI Core Services and EGI Marketplace, uses open standards (if available) for all the public APIs and uses the current AAI schema of EGI based on X.509 proxies with VOMS extensions.  


'''FIGURE'''
[[Image:Public_Federated_Cloud_Model.png|thumb|center|800px|Public Federated Cloud Architecture]]


Resource providers in the public Federated Cloud must offer one or both of the following IaaS cloud capabilities by implementing these interfaces:  
Resource providers in the public Federated Cloud must offer one or both of the following IaaS cloud capabilities by implementing these interfaces:  
Line 39: Line 40:
*[[Federated Cloud Architecture#Virtual_Organisation_Management_.26_AAI:_VOMS|Virtual Organisation Management &amp; AAI: VOMS]]  
*[[Federated Cloud Architecture#Virtual_Organisation_Management_.26_AAI:_VOMS|Virtual Organisation Management &amp; AAI: VOMS]]  
*[[Federated Cloud Architecture#Information_discovery:_BDII|Information discovery: BDII]]  
*[[Federated Cloud Architecture#Information_discovery:_BDII|Information discovery: BDII]]  
*[[Federated Cloud Architecture#Central_service_registry:_GOCDB|Central service registry: GOCDB]]
*[[Federated Cloud Architecture#Monitoring:_SAM|Monitoring: SAM]]  
*[[Federated Cloud Architecture#Monitoring:_SAM|Monitoring: SAM]]  
*[[Federated Cloud Architecture#Accounting|Accounting]]
*[[Federated Cloud Architecture#Accounting|Accounting]]
*[[Federated Cloud Architecture#Central_service_registry:_GOCDB|Central service registry: GOCDB]]


== Community Federated Cloud ==
The EGI Federated Clouds Task gives Resource Providers a platform to share their implementation solutions for a commonly deployed specific Cloud Management stack (e.g. OpenNebula and OpenStack). The [[Federated Cloud resource providers support]] page is dedicated to the documentation of the steps necessary to integrate a local deployment of a given Cloud Management stack into the EGI Cloud federation. The table below summarizes the current integration level of the participating Cloud Management stacks in the federation:
 
= Introduction  =
 
The current high throughput model of grid computing has proven to be extremely powerful for a small number of different communities. These communities have thrived in the current grid environment but the uptake of e-infrastructure by other communities has been limited. EGI has therefore strategically decided to investigate how it could broaden the uptake of its infrastructure to support other research communities and application design models, that would not only be able to take advantage of the existing functionality and investment already made in EGI’s Core Infrastructure, but also support different research communities and their applications on the current production infrastructure than it was previously able to.
 
The utilization of Virtualization and Infrastructure as a Service (IaaS) cloud computing was a clear candidate to enable this transformation. It was also clear that with a number of different open source technologies already in use across a number of different resource providers, that it would not be possible to mandate a single software stack. Instead, following on from a number of different activities already on-going in Europe including SIENA1, an approach that required the utilization of open standards where available and, where not, methods that have broad acceptance in the e-infrastructure community were essential.
 
This led to the current EGI Cloud Federation model, a model based on a set of agreed open standards. More details on the EGI Cloud Federation Model is the paragraph below.
 
= Federation Model  =
 
[[Image:Fedcloud-tf-Technology-EGICloudArchitecture.png|thumb|center|800px|Fedcloud-tf-Technology-EGICloudArchitecture.png]]
 
The federation of IaaS Cloud resources in EGI is built upon the extensive autonomy of Resource Providers in terms of ownership of exposed resources. The federation model for distributed IaaS Cloud resources allows a lightweight aggregation of local Cloud resources into the EGI Cloud Infrastructure Platform (CLIP). At the heart of the federation are the locally deployed Cloud Management stacks. In compliance with the Cloud computing model, the EGI CLIP does not mandate deploying any particular or specific Cloud Management stack; it is the responsibility of the Resource Providers to investigate, identify and deploy the solution that fits best their individual needs for as long as the offered services implement the required interfaces and domain languages. These interfaces and domain languages, and the interoperability of their implementation with other solutions are the focus of the federation.
 
Consequently, the EGI CLIP is modelled around the concept of an abstract Cloud Management stack subsystem that is integrated with components of the EGI Core Infrastructure Platform (see Figure 1).
 
This architecture allows EGI to define the CLIP as a relatively thin layer of federation and interoperability around local deployments and integrations of Cloud Management stacks. This architecture defines interaction ports with a number of services from the EGI Core Infrastructure Platform, and the EGI Collaboration Platform. At the same time, it defines the required external interfaces and corresponding interaction ports. All these ports will have to be realized by local Cloud Management stack deployments. The main interaction points of Resource Providers must take care of:
 
*Integrate with the EGI Core Authentication &amp; Authorisation Infrastructure
*Integrate with the EGI Core Accounting system
*Integrate with the EGI Core Monitoring system
*Provide a standardised Cloud Computing management interface (OCCI)
*Provide a standardised Cloud Storage interface (CDMI)
*Provide a standardised interface to an Information Service
 
Additionally, by means of using the Appliance Repository and the VM Marketplace from the EGI Collaboration Platform the EGI Cloud Infrastructure Platform is providing VM image sharing and re-use across EGI Research Communities.
 
[[Image:Fedcloud-tf-Technology-CloudManagementStackComponent.png|thumb|center|800px|Fedcloud-tf-Technology-CloudManagementStackComponent.png]]
 
Figure 2 provides an overview of the current realizations of the abstract Cloud Management stack subsystem in the EGI Cloud federation. It illustrates that each existing realisation inherits the obligation to implement the interaction points from the generalized parent Cloud Management stack. At the same time, the EGI Federated Clouds Task (funded through the EGI-InSPIRE project) gives Resource Providers a platform to share their implementation solutions for a commonly deployed specific Cloud Management stack (e.g. OpenNebula and OpenStack). [[Fedcloud-tf:ResourceProviders#Deployment|This page]] is dedicated to the documentation of the steps necessary to integrate a local deployment of a given Cloud Management stack into the EGI Cloud federation.  
 
Through this collaboration, Resource Providers gradually develop and mature deployment and configuration profiles around common Cloud Management stacks as illustrated in Figure 2. Through mutual support Resource Providers begin to build communities around the deployed Cloud Management Frameworks – the result is better integration of the most popular Cloud Management Frameworks in the Federated Clouds Task as illustrated in table below.
 
<br>


{| class="wikitable"
{| class="wikitable"
Line 84: Line 50:
! Cloud Mgmt. Stack  
! Cloud Mgmt. Stack  
! Fed. AAI  
! Fed. AAI  
! Info. Discovery
! Monitoring  
! Monitoring  
! Accounting  
! Accounting  
! Img. Mgmt.  
! VM Img. Mgmt.  
! OCCI  
! OCCI  
! CDMI
! CDMI
Line 94: Line 61:
| Yes  
| Yes  
| Yes  
| Yes  
| Yes
| Yes  
| Yes  
| Yes  
| Yes  
| Yes
| In progress
|-
|-
! OpenNebula  
! OpenNebula  
Line 102: Line 70:
| Yes  
| Yes  
| Yes  
| Yes  
| Yes
| Yes  
| Yes  
| Yes  
| Yes  
| Yes
| N/A
|-
|-
! StratusLab
! Synnefo
| Yes  
| Yes  
| Yes  
| Yes  
| Yes
| Yes  
| Yes  
| -
| Yes
| Yes  
| Yes  
| -
| Yes
|}
|}


= Interfaces and standards  =


To federate a cloud system there are several functions for which a common interface must be defined. These are each described below and overall provide the definition of the method by which a ‘user’ of the service would be able to interact.  
== Community Federated Cloud ==
 
A community cloud is accessible to a selected group of users or Virtual Organizations. These clouds have a looser federation model hence the level of integration with the EGI services depends on the needs of the community it serves. Community clouds may also choose the interfaces to access the IaaS capabilities as long as the selected interfaces assure the interoperability within the federated resources. The standards used in the Public Federated Cloud are recommended if there are no requirements that prevent their usage.
 
Community clouds may profit the existing developments for the integration of Cloud Management stacks into the public federated cloud.


*Cloud interfaces
**[[Federated Cloud Architecture#VM_management_interface:_OCCI|VM management interface: OCCI]]
**[[Federated Cloud Architecture#Data_management_interface:_CDMI|Data management interface: CDMI]]
**[[Federated Cloud Architecture#VM_Image_management|VM Image management]]
*Integration with EGI Core services interfaces
**[[Federated Cloud Architecture#Virtual_Organisation_Management_.26_AAI:_VOMS|Virtual Organisation Management &amp; AAI: VOMS]]
**[[Federated Cloud Architecture#Information_discovery:_BDII|Information discovery: BDII]]
**[[Federated Cloud Architecture#Central_service_registry:_GOCDB|Central service registry: GOCDB]]
**[[Federated Cloud Architecture#Monitoring:_SAM|Monitoring: SAM]]
**[[Federated Cloud Architecture#Accounting|Accounting]]
**[[Federated Cloud Architecture#Image_metadata_publishing_.26_repository|Image metadata publishing &amp; repository]]


= Extending the EGI Federated Cloud  =
= Extending the EGI Federated Cloud  =
'''TO REMOVE'''


An efficient and easy to use federated cloud needs more than just the infrastructure services. Convenient, user friendly interfaces for users, operators and developers are also a must. The EGI community is open to new projects which aim to improve the use and access to the EGI Federated Cloud work groups. If you are interested in participating or have questions please [mailto:support@egi.eu email] the User Community Support Team. Below are some example projects:  
An efficient and easy to use federated cloud needs more than just the infrastructure services. Convenient, user friendly interfaces for users, operators and developers are also a must. The EGI community is open to new projects which aim to improve the use and access to the EGI Federated Cloud work groups. If you are interested in participating or have questions please [mailto:support@egi.eu email] the User Community Support Team. Below are some example projects:  

Revision as of 18:47, 28 May 2015

Overview For users For resource providers Infrastructure status Site-specific configuration Architecture




Baustelle.png This page is under construction.


Introduction

The current high throughput model of grid computing has proven to be extremely powerful for a small number of different communities. These communities have thrived in the current grid environment but the uptake of e-infrastructure by other communities has been limited. EGI has therefore strategically decided to investigate how it could broaden the uptake of its infrastructure to support other research communities and application design models, that would not only be able to take advantage of the existing functionality and investment already made in EGI’s Core Infrastructure, but also support different research communities and their applications on the current production infrastructure than it was previously able to.

The utilization of Virtualization and Infrastructure as a Service (IaaS) cloud computing was a clear candidate to enable this transformation. It was also clear that with a number of different open source technologies already in use across a number of different resource providers, that it would not be possible to mandate a single software stack. Instead, following on from a number of different activities already on-going in Europe including SIENA1, an approach that required the utilization of open standards where available and, where not, methods that have broad acceptance in the e-infrastructure community were essential.

This led to the current EGI Cloud Federation model, a model based on the needs of each community using the IaaS cloud as described below.

Federation Model

The EGI Federated Cloud is a set of clusters targeting global or specific user communities: a public federated cloud open for any researchers and various community clouds accessible to one or more selected Virtual Organizations. Every cluster follows the model shown in the figure:

Federated Cloud Model

The federation of IaaS Cloud resources in EGI is built upon the extensive autonomy of Resource Providers in terms of ownership of exposed resources. The federation model for distributed IaaS Cloud resources allows a lightweight aggregation of local Cloud resources into the EGI Cloud Infrastructure Platform (CLIP). At the heart of the federation are the locally deployed Cloud Management stacks. In compliance with the Cloud computing model, the EGI CLIP does not mandate deploying any particular or specific Cloud Management stack; it is the responsibility of the Resource Providers to investigate, identify and deploy the solution that fits best their individual needs for as long as the offered services implement the required interfaces and domain languages. These interfaces and domain languages, and the interoperability of their implementation with other solutions are the focus of the federation.

Every cluster of the federation, defines the actual level of integration with the EGI Federated Operations Services, the EGI marketplace and their Federated AAI architecture and technologies. The IaaS services of the cluster must be offered with interfaces that assure the interoperability within the community and whenever possible open standards should be used.

describe what's inside each box of the figure!


Public Federated Cloud

The following figure depicts the concrete model for the public federated cloud, open to any research community, which is completely integrated with all EGI Core Services and EGI Marketplace, uses open standards (if available) for all the public APIs and uses the current AAI schema of EGI based on X.509 proxies with VOMS extensions.

Public Federated Cloud Architecture

Resource providers in the public Federated Cloud must offer one or both of the following IaaS cloud capabilities by implementing these interfaces:

Moreover, to federate a resource provider in the public EGI Federated Cloud, it must integrate with the EGI core services and interfaces listed below:

The EGI Federated Clouds Task gives Resource Providers a platform to share their implementation solutions for a commonly deployed specific Cloud Management stack (e.g. OpenNebula and OpenStack). The Federated Cloud resource providers support page is dedicated to the documentation of the steps necessary to integrate a local deployment of a given Cloud Management stack into the EGI Cloud federation. The table below summarizes the current integration level of the participating Cloud Management stacks in the federation:

Cloud Mgmt. Stack Fed. AAI Info. Discovery Monitoring Accounting VM Img. Mgmt. OCCI CDMI
OpenStack Yes Yes Yes Yes Yes Yes In progress
OpenNebula Yes Yes Yes Yes Yes Yes N/A
Synnefo Yes Yes Yes Yes Yes Yes Yes


Community Federated Cloud

A community cloud is accessible to a selected group of users or Virtual Organizations. These clouds have a looser federation model hence the level of integration with the EGI services depends on the needs of the community it serves. Community clouds may also choose the interfaces to access the IaaS capabilities as long as the selected interfaces assure the interoperability within the federated resources. The standards used in the Public Federated Cloud are recommended if there are no requirements that prevent their usage.

Community clouds may profit the existing developments for the integration of Cloud Management stacks into the public federated cloud.


Extending the EGI Federated Cloud

TO REMOVE

An efficient and easy to use federated cloud needs more than just the infrastructure services. Convenient, user friendly interfaces for users, operators and developers are also a must. The EGI community is open to new projects which aim to improve the use and access to the EGI Federated Cloud work groups. If you are interested in participating or have questions please email the User Community Support Team. Below are some example projects:

Project name Description
OCCI Web interface The start-up and management of virtual machine images in the EGI Federated Cloud is possible through the ‘Open Cloud Computing Interface’ API. While the OCCI API enables flexible management of images, it is a low level tool an its use requires an experienced programmer/user. This project aims to create and provide a high level, graphical front end interface for OCCI that would empower non-IT communities managing virtual machines on the EGI federated cloud. The interface can be a web browser plug-in, for example an extension of Hybridfox) or can be a portal that is dedicated for serving users of the EGI Federated Cloud.
Grid middleware images Those who wish to operate a distributed computing environment within the EGI cloud would benefit from having grid middleware components available in the EGI Virtual Machine Marketplace. While some of the gLite middleware components have images, many components are missing, and other middleware services (e.g. UNICORE, ARC) are not covered at all. This project aims to create, publish and keep up to date virtual machine images about the complete EGI Unified Middleware Distribution stack in the EGI Virtual Machine Marketplace.
BDII web interface BDII web interface Static information about the sites of the EGI Federated Cloud is stored in a ‘BDII Information System’. BDII is an LDAP based database optimised for frequent queries. The standard query interface of BDII is a command line tool. This project aims to provide a graphical interface that could be used to browse up-to-date information about the EGI Federated Cloud from its BDII information system. The interface could be a web portal, a web gadget or a web browser plug-in.
Retrieved from "https://wiki.egi.eu/w/index.php?title=Federated_Cloud_Technology&oldid=79769"