Alert.png The wiki is deprecated and due to be decommissioned by the end of September 2022.
The content is being migrated to other supports, new updates will be ignored and lost.
If needed you can get in touch with EGI SDIS team using operations @ egi.eu.

Difference between revisions of "Federated Cloud Architecture"

From EGIWiki
Jump to navigation Jump to search
Line 6: Line 6:
{{TOC_right}}
{{TOC_right}}


= EGI Cloud Federation =
= Federation Model =  


The EGI Federated Cloud is a multi-national cloud system that integrates community, private and/or public clouds into a scalable computing platform for data and/or compute driven applications and services. The architecture of the federation is modelled around the concept of an abstract Cloud Management Framework (CMF) subsystem that provides a set of agreed interfaces within the community it provides services to. Each resource center of the infrastructure operates an instance of a Cloud Management Framework according to its own preferences and constraints and federates by integrating with selected components of the EGI Core Infrastructure. The integration with EGI Core infrastructure is performed whenever possible using public interfaces of the supported CMFs thus minimising the impact on operations of the site.  
The EGI Federated Cloud is a multi-national cloud system that integrates community, private and/or public clouds into a scalable computing platform for research. The Federation pools IaaS, PaaS and SaaS services from a heterogeneous set of cloud providers using a single authentication and authorization framework that allows the portability of workloads across multiple providers and enable bringing computing to data.


Providers are organised into ''realms'', with each realm having homogeneous cloud management interfaces. A cloud realm is a subset of cloud providers exposing homogeneous cloud management interfaces and capabilities. A Community Platform provides community-specific data, tools and applications, which can be supported by one or more realms. EGI currently operates two realms: the '''Open Standards Realm''' and the '''OpenStack Realm'''. Both are completely integrated with the EGI federator services described below but use different interfaces to offer the IaaS capabilities to the users: the Open Standards Realm uses OCCI standard (supported by providers with OpenNebula, OpenStack or Synnefo cloud management frameworks), while the OpenStack Realm uses OpenStack native Nova and Swift APIs (support limited to OpenStack providers).  This OpenStack Realm was introduced in the federation during November 2015 and can co-exist with the Open Standards Realm within the same resource provider.
Each resource center of the federated infrastructure operates a Cloud Management Framework (CMF) according to its own preferences and constraints and joins the federation by integrating this CMF with components of the EGI Federation and Collaboration Services. All services provided by the CMFs must at least be integrated with EGI AAI so users can access services with a single identity, integration with other components and APIs to be provided are agreed by the community the resource center provides services to.


[[Image:Federated_Cloud_Model.png|thumb|center|600px|Federated Cloud Model]]
[[Image:Federated_Cloud_Model.png|thumb|center|600px|Federated Cloud Model]]


New realms can be defined as needed by the user communities by agreeing with the providers which interfaces to expose and which of the EGI core services to use for the federation. The EGI Federated Cloud integrates and maintains a flexible solution portfolio that enables various types of cloud federations with IaaS, PaaS and SaaS capabilities. The collaboration is committed to the use of open source tools and services that are reusable across scientific disciplines. A scientific community can mix and match items from this portfolio to establish its own, customised cloud federation. EGI currently provides services in these areas:
Providers are organised into realms, with each realm having homogeneous cloud interfaces and capabilities. Community Platform provide community-specific data, tools and applications, which can be supported by one or more realms. New realms can be defined as needed by the user communities by agreeing with the providers which interfaces to expose and which of the EGI core services to use for the federation. EGI integrates and maintains a flexible solution portfolio that enables various types of cloud federations with IaaS capabilities and seeking to expand to PaaS and SaaS capabilities.


* Federated service registry for configuration management of the federated cloud services.
EGI follows a Service Integration and Management (SIAM) approach to manage the federation with processes that cover the different aspects of the IT Service Management. Providers in the federation keep complete control of their services and resources. EGI VO OLAs establish a reliable, trust-based communication channel between the Customer and the providers to agree on the services, their levels and the types of support. The EGI VO OLAs are not legal contracts but, as agreements, they outline the clear intentions to collaborate and support research.
* SSO for authentication and authorization across the whole cloud federation.
* Federated accounting system to collect, aggregate and display usage information.
* Federated information discovery, allowing users and tools to have information about capabilities and services available in the federation.
* VM image catalogue and management: open library of VM images for use on a cloud or for personal download which can be automatically and securely distributed to any resource provider.
* Federated monitoring to perform federated service availability monitoring and reporting of the distributed cloud service end-points, and to retrieve this information programmatically.
* A federated service management system of processes, policies, activities and supporting tools customized to the federated cloud.


Currently the EGI Federated Cloud is looking for services that provide an orchestration and brokering layer on top of the individual site APIs to enhance the portability of applications between resources and ease the usage of the infrastructure.
== Federated IaaS ==
The EGI Federated Cloud Infrastructure as a Service (IaaS) resource centers deploy a Cloud Management Framework (CMF) that provide one or more of the following end-user capabilities:
* Management of Virtual Machines and of persistent Block Storage devices that can be associated to the Virtual Machines within a single resource center.
* Object storage to manage data as objects with a variable amount of metadata and a globally unique identifier.


= Federation Enabling Services =
These end-user capabilities must be provided via community agreed APIs that can be integrated with the following EGI services:
* AAI to provide Single Sign-On for authentication and authorization across the whole cloud federation.
* Configuration Database, to record information about the topology of the e-infrastructure.
* Accounting to collect, aggregate and display usage information.
* Monitoring to perform federated service availability monitoring and reporting of the distributed cloud service endpoints, and to retrieve this information programmatically. Integration with monitoring is a passive activity of the resource center, the monitoring is performed using the end-user APIs with regular user credentials from EGI AAI.


The Federation is composed of independent providers that are organised into realms by exposing a set of agreed interfaces to the users and by integrating a set of federation enabling services according to the preferences, choices and constraints set by the realm members and users. Despite the large diversity in the type of cloud realms, a relatively small number of identical building blocks (or federator services) can be identified in almost all of them. These services turn individual clouds into a federation. The table collects these common services to help architects identify topics they should focus on when designing a cloud federation and links to the [[Federated Cloud Technology]] page with detailed information.
Additionally, realms of the EGI IaaS Cloud can integrate with:
* Information Discovery, allowing users and tools to have information about capabilities and services available in the federation.
* AppDB Community-curated catalogue of Virtual Appliances (Virtual Machine Images) and distribution of appliances to the providers of the infrastructure.


{| class="wikitable" style="margin: auto;"
EGI does not mandate deploying any particular or specific Cloud Management Framework; it is the responsibility of the Resource Center to investigate, identify and deploy the solution that fits best their individual needs whilst ensuring that the offered services implement the required interfaces and domain languages of the federation realms they are member of.  
|-
! Federation Service
! Role within the federation
! Existing technical solution in EGI
|-
! Service Registry
| A registry where all the federated sites and services are registered and state their capabilities. The registry provides the ‘big picture view’ about the federation for both human users and online services (such as service monitors).
| [[Federated_Cloud_Technology#GOCDB | GOCDB]]
|-
! Single sign-on for users
| Ensuring that users of the federation need to register for access only once before they can use the federated services. Single sign-on is increasingly implemented in the form of identity federations in both industry and academia.
|  [[Federated_Cloud_Technology#Virtual_Organisation_Management_.26_AAI | X.509 proxies with VOMS extensions]]
|-
! Integrated view about resource/service usage
|A system that pulls together usage (accounting) information from the federated sites and services, integrates the data and presents them in such a way that both individual users and communities can monitor their own resource/service usage across the whole federation.
| [[Federated_Cloud_Technology#Accounting |Cloud Usage Record, Accounting repository and portal]]
|-
!Information System
|A database that provides real-time view about the actual capabilities and load of federation participants. Can be used by both human users and online services.
|[[Federated_Cloud_Technology#Information_Discovery|BDII]]
|-
! Virtual Machine Image Catalogue
| A catalogue of Virtual Machine Images (VMIs) that encapsulate those software configurations that is useful and relevant for the given community (typically pre-configured scientific models and algorithms).
| [[Federated_Cloud_Technology#AppDB_Cloud_MarketPlace|AppDB]]
|-
! Image replication mechanism
| A system that automatically replicates VMIs from the federation VMI catalogue to each of the member sites, as well as removes them when needed. Automated replication can ensure consistency of capabilities across sites and is very often coupled with a VMI vetting process to ensure that only properly working, and relevant VMIs are replicated to the cloud sites of the community.
| [[Federated_Cloud_Technology#HEPiX_image_lists|HEPiX image lists]]
|-
!Availability Monitoring
| Use a shared system to monitor and collect availability and reliability statistics about the distributed cloud service providers and to retrieve this information programmatically.
| [[Federated_Cloud_Technology#EGI_A.2FR_Monitoring|ARGO]]
|-
! Integrated interfaces or user environments
|Having interfaces through which users and user applications can interact with the services offered by the various cloud providers. In case of an IaaS cloud federation these interfaces offer compute, storage and network management capabilities.
| [[Federated_Cloud_Technology#OCCI|OCCI]] and [[Federated_Cloud_Technology#OpenStack_Compute|OpenStack API]]
|-
! Federated service management tools
| A set of processes, policies, activities and supporting tools customized to the federated cloud.
| EGI federated service management
|}


[[Image:Federated_Cloud_IaaS_Model.png|thumb|center|600px|Federated Cloud Model]]


Users and Community platforms built on top of the EGI Federated Cloud IaaS have several ways of interacting with the cloud providers:
* Directly using the IaaS APIs to manage individual resources. This option is recommended for pre-existing use cases with requirements on specific APIs.
* Leveraging IaaS Federated Access Tools that allow managing the complexity of dealing with different providers in a uniform way. These tools include
**IaaS provisioning systems that allow to define infrastructure as code and manage and combine resources from different providers, thus enabling the portability of application deployments between them (e.g. IM or Terraform);
**Cloud brokers, that provide matchmaking for workloads to available providers (e.g. the INDIGO-DataCloud Orchestrator); and
**Cloud Management Software that provides a unified console for accessing resources and deploy workloads following a set of user-defined established policies (e.g. Scalr or RightScale)
* Using the AppDB VMOps dashboard, a web-based GUI that simplifies the management of VMs on any provider of the EGI infrastructure. AppDB VMOps in turn relies on the Infrastructure Manager, a federated IaaS provisioning tool documented in the aforementioned wiki.
EGI currently operates two IaaS realms: the Open Standards Realm and the OpenStack Realm. Both are fully integrated with all EGI core activities (AAI, Configuration Database, Accounting, Monitoring, Information Discovery and Virtual Appliance Catalogue) but use different interfaces to offer the IaaS capabilities to the users: the Open Standards Realm uses OCCI standard, while the OpenStack Realm uses OpenStack native APIs.
Currently, EGI supports the integration of OpenStack, OpenNebula and Synnefo Cloud Management Frameworks via a set of technology components that interact whenever possible using the public interfaces of these CMFs, thus minimising the impact on operations of the site. A detailed listing of tools and how they interact with the underlying infrastructure is available at the [[EGI Federated Cloud Technology]].


[[Category:Federated_Cloud]]
[[Category:Federated_Cloud]]

Revision as of 16:21, 9 May 2017

Overview For users For resource providers Infrastructure status Site-specific configuration Architecture



Architecture Technology Roadmap FedCloud Task Force





Federation Model

The EGI Federated Cloud is a multi-national cloud system that integrates community, private and/or public clouds into a scalable computing platform for research. The Federation pools IaaS, PaaS and SaaS services from a heterogeneous set of cloud providers using a single authentication and authorization framework that allows the portability of workloads across multiple providers and enable bringing computing to data.

Each resource center of the federated infrastructure operates a Cloud Management Framework (CMF) according to its own preferences and constraints and joins the federation by integrating this CMF with components of the EGI Federation and Collaboration Services. All services provided by the CMFs must at least be integrated with EGI AAI so users can access services with a single identity, integration with other components and APIs to be provided are agreed by the community the resource center provides services to.

Federated Cloud Model

Providers are organised into realms, with each realm having homogeneous cloud interfaces and capabilities. Community Platform provide community-specific data, tools and applications, which can be supported by one or more realms. New realms can be defined as needed by the user communities by agreeing with the providers which interfaces to expose and which of the EGI core services to use for the federation. EGI integrates and maintains a flexible solution portfolio that enables various types of cloud federations with IaaS capabilities and seeking to expand to PaaS and SaaS capabilities.

EGI follows a Service Integration and Management (SIAM) approach to manage the federation with processes that cover the different aspects of the IT Service Management. Providers in the federation keep complete control of their services and resources. EGI VO OLAs establish a reliable, trust-based communication channel between the Customer and the providers to agree on the services, their levels and the types of support. The EGI VO OLAs are not legal contracts but, as agreements, they outline the clear intentions to collaborate and support research.

Federated IaaS

The EGI Federated Cloud Infrastructure as a Service (IaaS) resource centers deploy a Cloud Management Framework (CMF) that provide one or more of the following end-user capabilities:

  • Management of Virtual Machines and of persistent Block Storage devices that can be associated to the Virtual Machines within a single resource center.
  • Object storage to manage data as objects with a variable amount of metadata and a globally unique identifier.

These end-user capabilities must be provided via community agreed APIs that can be integrated with the following EGI services:

  • AAI to provide Single Sign-On for authentication and authorization across the whole cloud federation.
  • Configuration Database, to record information about the topology of the e-infrastructure.
  • Accounting to collect, aggregate and display usage information.
  • Monitoring to perform federated service availability monitoring and reporting of the distributed cloud service endpoints, and to retrieve this information programmatically. Integration with monitoring is a passive activity of the resource center, the monitoring is performed using the end-user APIs with regular user credentials from EGI AAI.

Additionally, realms of the EGI IaaS Cloud can integrate with:

  • Information Discovery, allowing users and tools to have information about capabilities and services available in the federation.
  • AppDB Community-curated catalogue of Virtual Appliances (Virtual Machine Images) and distribution of appliances to the providers of the infrastructure.

EGI does not mandate deploying any particular or specific Cloud Management Framework; it is the responsibility of the Resource Center to investigate, identify and deploy the solution that fits best their individual needs whilst ensuring that the offered services implement the required interfaces and domain languages of the federation realms they are member of.

Federated Cloud Model


Users and Community platforms built on top of the EGI Federated Cloud IaaS have several ways of interacting with the cloud providers:

  • Directly using the IaaS APIs to manage individual resources. This option is recommended for pre-existing use cases with requirements on specific APIs.
  • Leveraging IaaS Federated Access Tools that allow managing the complexity of dealing with different providers in a uniform way. These tools include
    • IaaS provisioning systems that allow to define infrastructure as code and manage and combine resources from different providers, thus enabling the portability of application deployments between them (e.g. IM or Terraform);
    • Cloud brokers, that provide matchmaking for workloads to available providers (e.g. the INDIGO-DataCloud Orchestrator); and
    • Cloud Management Software that provides a unified console for accessing resources and deploy workloads following a set of user-defined established policies (e.g. Scalr or RightScale)
  • Using the AppDB VMOps dashboard, a web-based GUI that simplifies the management of VMs on any provider of the EGI infrastructure. AppDB VMOps in turn relies on the Infrastructure Manager, a federated IaaS provisioning tool documented in the aforementioned wiki.

EGI currently operates two IaaS realms: the Open Standards Realm and the OpenStack Realm. Both are fully integrated with all EGI core activities (AAI, Configuration Database, Accounting, Monitoring, Information Discovery and Virtual Appliance Catalogue) but use different interfaces to offer the IaaS capabilities to the users: the Open Standards Realm uses OCCI standard, while the OpenStack Realm uses OpenStack native APIs.


Currently, EGI supports the integration of OpenStack, OpenNebula and Synnefo Cloud Management Frameworks via a set of technology components that interact whenever possible using the public interfaces of these CMFs, thus minimising the impact on operations of the site. A detailed listing of tools and how they interact with the underlying infrastructure is available at the EGI Federated Cloud Technology.