The wiki is deprecated and due to be decommissioned by the end of September 2022.
The content is being migrated to other supports, new updates will be ignored and lost.
If needed you can get in touch with EGI SDIS team using operations @ egi.eu.
Difference between revisions of "Federated Cloud Architecture"
Jump to navigation
Jump to search
| Line 8: | Line 8: | ||
= EGI Cloud Federation = | = EGI Cloud Federation = | ||
The EGI Federated Cloud is a multi-national cloud system that integrates | The EGI Federated Cloud is a multi-national cloud system that integrates community, private and/or public clouds into a scalable computing platform for data and/or compute driven applications and services. The architecture of the federation is modelled around the concept of an abstract Cloud Management Framework (CMF) subsystem that provides a set of agreed interfaces within the community it provides services to. Each resource center of the infrastructure operates an instance of a Cloud Management Framework according to its own preferences and constraints and federates by integrating with selected components of the EGI Core Infrastructure. | ||
Providers are organised into ''realms'', with each realm having homogeneous cloud management interfaces. A cloud realm is a subset of cloud providers exposing homogeneous cloud management interfaces and capabilities. A Community Platform provides community-specific data, tools and applications, which can be supported by one or more realms. EGI currently operates two realms: the '''Open Standards Realm''' and the '''OpenStack Realm'''. Both are completely integrated with the EGI federator services described below but use different interfaces to offer the IaaS capabilities to the users: the Open Standards Realm uses OCCI standard (supported by providers with OpenNebula, OpenStack or Synnefo cloud management frameworks), while the OpenStack Realm uses OpenStack native Nova and Swift APIs (support limited to OpenStack providers). This OpenStack Realm was introduced in the federation during November 2015 and can co-exist with the Open Standards Realm within the same resource provider. | |||
The EGI Federated Cloud | [[Image:Federated_Cloud_Model.png|thumb|center|600px|Federated Cloud Model]] | ||
New realms can be defined as needed by the user communities by agreeing with the providers which interfaces to expose and which of the EGI core services to use for the federation. The EGI Federated Cloud integrates and maintains a flexible solution portfolio that enables various types of cloud federations with IaaS, PaaS and SaaS capabilities. The collaboration is committed to the use of open source tools and services that are reusable across scientific disciplines. A scientific community can mix and match items from this portfolio to establish its own, customised cloud federation. EGI currently provides services in these areas: | |||
* Federated service registry for configuration management of the federated cloud services. | |||
* SSO for authentication and authorization across the whole cloud federation. | |||
* Federated accounting system to collect, aggregate and display usage information. | |||
* Federated information discovery, allowing users and tools to have information about capabilities and services available in the federation. | |||
* VM image catalogue and management: open library of VM images for use on a cloud or for personal download which can be automatically and securely distributed to any resource provider. | |||
* Federated monitoring to perform federated service availability monitoring and reporting of the distributed cloud service end-points, and to retrieve this information programmatically. | |||
* A federated service management system of processes, policies, activities and supporting tools customized to the federated cloud. | |||
== Federation Enabling Services == | |||
The Federation is composed of independent providers that are organised into realms by exposing a set of agreed interfaces to the users and by integrating a set of federation enabling services according to the preferences, choices and constraints set by the realm members and users. Despite the large diversity in the type of cloud realms, a relatively small number of identical building blocks (or federator services) can be identified in almost all of them. These services turn individual clouds into a federation. The table collects these common services to help architects identify topics they should focus on when designing a cloud federation and links to the [[Federated Cloud Technology]] page with detailed information. | |||
Despite the large diversity in the type of cloud realms, a relatively small number of identical building blocks (or federator services) can be identified in almost all of them. These services turn individual clouds into a federation. The table collects these common services to help architects identify topics they should focus on when designing a cloud federation | |||
{| class="wikitable" style="margin: auto;" | {| class="wikitable" style="margin: auto;" | ||
| Line 28: | Line 37: | ||
! Service Registry | ! Service Registry | ||
| A registry where all the federated sites and services are registered and state their capabilities. The registry provides the ‘big picture view’ about the federation for both human users and online services (such as service monitors). | | A registry where all the federated sites and services are registered and state their capabilities. The registry provides the ‘big picture view’ about the federation for both human users and online services (such as service monitors). | ||
|GOCDB | | [[Federated_Cloud_Technology#GOCDB | GOCDB]] | ||
|- | |||
! Single sign-on for users | |||
| Ensuring that users of the federation need to register for access only once before they can use the federated services. Single sign-on is increasingly implemented in the form of identity federations in both industry and academia. | |||
| [[Federated_Cloud_Technology#Virtual_Organisation_Management_.26_AAI | X.509 proxies with VOMS extensions]] | |||
|- | |||
! Integrated view about resource/service usage | |||
|A system that pulls together usage (accounting) information from the federated sites and services, integrates the data and presents them in such a way that both individual users and communities can monitor their own resource/service usage across the whole federation. | |||
| [[Federated_Cloud_Technology#Accounting |Cloud Usage Record, Accounting repository and portal]] | |||
|- | |- | ||
!Information System | !Information System | ||
|A database that provides real-time view about the actual capabilities and load of federation participants. Can be used by both human users and online services. | |A database that provides real-time view about the actual capabilities and load of federation participants. Can be used by both human users and online services. | ||
|BDII | |[[Federated_Cloud_Technology#Information_Discovery|BDII]] | ||
|- | |- | ||
! Virtual Machine Image Catalogue | ! Virtual Machine Image Catalogue | ||
| A catalogue of Virtual Machine Images (VMIs) that encapsulate those software configurations that is useful and relevant for the given community (typically pre-configured scientific models and algorithms). | | A catalogue of Virtual Machine Images (VMIs) that encapsulate those software configurations that is useful and relevant for the given community (typically pre-configured scientific models and algorithms). | ||
|AppDB | | [[Federated_Cloud_Technology#AppDB_Cloud_MarketPlace|AppDB]] | ||
|- | |- | ||
! Image replication mechanism | ! Image replication mechanism | ||
| A system that automatically replicates VMIs from the federation VMI catalogue to each of the member sites, as well as removes them when needed. Automated replication can ensure consistency of capabilities across sites and is very often coupled with a VMI vetting process to ensure that only properly working, and relevant VMIs are replicated to the cloud sites of the community. | | A system that automatically replicates VMIs from the federation VMI catalogue to each of the member sites, as well as removes them when needed. Automated replication can ensure consistency of capabilities across sites and is very often coupled with a VMI vetting process to ensure that only properly working, and relevant VMIs are replicated to the cloud sites of the community. | ||
| | | [[Federated_Cloud_Technology#HEPiX_image_lists|HEPiX image lists]] | ||
|- | |- | ||
! | !Availability Monitoring | ||
| | | Use a shared system to monitor and collect availability and reliability statistics about the distributed cloud service providers and to retrieve this information programmatically. | ||
| | | [[Federated_Cloud_Technology#EGI_A.2FR_Monitoring|ARGO]] | ||
| | |||
|- | |- | ||
! Integrated interfaces or user environments | ! Integrated interfaces or user environments | ||
|Having interfaces through which users and user applications can interact with the services offered by the various cloud providers. In case of an IaaS cloud federation these interfaces offer compute, storage and network management capabilities. | |Having interfaces through which users and user applications can interact with the services offered by the various cloud providers. In case of an IaaS cloud federation these interfaces offer compute, storage and network management capabilities. | ||
|OCCI | | [[Federated_Cloud_Technology#OCCI|OCCI]] and [[Federated_Cloud_Technology#OpenStack_Compute|OpenStack API]] | ||
|- | |- | ||
! Federated service management tools | ! Federated service management tools | ||
| Line 63: | Line 72: | ||
|} | |} | ||
[[Category:Federated_Cloud]] | [[Category:Federated_Cloud]] | ||
Revision as of 12:50, 1 February 2017
| Overview | For users | For resource providers | Infrastructure status | Site-specific configuration | Architecture |
| Architecture | Technology | Roadmap | FedCloud Task Force |
EGI Cloud Federation
The EGI Federated Cloud is a multi-national cloud system that integrates community, private and/or public clouds into a scalable computing platform for data and/or compute driven applications and services. The architecture of the federation is modelled around the concept of an abstract Cloud Management Framework (CMF) subsystem that provides a set of agreed interfaces within the community it provides services to. Each resource center of the infrastructure operates an instance of a Cloud Management Framework according to its own preferences and constraints and federates by integrating with selected components of the EGI Core Infrastructure.
Providers are organised into realms, with each realm having homogeneous cloud management interfaces. A cloud realm is a subset of cloud providers exposing homogeneous cloud management interfaces and capabilities. A Community Platform provides community-specific data, tools and applications, which can be supported by one or more realms. EGI currently operates two realms: the Open Standards Realm and the OpenStack Realm. Both are completely integrated with the EGI federator services described below but use different interfaces to offer the IaaS capabilities to the users: the Open Standards Realm uses OCCI standard (supported by providers with OpenNebula, OpenStack or Synnefo cloud management frameworks), while the OpenStack Realm uses OpenStack native Nova and Swift APIs (support limited to OpenStack providers). This OpenStack Realm was introduced in the federation during November 2015 and can co-exist with the Open Standards Realm within the same resource provider.
New realms can be defined as needed by the user communities by agreeing with the providers which interfaces to expose and which of the EGI core services to use for the federation. The EGI Federated Cloud integrates and maintains a flexible solution portfolio that enables various types of cloud federations with IaaS, PaaS and SaaS capabilities. The collaboration is committed to the use of open source tools and services that are reusable across scientific disciplines. A scientific community can mix and match items from this portfolio to establish its own, customised cloud federation. EGI currently provides services in these areas:
- Federated service registry for configuration management of the federated cloud services.
- SSO for authentication and authorization across the whole cloud federation.
- Federated accounting system to collect, aggregate and display usage information.
- Federated information discovery, allowing users and tools to have information about capabilities and services available in the federation.
- VM image catalogue and management: open library of VM images for use on a cloud or for personal download which can be automatically and securely distributed to any resource provider.
- Federated monitoring to perform federated service availability monitoring and reporting of the distributed cloud service end-points, and to retrieve this information programmatically.
- A federated service management system of processes, policies, activities and supporting tools customized to the federated cloud.
Federation Enabling Services
The Federation is composed of independent providers that are organised into realms by exposing a set of agreed interfaces to the users and by integrating a set of federation enabling services according to the preferences, choices and constraints set by the realm members and users. Despite the large diversity in the type of cloud realms, a relatively small number of identical building blocks (or federator services) can be identified in almost all of them. These services turn individual clouds into a federation. The table collects these common services to help architects identify topics they should focus on when designing a cloud federation and links to the Federated Cloud Technology page with detailed information.
| Federation Service | Role within the federation | Existing technical solution in EGI |
|---|---|---|
| Service Registry | A registry where all the federated sites and services are registered and state their capabilities. The registry provides the ‘big picture view’ about the federation for both human users and online services (such as service monitors). | GOCDB |
| Single sign-on for users | Ensuring that users of the federation need to register for access only once before they can use the federated services. Single sign-on is increasingly implemented in the form of identity federations in both industry and academia. | X.509 proxies with VOMS extensions |
| Integrated view about resource/service usage | A system that pulls together usage (accounting) information from the federated sites and services, integrates the data and presents them in such a way that both individual users and communities can monitor their own resource/service usage across the whole federation. | Cloud Usage Record, Accounting repository and portal |
| Information System | A database that provides real-time view about the actual capabilities and load of federation participants. Can be used by both human users and online services. | BDII |
| Virtual Machine Image Catalogue | A catalogue of Virtual Machine Images (VMIs) that encapsulate those software configurations that is useful and relevant for the given community (typically pre-configured scientific models and algorithms). | AppDB |
| Image replication mechanism | A system that automatically replicates VMIs from the federation VMI catalogue to each of the member sites, as well as removes them when needed. Automated replication can ensure consistency of capabilities across sites and is very often coupled with a VMI vetting process to ensure that only properly working, and relevant VMIs are replicated to the cloud sites of the community. | HEPiX image lists |
| Availability Monitoring | Use a shared system to monitor and collect availability and reliability statistics about the distributed cloud service providers and to retrieve this information programmatically. | ARGO |
| Integrated interfaces or user environments | Having interfaces through which users and user applications can interact with the services offered by the various cloud providers. In case of an IaaS cloud federation these interfaces offer compute, storage and network management capabilities. | OCCI and OpenStack API |
| Federated service management tools | A set of processes, policies, activities and supporting tools customized to the federated cloud. | EGI federated service management |