Federated Cloud Architecture

Overview

For users

For resource providers

Infrastructure status

Site-specific configuration

Architecture

Architecture

Technology

Roadmap

FedCloud Task Force

Federation Model

The EGI Federated Cloud is a multi-national cloud system that integrates community, private and/or public clouds into a scalable computing platform for research. The Federation pools IaaS, PaaS and SaaS services from a heterogeneous set of cloud providers using a single authentication and authorization framework that allows the portability of workloads across multiple providers and enable bringing computing to data.

Each resource center of the federated infrastructure operates a Cloud Management Framework (CMF) according to its own preferences and constraints and joins the federation by integrating this CMF with components of the EGI Federation and Collaboration Services. All services provided by the CMFs must at least be integrated with EGI AAI so users can access services with a single identity, integration with other components and APIs to be provided are agreed by the community the resource center provides services to.

Federated Cloud Model

Providers are organised into realms, with each realm having homogeneous cloud interfaces and capabilities. Community Platform provide community-specific data, tools and applications, which can be supported by one or more realms. New realms can be defined as needed by the user communities by agreeing with the providers which interfaces to expose and which of the EGI core services to use for the federation. EGI integrates and maintains a flexible solution portfolio that enables various types of cloud federations with IaaS capabilities and seeking to expand to PaaS and SaaS capabilities.

EGI follows a Service Integration and Management (SIAM) approach to manage the federation with processes that cover the different aspects of the IT Service Management. Providers in the federation keep complete control of their services and resources. EGI VO OLAs establish a reliable, trust-based communication channel between the Customer and the providers to agree on the services, their levels and the types of support. The EGI VO OLAs are not legal contracts but, as agreements, they outline the clear intentions to collaborate and support research.

Federated IaaS

The EGI Federated Cloud Infrastructure as a Service (IaaS) resource centers deploy a Cloud Management Framework (CMF) that provide one or more of the following end-user capabilities:

• Management of Virtual Machines and of persistent Block Storage devices that can be associated to the Virtual Machines within a single resource center.
• Object storage to manage data as objects with a variable amount of metadata and a globally unique identifier.

These end-user capabilities must be provided via community agreed APIs that can be integrated with the following EGI services:

• AAI to provide Single Sign-On for authentication and authorization across the whole cloud federation.
• Configuration Database, to record information about the topology of the e-infrastructure.
• Accounting to collect, aggregate and display usage information.
• Monitoring to perform federated service availability monitoring and reporting of the distributed cloud service endpoints, and to retrieve this information programmatically. Integration with monitoring is a passive activity of the resource center, the monitoring is performed using the end-user APIs with regular user credentials from EGI AAI.

Additionally, realms of the EGI IaaS Cloud can integrate with:

• Information Discovery, allowing users and tools to have information about capabilities and services available in the federation.
• AppDB Community-curated catalogue of Virtual Appliances (Virtual Machine Images) and distribution of appliances to the providers of the infrastructure.

EGI does not mandate deploying any particular or specific Cloud Management Framework; it is the responsibility of the Resource Center to investigate, identify and deploy the solution that fits best their individual needs whilst ensuring that the offered services implement the required interfaces and domain languages of the federation realms they are member of.

Federated Cloud Model

Users and Community platforms built on top of the EGI Federated Cloud IaaS have several ways of interacting with the cloud providers:

• Directly using the IaaS APIs to manage individual resources. This option is recommended for pre-existing use cases with requirements on specific APIs.
• Leveraging IaaS Federated Access Tools that allow managing the complexity of dealing with different providers in a uniform way. These tools include
  • IaaS provisioning systems that allow to define infrastructure as code and manage and combine resources from different providers, thus enabling the portability of application deployments between them (e.g. IM or Terraform);
  • Cloud brokers, that provide matchmaking for workloads to available providers (e.g. the INDIGO-DataCloud Orchestrator); and
  • Cloud Management Software that provides a unified console for accessing resources and deploy workloads following a set of user-defined established policies (e.g. Scalr or RightScale)
• Using the AppDB VMOps dashboard, a web-based GUI that simplifies the management of VMs on any provider of the EGI infrastructure. AppDB VMOps in turn relies on the Infrastructure Manager, a federated IaaS provisioning tool documented in the aforementioned wiki.

EGI currently operates two IaaS realms: the Open Standards Realm and the OpenStack Realm. Both are fully integrated with all EGI core activities (AAI, Configuration Database, Accounting, Monitoring, Information Discovery and Virtual Appliance Catalogue) but use different interfaces to offer the IaaS capabilities to the users: the Open Standards Realm uses OCCI standard, while the OpenStack Realm uses OpenStack native APIs.

Currently, EGI supports the integration of OpenStack, OpenNebula and Synnefo Cloud Management Frameworks via a set of technology components that interact whenever possible using the public interfaces of these CMFs, thus minimising the impact on operations of the site. A detailed listing of tools and how they interact with the underlying infrastructure is available at the EGI Federated Cloud Technology.