Federated Cloud Technology

Overview

For users

For resource providers

Infrastructure status

Site-specific configuration

Architecture

This page is under construction.

Cloud Federations

EGI Federated Cloud provides the services and technologies to create federation of clouds (community, private or public clouds) that operate according to the preferences, choices and contraints set by its members and users. The EGI Cloud Federations are modeled around the concept of an abstract Cloud Management stack subsystem that is integrated with components of the EGI Core Infrastructure and that provides a set of agreed uniform interfaces within the community it provides services to.

Federated Cloud Model

Despite the large diversity in the type of cloud federations, a relatively small number of building blocks (or federator services) can be identified is almost all of them. These services turn individual clouds into a federation. The sections below describe these common services and the EGI solutions provided for them.

Integrated interfaces or user environments

Cloud systems must provide a set of interfaces through which users and user applications can interact with the services offered. In case of an IaaS cloud federation these interfaces offer compute, storage and network management capabilities. The interfaces can be harmonised across all participating cloud providers - in which case the providers are responsible for implementing the agreed standard - or can be native at the different sites. In this latter case, libraries or portals can hide heterogeneity from the users and can translate user requests to diverse native formats.

EGI Federated promotes the use of open standards for providing a uniform interface and behaviour across different providers: OCCI for management of compute resources and CDMI for management of storage. The Federated Cloud APIs and SDKs page describes from a user point of view how these interfaces can be used.

OCCI

The Open Cloud Computing Interface (OCCI) is a RESTful Protocol and API designed to facilitate interoperable access to, and query of, cloud-based resources across multiple resource providers and heterogeneous environments. The formal specification is maintained and actively worked on by OGF’s OCCI-WG.

EGI Federated Cloud uses v1.1 of OCCI's specification, which is defined in three documents:

• OCCI Core that describes the formal definition of the OCCI core model,
• OCCI HTTP Rendering defines how to interact with the OCCI Core Model using the RESTful OCCI API
• OCCI Infrastructure contains the definition of the OCCI Infrastructure extension for the IaaS domain

The OCCI Infrastructure is further extended with two new mixins for contextualization in EGI Federated Cloud:

term	scheme	attributes
user_data	http://schemas.openstack.org/compute/instance#	org.openstack.compute.user_data: string that holds base64 encoded data to be available at the VM upon instantiation
public_key	http://schemas.openstack.org/instance/credentials#	org.openstack.credentials.publickey.name: string with the name of the public key (optional) org.openstack.credentials.publickey.data: string with the public key

The VM Management scenario page contains further details on the support for OCCI on different Cloud Management Stacks.

Implementations

rOCCI - A Ruby OCCI Framework

Provides OCCI support for various Cloud Management Frameworks, including OpenNebula

OCCI-OS

OpenStack OCCI interface

ooi

New implemenation of OCCI for OpenStack, upcoming release

CDMI

The SNIA Cloud Data Management Interface (CDMI) defines a RESTful open standard for operations on storage objects. Semantically the interface is very close to AWS S3 and MS Azure Blob, but is more open and flexible for implementation.

CDMI offers clients a way for operating both on a storage management system and single data items. The exact level of support depends on the concrete implementation and is exposed to the client as part of the protocol.The design of the protocol is aimed both at flexibility and efficiency. Certain heavyweight operations, e.g. blob download, can be performed also with a pure HTTP client to make use of the existing ecosystem of tools. CDMI is built around the concept of Objects, which vary in supported operations and metadata schema. Each Object has an ID, which is unique across all CDMI deployments.

There are 4 CDMI objects most relevant in the context of EGI’s Federated Cloud:

• Data object: Abstraction for a file with rich metadata.
• Container: Abstraction for a folder. Export to non-HTTP protocols is performed on the container level. Container might have other containers inside of them.
• Capability: Exposes information about a feature set of a certain object. CDMI supports partial implementation of the standards by defining optional features and parameters. In order to discover what functionality is supported by a specific implementation, CDMI client can issue a GET request to a fixed url: /cdmi_capabilities.
• Domain: Deployment specific information.

Attachment of the storage items to a VM can often be performed more efficiently using protocols like NFS or iSCSI. CDMI supports exposing of this information via container metadata. A client can make use of this information to attach a storage item to a VM over an OCCI protocol.

Implementations

TBC

Virtual Machine Image Management

In a distributed, federated Cloud infrastructure, users will often face the situation of efficiently managing and distributing their VM Images across multiple resource providers. Users need a catalogue of Virtual Machine images (VMIs) that are usable on the IaaS cloud provider sites and encapsulate those software configurations that are useful and relevant for the given community. (Typically pre-configured scientific models and algorithms). To maximise usability of VMIs across cloud sites the images should be in a format that’s supported at every federation member site (Or at least can be converted to such formats). Users also need a system that automatically replicates VMIs from the VMI catalogue to the federation member sites, as well as removes them when needed. Automated replication can ensure consistency of capabilities across sites and is very often coupled with a VMI vetting process to ensure that only properly working, and relevant VMIs are replicated to the cloud sites of the community

AppDB Cloud MarketPlace

The EGI AppDB service has been extended to a Virtual Appliance Marketplace. This brings about a new category of software entries, called Virtual Appliances (VAs), which are, in all practical manners, clean-and mean virtual machine images designed to run on a virtualization platform, that provide a software solution out-of-the-box, ready to be used with minimal or no set-up needed within the EGI Federated Cloud infrastruture.

HEPiX image lists

AppDB's Virtual Appliance Marketplace provides the ground for managing and publishing versioned repositories of virtual appliances, in a way that integrates with the existing HEPiX VMCaster / VMCatcher framework, currently in use by the EGI.

Research Communities ultimately create and update VM Images (or delegate this functionality). The Images themselves are stored in Appliance repositories that are provided and managed elsewhere, typically by the Research Community itself. A representative of the Research Community then generates a VM Image list (or updates an existing one) using AppDB. Federated Clouds Resource Provider then subscribe to changes in VM Image lists by regularly downloading the list from AppDB, and comparing it against local copies. New and updated VM Images are downloaded from the appliance repository referenced in the VM Image list into a local staging cache and, where required, made available for further examination and assessment.

Ultimately, Cloud resource Providers will make VM Images available for immediate instantiation by the Research Community.

Service Registry

The Service Registry contains general information about the providers participating to the infrastructure and their capabilities. The registry provides the ‘big picture view’ about the federation for both human users and online services (such as service monitors).

GOCDB

EGI’s central service catalogue is used to catalogue the static information of the production infrastructure topology. The service is provided using the GOCDB tool that is developed and deployed within EGI. To allow Resource Providers to expose Cloud resources to the production infrastructure, a number of service types are available:

• eu.egi.cloud.accounting
• eu.egi.cloud.storage-management.cdmi
• eu.egi.cloud.vm-management.occi.
• eu.egi.cloud.vm-metadata.marketplace
• eu.egi.cloud.vm-metadata.vmcatcher
• eu.egi.cloud.vm-metadata.appdb-vmcaster
• org.openstack.nova

All providers must enter cloud service endpoints to GOCDB in order to enable integration with other operational tools.

Higher level broker services also have its own service types:

• eu.egi.cloud.broker.compss
• eu.egi.cloud.broker.proprietary.slipstream
• eu.egi.cloud.broker.vmdirac

Further information about GOCDB can be find on the following page: GOCDB/Input System User Documentation.

Special rules apply for the following service types:

eu.egi.cloud.storage-management.cdmi

Endpoint URL field must contain the following info:

http[s]://hostname:port

eu.egi.cloud.vm-management.occi

Endpoint URL field must contain the following info:

https://hostname:port/?image=<image_name>&resource=<resource_name>

Both <image_name> and <resource_name> cannot contain spaces. These attributes map to os_tpl and resource_tpl respectively.

org.openstack.nova

Endpoint URL field must contain Keystone URL (https://hostname:port/url) with the following additional info:

https://hostname:port/url?image=<image_uuid>&resource=<flavor_name>

Both <image_name> and <flavour_name> cannot contain spaces.

Information system

The information system provides a real-time view about the actual capabilities and load of federation participants. The information system can be used by both human users and online services.

BDII and GlueSchema

Users and tools can discover the available resource in the infrastructure by querying EGI information discovery services. The common information system deployed at EGI is based on the Berkeley Database Information Index (BDII) with a hierarchical structure distributed over the whole infrastructure.

The information system is structured in three levels: the services publish their information (e.g. specific capabilities, total and available capacity or user community supported by the service) using an OGF recommended standard format, GLUE2. The information published by the services is collected by a Site-BDII, a service deployed in every site in EGI. The Site-BDIIs are queried by the Top-BDIIs - a national or regional located level of the hierarchy, which contain the information of all the site services available in the infrastructure and their services. NGIs usually provide an authoritative instance of Top-BDII, but every Top-BDII, if properly configured, should contain the same set of information.

Resource Providers must provide a Site-BDII endpoint that published information on the available resource following the GLUE2 schema. Even if the GLUE2 schema defines generic computing and storage entities, it was developed originally for Grid resources and can represent only partially the information needed by the Cloud users. Thus, the EGI Federated Cloud is working within the GLUE2 WG at OGF to profile and extend the schema to represent Cloud Computing, Storage and in the future Platform and Software services. The proposed extensions are currently under discussion at the WG. EGI provides an implementation for service-level information that generates information supporting OpenStack and OpenNebula, Synnefo support is currently being added. The information is published in a different subtree (Glue2GroupID=cloud) so it can coexist with grid information and is easily discoverable by users.

Information available for each provider:

• Cloud computing resources
• Service endpoint
• Capabilities provided by the service, such as: virtual machine management or snapshot taking. The labels that identify the capabilities are agreed within the taskforce.
• Interface, the type of interface – e.g. webservice or webportal – and the interface name and version, for example OCCI 1.2.0
• User authentication and authorization profiles supported by the service, e.g. X.509 certificates
• Virtual machines images made available by the cloud provider
• Resource templates (number of cores and physical memory) allocable in a virtual machine.

Schema

TBC, Cloud schema specification

Introduction

The EGI Federated Cloud is shifting its model to become a wider collaboration to include several cloud federations (community, private or public clouds), each developing, operating and exploiting clouds according to the preferences, choices and constraints set by its members and users. The EGI Federated Cloud Collaboration acts as a meeting point for these federations to share technologies, services and operational practices among each in order e.g. to lower maintenance cost and to increase sustainability and the speed of innovation.

Cloud Federations

The EGI Federated Cloud is a collaboration of clouds targeting global or specific user communities. Each cloud follows the model shown in the figure:

The federation of IaaS Cloud resources in EGI iss. The federation model for distributed IaaS Cloud resources allows a lightweight aggregation of local Cloud resources into the EGI Cloud Infrastructure Platform (CLIP). At the heart of the federation are the locally deployed Cloud Management stacks.

Every member of the collaboration defines the actual level of integration with the EGI Federated Operations Services, the EGI marketplace and their Federated AAI architecture and technologies. The IaaS services of the cluster should be offered with interfaces that assure the interoperability within the community and whenever possible open standards should be used. The EGI Federated Operations services include the following:

• Information discovery (BDII): The EGI Information Discovery allows users and tools to look for the services that provide the capabilities and the resources to run their activities. It's based on OGF GLUE2 and uses LDAP as protocol.
• Accounting: EGI collects all the CPU and storage information to the Accounting Repository. These data is also provided in a web-based Accounting Portal where users and VO managers have a detailed view of the resources consumed.
• Monitoring (SAM): Resources in the infrastructure are monitored via the Service Availability Monitoring (SAM) system that periodically executes tests on the provided services. The results of the tests are used to calculate Availability and Reliability of the resources.
• Central service registry (GOCDB): GOCDB contains general information about the sites participating to the production infrastructure. It collects information about different entities such as the Operations Centres, the Resource Centres, service endpoints and the contact information and roles of people responsible for operations at different levels.

The EGI MarketPlace provides VM Image management implemented in the EGI Applications Database. AppDB provides a central registration point for virtual appliances which can be endorsed by VO managers and automatically and securely distributed to any resource provider subscribed to the VO virtual appliance lists.

The EGI Cloud Federation includes a key member, the EGI Public Cloud, that is open to any research community and is completely integrated with the EGI production infrastructure. The table below summarizes the differences between the public and community federated clouds:

	Public Cloud	Community Cloud
Users	Open to any researcher community	Targets specific community (or set of communities)
IaaS interfaces	Open standard interfaces for providing the IaaS capabilities	Community chosen interfaces, standards are recommended
VM Image Management	Fully integrated	Optional integration
EGI services integration	Tight integration with EGI core services (information system, monitoring, accounting, GOCDB)	Looser federation profile, may only use a subset of the EGI core services

Public Federated Cloud

The following figure depicts the specific model for the public federated cloud, open to any research community, which is completely integrated with all EGI Core Services and EGI Marketplace, uses open standards (if available) for all the public APIs and uses the current AAI schema of EGI based on X.509 proxies with VOMS extensions.

Public Federated Cloud Architecture

This federation does not mandate deploying any particular or specific Cloud Management stack; it is the responsibility of the Resource Providers to investigate, identify and deploy the solution that fits best their individual needs for as long as the offered services implement the required interfaces and domain languages. These interfaces and domain languages, and the interoperability of their implementation with other solutions are the focus of the federation.

Resource providers in the public Federated Cloud must offer one or both of the following IaaS cloud capabilities by implementing these interfaces:

• IaaS VM management capability:
  • VM management interface: OCCI
  • VM Image management
• IaaS Data Management capability:
  • Data management interface: CDMI

Moreover, to federate a resource provider in the public EGI Federated Cloud, it must integrate with the EGI core services and interfaces listed below:

• Virtual Organisation Management & AAI: VOMS
• Information discovery: BDII
• Monitoring: SAM
• Accounting
• Central service registry: GOCDB

The EGI Federated Clouds Task gives Resource Providers a platform to share their implementation solutions for a commonly deployed specific Cloud Management stack (e.g. OpenNebula and OpenStack). The Federated Cloud resource providers support page is dedicated to the documentation of the steps necessary to integrate a local deployment of a given Cloud Management stack into the EGI Cloud federation. The table below summarizes the current integration level of the participating Cloud Management stacks in the federation:

Cloud Mgmt. Stack	Fed. AAI	Info. Discovery	Monitoring	Accounting	VM Img. Mgmt.	OCCI	CDMI
OpenStack	Yes	Yes	Yes	Yes	Yes	Yes	In progress
OpenNebula	Yes	Yes	Yes	Yes	Yes	Yes	N/A
Synnefo	Yes	Yes	Yes	Yes	Yes	Yes	Yes