Fedcloud-tf:Roadmap

From EGIWiki
Jump to: navigation, search
Main Roadmap and Innovation Technology For Users For Resource Providers Media


Roadmap & Innovation: Roadmap Zero surface for
teaching classes


Road to Production

The EGI Federated IaaS Cloud platform will go into production in the coming year. The following overview indicates the necessary steps and actions towards that goal.

Milestone Cloud Compute Cloud Storage Monitoring Accounting Federated AAI Monitoring SLAs and business models
17 Dec '13
24 Dec '13
31 Dec '13
7 Jan '14
14 Jan '14
21 Jan '14
28 Jan '14
4 Feb '14
11 Feb '14
18 Feb '14
25 Feb '14
4 Mar '14
11 Mar '14
18 Mar '14
25 Mar '14
1 Apr '14
8 Apr '14
15 Apr '14
22 Apr '14
29 Apr '14
6 May '14
13 May '14

EGI Cloud Infrastructure Platform Roadmap

EGI Cloud Infrastructure Platform Roadmap [updated on 04/06/2013]. Information extracted from D2.33 EGI Technical Roadmap

Cloud Infrastructure Platform

Deployed on top of the EGI Core Infrastructure Platform, the EGI Cloud Infrastructure Platform provides “a federated IaaS Cloud infrastructure” based on institutional private IaaS Clouds. It wholly embraces the Cloud paradigm and extends it with a federation mechanism that is partially based on the EGI Core Infrastructure Platform, and partially provides new federation and distribution services geared towards Cloud computing.

The Cloud Infrastructure Platform is a young addition to the EGI ecosystem, and needs time to mature towards integrating it into the EGI production infrastructure. Naturally, the overarching theme of the roadmap for the Cloud Infrastructure Platform is the preparation towards production inclusion in the coming year.

Federated Cloud management services

VM Management

In the EGI Cloud Infrastructure Platform, managing Virtual Machines (not the services and software that are deployed within that virtual machine) is done through the standardised OCCI interface. In principal, this will provide interoperability across different implementations and easier portability of integrations from one Cloud provider to another. Since not all Cloud Management Frameworks within EGI natively support OCCI, this gap is bridged with the rOCCI framework and rOCCI-Server, a server-side implementation of OCCI and an existing integration with OpenNebula. To further improve the VM Management interface the following will be rolled out over the course of the next year:

  • Consistent Authentication using Grid technology: By integrating with the X.509 based PKI used in the exiting EGI production infrastructure, a large number of EGI research communities will be able to access the EGI Cloud Infrastructure without having to obtain new credentials – and use the Cloud infrastructure the same way they use the EGI Grid infrastructure.
  • Re-design of rOCCI-Server: The current architecture of the rOCCI-Server has reached its limits in terms or scalability, maintainability and extensibility. To support integration with more diverse Cloud Management Frameworks, the rOCCI-Server needs a ground-up re-design and implementation.
  • OCCI client library in Java: To foster broader uptake of the OCCI interface on the client side, a Java library will expose an API that is capable of marshalling and un-marshalling OCCI messages (including the Grid Authentication extensions) in a conversation with an OCCI-based Cloud Management Framework.

Data Management

Standards-based Cloud Storage and Data management in EGI has a very similar architecture as the VM Management service: A proxy-server is fronting actual Cloud Management Frameworks that do not support CDMI (the chosen standard in EGI) as their access interface. Based on an existing prototype implementation, the following new features are foreseen in the coming year:

  • Consistent Authentication using Grid technology: To provide consistent Authentication mechanisms across all federated institutional Clouds, the CDMI proxy server needs to integrate with the current Federated AAI deployed in EGI.
  • CRUD operations for Object Storage: Based on the existing Cloud Object Storage feature, and the integration with EGI Federated AAI, full support for CRUD operations (Create, Read, Update, Delete) will be rolled out over the course of the next year.
  • Block storage support: Block storage is an increasingly popular means of Cloud Storage in that it provides storage as a virtualised device that fits perfectly with the device virtualisation paradigm used in Cloud Computing. Users will be able to manage block devices (e.g. create, delete, grow, shrink) and attaching available block devices to OCCI-compliant VMs via the CDMI interface. The exposed functionality will depend on the underlying Cloud Management Framework, with features outside the least common denominator being available through proprietary means only.
  • Extend CDMI metadata support: Particularly in Cloud storage scenarios, metadata plays an important role for not only miscellaneous information about the data itself but also for the automation of data management in larger usage scenarios. Extending the support for metadata that is mandated by the CDMI specification, and metadata that is necessary for CRUD operations on object storage operations (see above), additional metadata will be supported for a number of use cases. Candidates for metadata support (pending analysis) are: Geographic data placement, Data retention policies and Backup policies
  • JavaScript based CDMI storage browser: Users will get early and lightweight access to CDMI-based Cloud storage within and external to the EGI Cloud Infrastructure using a CDMI based Cloud storage browser.
  • SDKs in Python and Java: To facilitate 3rd party tool and platform integration with the EGI Cloud Infrastructure Platform, SDKs for Python and Java will be made available.
  • Accounting integration: Accounting for Cloud storage is necessary for it being integrated into the EGI Cloud Infrastructure Platform. This solution will integrate with the EGI Accounting service provided as part of the EGI Core Infrastructure Platform.

Image management and distribution

VM Images, once made available for use in the Cloud Infrastructure Platform, need to be distributed to the supporting Resource Providers who then in turn may conduct discrete tests before acceptance for instantiation. Also, this capability needs to support the distribution of VM Image updates so that older versions can be superseded. The following enhancements are foreseen:

  • Full deployment of vmcatcher/vmcaster in the testbed: The vmcatcher/vmcaster software toolkit was evaluated by a subset of Cloud resource providers, and it was accepted to be included into the regular toolchain of the Cloud Infrastructure Platform. It now needs to be deployed at all federated Cloud resource providers.
  • Integrate with the OCCI client libraries: Integrating with the OCCI client libraries allows for transparent selection of images and sites for managing the virtual infrastructure on top of the EGI Cloud services.
  • Integrate with the EGI Federated AAI: Only allow certain users in certain groups (i.e. VOs) to publish VM images and image updates.
  • Support single VM Image updates: Formally, each VM image has a unique identifier (a UUID). Consequently a second VM Image that constitutes and update of the first VM Image must have a different unique identifier, yet be flagged as a successor of the first VM Image, so that updates to a VM image can be distributed and managed in an automated way.
  • Support multi-VM image appliance updates: Frequently though not always one VM Image contains software bundled into one “Appliance”. More complex Appliances may require bundling its software into more than one distinct VM Image (e.g. a “head VM” and a “worker VM”). The VM Image distribution infrastructure consequently needs to support distributing updates to Appliances (both full updates and partial updates to the VM Image set).
  • Integration with AppDB: To further increase the attraction of the Cloud Infrastructure Platform, User generated scientific software may be registered in the AppDB as available in pre-packaged VM images that then can be distributed through AppDB acting as a VM Image publisher towards the Image Management infrastructure.

Information Discovery

Information Discovery in the EGI Cloud Infrastructure Platform is used to provide a mix of static and semi-static information about a resource provider in the context of Cloud computing. The purpose of this service is to describe the technical details of a Cloud resource provider’s service offering, either as complementary information to the baseline federated information (for example, which Cloud Management Framework in which version is deployed), or as a means to describe specialised services that provide added value to a subset of the EGI Cloud infrastructure research communities (e.g. specialised AAI integrations, special VM image audit and endorsement procedures).

Effectively re-using components of the EGI Core Infrastructure Information Discovery subsystem the counterpart in the EGI Cloud Infrastructure Platform is using a scaled-down version of the full Information Discovery subsystem deployment. The goal is however, that over time this can be replaced by integrating with the existing Information Discovery subsystem coming from the EGI Core Infrastructure Platform. The reason for this parallel deployment is at the same time the topic for future plans in this area:

  • Finalise Cloud-related GLUE2 extension: Related and synchronised to the EGI GLUE2 profile initiative, extensions to the GLUE2 information model that properly describe Cloud-specific resources need to be finalised, integrated into the profile and eventually implemented.

Core Infrastructure integration

Each higher-level platform (be it a Community Platform or the Clouds Infrastructure Platform) must integrate with a number of capabilities and services provided by the EGI Core Infrastructure Platform. The EGI Cloud Infrastructure Platform plays a pioneering role in this context, as it is the first well-defined platform within the EGI community that integrates in the desired way with the EGI Core Infrastructure Platform.

  • Extend site certification procedures with Cloud aspects: EGI formally certifies sites ready for integration into the federated production infrastructure. This procedure needs to be extended to be able to include sites into the production infrastructure that expose Cloud resources. Volunteer Cloud Resource providing sites will work with the EGI Operations team to evaluate the necessary changes, work with the OMB to approve these changes, and eventually start taking sites through the certification process into production.

Virtual Organisation management & AAI

Virtual Organisation management is a required feature in a federated, distributed infrastructure that is used / shared between multiple user communities, and this is no different for a federated Cloud infrastructure. As this is a cross cutting activity, the integration work with the other management interfaces and capabilities are not mentioned here.

  • Uniform X.509, RFC Proxy certificate and proxy-of-proxy certificate support: Principle X.509 certificate based user authentication and authorisation works in a federated environment, but “falls apart” at the fringes and certain corner cases that are frequent in the Grid infrastructure, for example using proxy certificates that were issued using a proxy certificate. Also, full RFC proxy style authentication is not fully available in the whole federated Clouds infrastructure. Changes to existing Cloud Management frameworks will be deployed over the next year to address these issues.
  • Multi-VO support: Before being used in production, support for more than one VO across the different Cloud Management Frameworks must be implemented uniformly; particularly VO membership across Cloud providers must be maintained so that resource usage per VO will be correctly accounted for.

Monitoring

Currently the integration with the Core Infrastructure Platform Monitoring Service is limited to operating an independent, production quality SAM instance. A number of custom, Cloud-specific Nagios probes are deployed monitoring the general service reachability, OCCI availability and reliability, EGI Accounting compliance, and Information Discovery service compliance. The following enhancements are planned to prepare the Cloud Infrastructure monitoring for regular production use:

  • Implement and deploy a Cloud Storage probe based on CDMI: As EGI is extending its federated Cloud infrastructure to support Cloud Storage. These Cloud storage services need to be monitored as well. A monitoring probe will be developed and deployed in the current independent test-bed SAM instance.
  • Integrate Cloud probes in production Monitoring releases: As part of the site certification procedure all or a subset of the current Cloud monitoring probes need to be integrated into the production SAM releases that are deployed in the NGIs; also. the monitoring profiles for NGIs need to be updated so that NGIs can enable or disable Cloud monitoring when providing production Cloud infrastructure within EGI.

Accounting

Accounting for Cloud resources is currently at an early stage. Based on a small subset of Grid Compute accounting (i.e. Number of cores, wallclock time, allocated RAM, storage for the VM image), the infrastructure has been set up to collect Cloud accounting data from the various Cloud Management Frameworks, and to persist and aggregate them using the current production Accounting service. Nonetheless, until the current solution has been certified for production use, Cloud accounting data is processed using the same infrastructure, but separate persistence entities so as to not interfere with production-grade accounting. The following is anticipated for preparing the Accounting integration into a pre-production state:

  • Extend Cloud accounting to cover Cloud Storage: It is necessary to account for the consumption of Cloud storage just as much for Cloud Computing. An initial set will include both temporal and spatial aspects, i.e. differential accounting in a similar model to the calculation of annual interest rates.

Central Service Catalogue

From the viewpoint of the EGI Core Infrastructure, the EGI Cloud Infrastructure is just another Community Platform and thus required to register the Cloud resources in EGI’s Central Service Catalogue. Since all necessary service types are already available in the current service, no further integration work with the Central Service Registry is planned.