EGI Federated Cloud Roadmap

Overview

For users

For resource providers

Infrastructure status

Site-specific configuration

Architecture

Architecture

Technology

Roadmap

FedCloud Task Force

The EGI Federated Cloud Roadmap is available at http://go.egi.eu/fedcloud-roadmap and is defined but the TCB-Cloud board.

TCB-Cloud

This board meets regularly to define the short-, mid- and long-term plans for the technology and architectural evolution of the cloud service in EGI. The current roadmap covers the 2017 period and provides hints for the upcoming years

• Chairman: E. Fernandez/EGI Foundation
• Cloud standards, OCCI, OpenNebula: B. Parak/CESNET
• Cloud standards, CDMI, data management: B. Kryza/CYFRONET
• Cloud integration modules: A. Lopez/CSIC
• VM Image Management, Cloud marketplace: M. Chatziangelou/IASA
• INDIGO DataCloud: G. Donvito/INFN
• EGI Technology and Operations: V. Spinoso/INFN, T. Ferrari/EGI Foundation
• Cloud providers:
  • M. Antonacci/INFN, RECAS (OpenStack)
  • J. Pansanel/France Grilles, IN2P3 (OpenStack);
  • V. Tran/II SAS (OpenStack)
  • K. Koumantaros/GRNET (Synnefo)

The board develops the roadmap in consultation with:

• the [EGI Federated Cloud task force]
• the User Community Board (UCB) and the team in charge of cloud support
• the Operations Management Board (OMB)
• the Security Coordination Team and AAI experts
• the Service and Solution Board (SSB)
• commercial cloud providers

Short term roadmap (2017)

Usability and accessibility of the EGI Cloud Service

These actions seek to improve the usability and accessibility of the EGI Cloud by removing some of the identified barriers for adoption.

Name	Description	Status
AppDB VMOps	A single GUI for the EGI Cloud that allows users to manage VMs on different providers just without the need to deal with X.509 certificates and completely integrated with the AppDB Cloud Marketplace.	Available for testing for selected VOs. Available on request for new VOs
OCCI 1.2	The latest version of the OCCI standard facilitates the development of new clients and introduces new features such as snapshotting and resizing VMs.	In production for OpenStack and Synnefo, release candidate for OpenNebula
OpenID Connect Support	The support of OIDC at the resource providers will allow use of federated identity mechanisms and integration with EGI CheckIn, thus completely removing the need for user's X.509 certificates and easing the development and deployment of web portals without token translation mechanisms. OIDC also allows using federated identity to interact with Command Line Interface tools or directly with APIs	Pilot OpenStack sites under integration. Synnefo and OpenNebula support under development
IaaS Federated Access Tools	This layer of the EGI Cloud Architecture hides the underlying heterogeneity from users and provides common ways of managing different resources. A set of tools to be used in this layer needs to be investigated and documented.	First set of tools evaluated and documented
Application portability	Users have difficulties on porting applications from one provider to another due to the big heterogeneity of the site configuration (especially in network).	First survey of site configurations available. Defining recommended configuration for sites

Site integration

EGI maintains a collection of open source components that facilitate the integration of different Cloud Management Frameworks into the Federation. The following table collects the main developments in this area:

Name	Description	Status
CloudKeeper	CloudKeeper is the replacement for the vmcatcher tool for managing VM Images at sites. This component has a more flexible architecture and is ready to be enhanced as needed in the future	Released into CMD, deployed in the Infrastructure
Monitoring probes	New probes are needed for better assessment of the availability and reliability of the sites. Namely: check the correct subscription to VM image lists complete checking of OCCI types use of EGI-provided images for testing VM instantiation functionality	Probes in testing phase, to be moved to production as sites are stable
Monitoring fedcloud.egi.eu VO	Monitoring is performed using ops VO, but it may not detect issues in other VOs relevant to the users	Further developments needed in the monitoring framework. Stalled
Glue 2.1 schema	The Glue2.1 schema includes several improvements for publishing missing information about the providers and the VOs supported.	First prototype implementation ready. Transition plan needs to be defined
Information discovery transport	BDII is found unsuitable for the current cloud infrastructure and alternatives need to be assessed	No progress
Accounting for Long Running VMs	Accounting information for VMs running for more than one month is incorrectly assigned the first month the VM has been running.	First implementation ready, testing scalability for going into production

Long term roadmap (2018 onwards)

New services

TCB-Cloud has identified a set of new services that could attract new communities and/or facilitate the porting of applications for existing ones.

Service	Description	Requirements
Filesystem as a Service (FSaaS)	Create shareable filesystems within one site that can scale as needed	Users with needs to share data between VMs but no real need for federation of the data (e.g. running MPI applications) Access to legacy filesystems at sites (e.g. GPFS, Lustre)
Container orchestration as a Service	Managed Kubernetes/Swarm/Mesos with elasticity	Communities with docker-based applications (e.g. ICOS)
Data Analytics as a Service	Managed Hadoop/Spark/Storm/Galaxy/Jupyter.... with elasticity	Several communities already requesting these services and some of them already integrated in AoD
HTC cluster as a Service	Managed HTC Cluster (SGE, Slurm, Mesos, Torque) with elasticity	Facilitate migration of existing use cases to the cloud
VPN as a Service	Enable creation of VPNs easily among members of the federation	Improved security of deployments by reducing risk surface Remove the need for large pools of public IPs at operators Sensitive data handling use cases
DNS as Service	Enable DNS management by users, provide names to VMs independent of the IP	Several providers facing issues with existing users requesting this kind of feature
LB as a Service	Load Balancing for large-scale applications	Simplify the use of distributed infrastructure
HPC as a Service	Bare-metal like performance for IaaS clouds Pluggable GPUs, FPGAs	Enable HPC applications in the cloud (meteo, Terradue)

Site integration

The core components that enable the EGI Federated Cloud need to be maintained and evolved to maintain compatibility with the resource providers technology and add new features as requested by the user communities. As general activity, the components are expected to:

• Improve their programmability, providing complete APIs specification in adequate format for facilitating the generation clients (e.g. following the OpenAPI initiative) and Swagger.
• Lower the barriers to integrate and operate resource centres in the federation by a) minimizing the number of invasive components used; b) contributing code to upstream distributions; and c) use only public APIs of the Cloud Management Frameworks.

The table below lists the main identified developments needed on each component:

Component	Evolution
Cloud information provider	New transport mechanisms to better handle dynamic information. Expand sources of information (data, quota information, …) Handle information that is user/VO specific (e.g. quota)
CloudKeeper	* Provide mechanisms to distribute images without delays
AppDB Cloud Marketplace	Integration with security tools, better control of images Distribution of software in docker containers (INDIGO) Provide mechanisms to distribute images almost immediately Endorser dashboard Security dashboard VO-wide image list dashboard Docker repository
AppDB VMOps	Improve contextualization support Definition of complex topologies (e.g. using IaaS Orchestrator templates)
AAI	Remove X.509 for users and sites Complete support for OIDC at the providers and clients Delegation of credentials User provisioning User suspension/banning Better AuthZ control
Monitoring probes	*New probes for any new services developed
Accounting	Public IP address accounting Storage accounting Handling of pricing information
IaaS Federated Access Tools	Evaluation of TOSCA as possible common language for tools Better AAI integration Elasticity Control Provision of templates for commonly used tools Brokering tools for moving computation near data
GPGPU support	Complete integration of GPGPUs on FedCloud (accounting, information discovery)