Difference between revisions of "EGI Federated Cloud Roadmap"
Line 43: | Line 43: | ||
| '''AppDB VMOps''' | | '''AppDB VMOps''' | ||
| A single GUI for the EGI Cloud that allows users to manage VMs on different providers just without the need to deal with X.509 certificates and completely integrated with the AppDB Cloud Marketplace. | | A single GUI for the EGI Cloud that allows users to manage VMs on different providers just without the need to deal with X.509 certificates and completely integrated with the AppDB Cloud Marketplace. | ||
| | | [https://dashboard.appdb.egi.eu/vmops Available for testing for selected VOs]. Available on request for new VOs | ||
|- | |- | ||
| '''OCCI 1.2''' | | '''OCCI 1.2''' |
Revision as of 16:18, 21 September 2017
Overview | For users | For resource providers | Infrastructure status | Site-specific configuration | Architecture |
Architecture | Technology | Roadmap | FedCloud Task Force |
The EGI Federated Cloud Roadmap is available at http://go.egi.eu/fedcloud-roadmap and is defined but the TCB-Cloud board.
TCB-Cloud
This board meets regularly to define the short-, mid- and long-term plans for the technology and architectural evolution of the cloud service in EGI. The current roadmap covers the 2017 period and provides hints for the upcoming years
- Chairman: E. Fernandez/EGI Foundation
- Cloud standards, OCCI, OpenNebula: B. Parak/CESNET
- Cloud standards, CDMI, data management: B. Kryza/CYFRONET
- Cloud integration modules: A. Lopez/CSIC
- VM Image Management, Cloud marketplace: M. Chatziangelou/IASA
- INDIGO DataCloud: G. Donvito/INFN
- EGI Technology and Operations: V. Spinoso/INFN, T. Ferrari/EGI Foundation
- Cloud providers:
- M. Antonacci/INFN, RECAS (OpenStack)
- J. Pansanel/France Grilles, IN2P3 (OpenStack);
- V. Tran/II SAS (OpenStack)
- K. Koumantaros/GRNET (Synnefo)
The board develops the roadmap in consultation with:
- the [EGI Federated Cloud task force]
- the User Community Board (UCB) and the team in charge of cloud support
- the Operations Management Board (OMB)
- the Security Coordination Team and AAI experts
- the Service and Solution Board (SSB)
- commercial cloud providers
Short term roadmap (2017)
Usability and accessibility of the EGI Cloud Service
These actions seek to improve the usability and accessibility of the EGI Cloud by removing some of the identified barriers for adoption.
Name | Description | Status |
---|---|---|
AppDB VMOps | A single GUI for the EGI Cloud that allows users to manage VMs on different providers just without the need to deal with X.509 certificates and completely integrated with the AppDB Cloud Marketplace. | Available for testing for selected VOs. Available on request for new VOs |
OCCI 1.2 | The latest version of the OCCI standard facilitates the development of new clients and introduces new features such as snapshotting and resizing VMs. | In production for OpenStack and Synnefo, release candidate for OpenNebula |
OpenID Connect Support | The support of OIDC at the resource providers will allow use of federated identity mechanisms and integration with EGI CheckIn, thus completely removing the need for user's X.509 certificates and easing the development and deployment of web portals without token translation mechanisms. OIDC also allows using federated identity to interact with Command Line Interface tools or directly with APIs | Pilot OpenStack sites under integration. Synnefo and OpenNebula support under development |
IaaS Federated Access Tools | This layer of the EGI Cloud Architecture hides the underlying heterogeneity from users and provides common ways of managing different resources. A set of tools to be used in this layer needs to be investigated and documented. | First set of tools evaluated and documented |
Application portability | Users have difficulties on porting applications from one provider to another due to the big heterogeneity of the site configuration (especially in network). | First survey of site configurations available. Defining recommended configuration for sites |
Site integration
EGI maintains a collection of open source components that facilitate the integration of different Cloud Management Frameworks into the Federation. The following table collects the main developments in this area:
Name | Description | Status |
---|---|---|
CloudKeeper | CloudKeeper is the replacement for the vmcatcher tool for managing VM Images at sites. This component has a more flexible architecture and is ready to be enhanced as needed in the future | Released into CMD, deployed in the Infrastructure |
Monitoring probes | New probes are needed for better assessment of the availability and reliability of the sites. Namely:
|
Probes in testing phase, to be moved to production as sites are stable |
Monitoring fedcloud.egi.eu VO | Monitoring is performed using ops VO, but it may not detect issues in other VOs relevant to the users | Further developments needed in the monitoring framework. Stalled |
Glue 2.1 schema | The Glue2.1 schema includes several improvements for publishing missing information about the providers and the VOs supported. | First prototype implementation ready. Transition plan needs to be defined |
Information discovery transport | BDII is found unsuitable for the current cloud infrastructure and alternatives need to be assessed | No progress |
Accounting for Long Running VMs | Accounting information for VMs running for more than one month is incorrectly assigned the first month the VM has been running. | First implementation ready, testing scalability for going into production |
Long term roadmap (2018 onwards)
New services
TCB-Cloud has identified a set of new services that could attract new communities and/or facilitate the porting of applications for existing ones.
Service | Description | Requirements |
---|---|---|
Filesystem as a Service (FSaaS) | Create shareable filesystems within one site that can scale as needed | Users with needs to share data between VMs but no real need for federation of the data (e.g. running MPI applications)
Access to legacy filesystems at sites (e.g. GPFS, Lustre) |
Container orchestration as a Service | Managed Kubernetes/Swarm/Mesos with elasticity |
Communities with docker-based applications (e.g. ICOS) |
Data Analytics as a Service | Managed Hadoop/Spark/Storm/Galaxy/Jupyter.... with elasticity | Several communities already requesting these services and some of them already integrated in AoD |
HTC cluster as a Service | Managed HTC Cluster (SGE, Slurm, Mesos, Torque) with elasticity | Facilitate migration of existing use cases to the cloud |
VPN as a Service | Enable creation of VPNs easily among members of the federation | Improved security of deployments by reducing risk surface
Remove the need for large pools of public IPs at operators Sensitive data handling use cases |
DNS as Service | Enable DNS management by users, provide names to VMs independent of the IP | Several providers facing issues with existing users requesting this kind of feature |
LB as a Service | Load Balancing for large-scale applications | Simplify the use of distributed infrastructure |
HPC as a Service | Bare-metal like performance for IaaS clouds
Pluggable GPUs, FPGAs |
Enable HPC applications in the cloud (meteo, Terradue) |
Site integration
The core components that enable the EGI Federated Cloud need to be maintained and evolved to maintain compatibility with the resource providers technology and add new features as requested by the user communities. As general activity, the components are expected to:
- Improve their programmability, providing complete APIs specification in adequate format for facilitating the generation clients (e.g. following the OpenAPI initiative) and Swagger.
- Lower the barriers to integrate and operate resource centres in the federation by a) minimizing the number of invasive components used; b) contributing code to upstream distributions; and c) use only public APIs of the Cloud Management Frameworks.
The table below lists the main identified developments needed on each component:
Component | Evolution |
---|---|
Cloud information provider | New transport mechanisms to better handle dynamic information.
Expand sources of information (data, quota information, …) Handle information that is user/VO specific (e.g. quota) |
CloudKeeper | Provide mechanisms to distribute images without delays |
AppDB Cloud Marketplace | Integration with security tools, better control of images
Distribution of software in docker containers (INDIGO) Provide mechanisms to distribute images almost immediately Endorser dashboard Security dashboard VO-wide image list dashboard Docker repository |
AppDB VMOps | Improve contextualization support
Definition of complex topologies (e.g. using IaaS Orchestrator templates) |
AAI | Remove X.509 for users and sites
Complete support for OIDC at the providers and clients Delegation of credentials User provisioning User suspension/banning Better AuthZ control |
Monitoring probes | New probes for any new services developed |
Accounting | Public IP address accounting
Storage accounting Handling of pricing information |
IaaS Federated Access Tools | Evaluation of TOSCA as possible common language for tools
Better AAI integration Elasticity Control Provision of templates for commonly used tools Brokering tools for moving computation near data |
GPGPU support | Complete integration of GPGPUs on FedCloud (accounting, information discovery) |