Federated Cloud Brokering

From EGIWiki
Jump to: navigation, search
Overview For users For resource providers Infrastructure status Site-specific configuration Architecture



Scenarios: Federated AAI Accounting VM Image Management Brokering IntraCloud Networking
Monitoring VM Management Data Management Information Discovery Security


Scope

This workgroup deals with the issues around cloud brokering. With 10+ resource providers integrated into the EGI federated cloud, users need effective ways to access cloud resources. The goal is for a user to have a choice between a unified, abstracted view of the cloud testbed as a whole and the opportunity to target specific providers for their needs. As a consequence, this workgroup is concerned with both brokers and OCCI clients.

Members

Role Institution Name
Leader EGI.eu Enol Fernández (interim)
Collaborator BSC Daniele Lezzi

Roadmap

  • Collect information about existing solutions for cloud brokering compatible with the OCCI and CDMI management interfaces;
  • Collect information about existing solutions for OCCI and CDMI clients;
  • Make an inventory of the available solutions;
  • Choose a broker and a client for the testbed;
  • If possible, join the development teams of the chosen solutions to the task force;
  • Coordinate with the TF Resorce Providers to deploy the chosen solutions;
  • Extend the demo testbed addressing a real-life use case.

Documentation

Clients Comparison Table

Client/API URL OS Support OCCI/CDMI Functionalities Effort required Comments
Hybridfox http://code.google.com/p/hybridfox/ Win/OS X/Linux AWS/Eucalyptus/OpenStack/OpenNebula/HP Cloud NO Manage Images/Instances/Elastic IPs/Security Groups/Key-pairs Low Hybridfox does not support x509 auth, it uses EC2 auth for OpenNebula and OpenStack
DeltaCloud http://deltacloud.apache.org Win/OS X/Linux EC2/Eucalyptus/OpenStack/OpenNebula/vSphere NO Create/Start/Stop/Reboot/Destroy instances Medium DeltaCloud uses its own delta-cloud driver for each framework instead of OCCI. It provides storage support for S3, Warlus, Azure and Google Storage.
Aeolus http://aeolusproject.org/about.html Win/OS X/Linux Same as DeltaCloud NO Create/Start/Stop/.. instances. Manage different instances and Images from different private, public, or hybrid cloud providers. Medium Aeolus uses DeltaCloud cross-cloud abstraction library and it includes some extra functionalities. It includes Aeolus Conductor/Composer/Orchestrator/HA Manager
rOCCI-cli https://github.com/gwdg/rOCCI-cli Linux OpenNebula (rOCCI-server)/OpenStack (OCCI-OS)/Synnefo (snf-occi) YES Create/Start/Stop/Reboot/Destroy instances. Dynamic linking, actions on resources, contextualization. Supports X.509 and VOMS auth. Low rOCCI-cli is an OCCI 1.1 client implementation compatible with multiple server implementations.

Cloud Brokering Solutions

Resource Brokers

Name URL Cloud SW Support OCCI External API Functionalities Effort required Comments
CompatibleOne http://www.compatibleone.org/ OpenStack, OpenNebula, Azure, Vcloud Yes, supports OCCI, but implements his own OCCI interface for each of the stacks (PROCCI). Accounting, Brokering, User management, Monitoring. High
  • Compatible One is a complete plattform, with its own user management, accounting and monitoring, so it overlaps in some aspects with the ongoing work.
  • It does not expose an OCCI api, the user has to write its own XML files and send them to the broker.
  • The credentials for each of the connectors and providers have to be configured in advance by the administrator of the broker service.
  • The user has to specify in the manifests the cloud account that he wants to use, and the specific name of the image in the site. It seems that the purpose of CompatibleONE is to deploy and take care of a complete manifest description: A user wants 3 machines of type A in site FOO, and 2 machine of type B in site BAR, so he explicitly defines it in his manifests and CompatibleOne will deploy it on them.
VMDIRAC https://github.com/DIRACGrid/VMDIRAC/wiki   OpenNebula, OpenStack, CloudStack and Amazon EC2 Yes, it supports rOCCI, OpenNebula "OCCI 0.8", Amazon EC2, OpenStack Nova 1.1 and CloudStack 2 APIs. Accounting, monitoring, brokering, scheduling, HEPiX and SSH based VM contextualisation, VM Horizontal Auto-scaling. Medium
  • rOCCI support with X509/VOMS auth.
  • In production: LHCb (OpenStack and OpenNebula) Belle (Amazon), Work in progress France Federated Cloud.
  • Web UI including VM browsing with VM monitoring, and VM Overview Plots (Load, RunningVM, RunnigVM by Endpoint, started jobs, data transfer, transfer files)
  • The number of VMs is dynamically adjusted to the current workload.
  • Automatic selection of the VMs types depending on the tasks constraints.
  • No necessary to install any pluging in the Cloud Managers, works with standard interfaces.
  • Separated user credentials are used for job running and VM operator credentials for VM management.
  • VMDIRAC developers are willing to collaborate actively with Fedcloud Taskforce.
SlipStream™ http://sixsq.com/products/slipstream.html StratusLab, OpenStack, Abiquo, CloudSigma and Amazon EC2 Supports OCCI via rOCCI-cli integration EC2 via bridge (experimental), proprietary SlipStream REST APIs VM image creation and deployment, contextualisation, VM cluster automated deployments. Medium
  • New release 2.0 scheduled for January 2014, with Autoscaling capabilities.
  • Acts as single interface and point of entrance to multiple clouds. Does not provide a brokering mechanism at application level.
  • It is able to use deploy VMs across different cloud frameworks using common deployment scripts.
  • Can deploy clusters concurrently over multiple clouds.
  • Deployments can be configured according to custom parameters (ex. number of nodes, services to install on the VM, etc...)
  • VM contextualization is performed by Slipstream orchestrator (a separately instantiated VM), using SSH or native RP contextualization methods.
  • Autoscaling capabilities will be implemented in 2014. Autoscaling will act at infrastructure level, using CPU/Disk/Memory metrics (not application metrix, ex. number of jobs in queue).
  • RPs don't need to configure new services. It uses native cloud APIs.
  • Provides EC2 (currently in beta) and SlipStream native REST API interface.
  • User do not need to have cloud credentials to instantiate VMs. Slipstream acts as account broker.
  • Does not provide accounting or cost information.
  • Test interface available on the FedCloud for users, to request access please contact salvatore.pinto@egi.eu

Application Brokers

Name URL Cloud SW Support OCCI External API Functionalities Effort required Comments
COMPSs http://www.bsc.es/compss   OpenNebula, Amazon, Azure, OpenStack Yes, support of rOCCI servers and OCCI+OVF OGF-BES SOAP  Automatic parallelization and orchestration of applications and services, elasticity, auto scaling Medium
  • COMPSs allows the execution of applications on distributed infrastructures (grids, clusters, cloud).
  • Already used for the BioVeL/OpenBio use case.
  • Transparent support to different infrastructures; the user only provides his credentials.
  • Automatic selection of the VMs types dependending on the tasks constraints.
  • The number of VMs is dynamically adjusted to the current workload.
VMDIRAC https://github.com/DIRACGrid/VMDIRAC/wiki   OpenNebula, OpenStack, CloudStack and Amazon EC2 Yes, it supports rOCCI, OpenNebula "OCCI 0.8", Amazon EC2, OpenStack Nova 1.1 and CloudStack 2 APIs. Accounting, monitoring, brokering, scheduling, HEPiX and SSH based VM contextualisation, VM Horizontal Auto-scaling. Medium
  • rOCCI support with X509/VOMS auth.
  • In production: LHCb (OpenStack and OpenNebula) Belle (Amazon), Work in progress France Federated Cloud.
  • Web UI including VM browsing with VM monitoring, and VM Overview Plots (Load, RunningVM, RunnigVM by Endpoint, started jobs, data transfer, transfer files)
  • The number of VMs is dynamically adjusted to the current workload.
  • Automatic selection of the VMs types depending on the tasks constraints.
  • No necessary to install any pluging in the Cloud Managers, works with standard interfaces.
  • Separated user credentials are used for job running and VM operator credentials for VM management.
  • VMDIRAC developers are willing to collaborate actively with Fedcloud Taskforce.

References