Difference between revisions of "Fedcloud-tf:WorkGroups:Brokering"

From EGIWiki
Jump to: navigation, search
(Resource Brokers)
Line 94: Line 94:
 
! Cloud SW Support  
 
! Cloud SW Support  
 
! OCCI  
 
! OCCI  
 +
! External API
 
! Functionalities  
 
! Functionalities  
 
! Effort required  
 
! Effort required  
Line 102: Line 103:
 
| OpenStack, OpenNebula, Azure, Vcloud  
 
| OpenStack, OpenNebula, Azure, Vcloud  
 
| Yes, supports OCCI, but implements his own OCCI interface for each of the stacks (PROCCI).  
 
| Yes, supports OCCI, but implements his own OCCI interface for each of the stacks (PROCCI).  
 +
|
 
| Accounting, Brokering, User management, Monitoring.  
 
| Accounting, Brokering, User management, Monitoring.  
 
| <span style="color:orange"> '''Medium'''</span>  
 
| <span style="color:orange"> '''Medium'''</span>  
Line 115: Line 117:
 
| &nbsp; OpenNebula, OpenStack, CloudStack and Amazon EC2  
 
| &nbsp; OpenNebula, OpenStack, CloudStack and Amazon EC2  
 
| Yes, it supports rOCCI, OpenNebula "OCCI 0.8", Amazon EC2, OpenStack Nova 1.1 and CloudStack 2 APIs.  
 
| Yes, it supports rOCCI, OpenNebula "OCCI 0.8", Amazon EC2, OpenStack Nova 1.1 and CloudStack 2 APIs.  
 +
|
 
| Accounting, monitoring, brokering, scheduling, HEPiX and SSH based VM contextualisation, [https://github.com/DIRACGrid/VMDIRAC/wiki#vm-horizontal-auto-scaling-setup VM Horizontal Auto-scaling].  
 
| Accounting, monitoring, brokering, scheduling, HEPiX and SSH based VM contextualisation, [https://github.com/DIRACGrid/VMDIRAC/wiki#vm-horizontal-auto-scaling-setup VM Horizontal Auto-scaling].  
 
| <span style="color:orange"> '''Medium'''</span>  
 
| <span style="color:orange"> '''Medium'''</span>  
Line 132: Line 135:
 
| StratusLab, OpenStack, Abiquo, CloudSigma and Amazon EC2  
 
| StratusLab, OpenStack, Abiquo, CloudSigma and Amazon EC2  
 
| Supports OCCI via rOCCI-cli integration
 
| Supports OCCI via rOCCI-cli integration
 +
| EC2 via bridge (experimental), proprietary SlipStream REST APIs
 
| VM image creation and deployment, contextualisation, VM cluster automated deployments.  
 
| VM image creation and deployment, contextualisation, VM cluster automated deployments.  
 
| <span style="color:orange"> '''Medium'''</span>  
 
| <span style="color:orange"> '''Medium'''</span>  

Revision as of 10:55, 17 December 2013

Main Roadmap and Innovation Technology For Users For Resource Providers Media


Workbenches: Open issues
Scenario 1
VM Management
Scenario 2
Data Management
Scenario 3
Information Systems
Scenario 4
Accounting
Scenario 5
Monitoring
Scenario 6
Notification
Scenario 7
Federated AAI
Scenario 8
VM Image Management
Scenario 9
Brokering
Scenario 10
Contextualisation
Scenario 11
Security



Leader: Alvaro Simon, JR1

Collaborators

Role Institution Name
Collaborator OeRC Matteo Turilli
Collaborator BSC Daniele Lezzi

Scope

This workgroup deals with the issues around cloud brokering. With 10+ resource providers lined up to be federated into the EGI cloud testbed, users need effective ways to access cloud resources. The goal is for a user to have a choice between a unified, abstracted view of the cloud testbed as a whole and the opportunity to target specific providers for their needs. As a consequence, this workgroup is concerned with both brokers and OCCI clients.

Roadmap

  • Collect information about existing solutions for cloud brokering compatible with the OCCI and CDMI management interfaces;
  • Collect information about existing solutions for OCCI and CDMI clients;
  • Make an inventory of the available solutions;
  • Choose a broker and a client for the testbed;
  • If possible, join the development teams of the chosen solutions to the task force;
  • Coordinate with the TF Resorce Providers to deploy the chosen solutions;
  • Extend the demo testbed addressing a real-life use case.

Clients Comparison Table

Client/API URL OS Support OCCI/CDMI Functionalities Effort required Comments
Hybridfox http://code.google.com/p/hybridfox/ Win/OS X/Linux AWS/Eucalyptus/OpenStack/OpenNebula/HP Cloud NO Manage Images/Instances/Elastic IPs/Security Groups/Key-pairs Low Hybridfox does not support x509 auth, it uses EC2 auth for OpenNebula and OpenStack
DeltaCloud http://deltacloud.apache.org Win/OS X/Linux EC2/Eucalyptus/OpenStack/OpenNebula/vSphere NO Create/Start/Stop/Reboot/Destroy instances Medium DeltaCloud uses its own delta-cloud driver for each framework instead of OCCI. It provides storage support for S3, Warlus, Azure and Google Storage.
Aeolus http://aeolusproject.org/about.html Win/OS X/Linux Same as DeltaCloud NO Create/Start/Stop/.. instances. Manage different instances and Images from different private, public, or hybrid cloud providers. Medium Aeolus uses DeltaCloud cross-cloud abstraction library and it includes some extra functionalities. It includes Aeolus Conductor/Composer/Orchestrator/HA Manager
rOCCI (API) http://dev.opennebula.org/projects/ogf-occi Linux OpenNebula/EC2 YES Create/Start/Stop/Reboot/Destroy instances. Upload and register an image. Network conf. x509 auth. Low rOCCI is an OCCI 1.1 implementation for OpenNebula 3.x

Cloud Brokering Solutions

Resource Brokers

Name URL Cloud SW Support OCCI External API Functionalities Effort required Comments
CompatibleOne http://www.compatibleone.org/ OpenStack, OpenNebula, Azure, Vcloud Yes, supports OCCI, but implements his own OCCI interface for each of the stacks (PROCCI). Accounting, Brokering, User management, Monitoring. Medium
  • Compatible One is a complete plattform, with its own user management, accounting and monitoring, so it overlaps in some aspects with the ongoing work.
  • It does not expose an OCCI api, the user has to write its own XML files and send them to the broker.
  • The credentials for each of the connectors and providers have to be configured in advance by the administrator of the broker service.
  • The user has to specify in the manifests the cloud account that he wants to use, and the specific name of the image in the site. It seems that the purpose of CompatibleONE is to deploy and take care of a complete manifest description: A user wants 3 machines of type A in site FOO, and 2 machine of type B in site BAR, so he explicitly defines it in his manifests and CompatibleOne will deploy it on them.
VMDIRAC https://github.com/DIRACGrid/VMDIRAC/wiki   OpenNebula, OpenStack, CloudStack and Amazon EC2 Yes, it supports rOCCI, OpenNebula "OCCI 0.8", Amazon EC2, OpenStack Nova 1.1 and CloudStack 2 APIs. Accounting, monitoring, brokering, scheduling, HEPiX and SSH based VM contextualisation, VM Horizontal Auto-scaling. Medium
  • rOCCI support with X509/VOMS auth.
  • In production: LHCb (OpenStack and OpenNebula) Belle (Amazon), Work in progress France Federated Cloud.
  • Web UI including VM browsing with VM monitoring, and VM Overview Plots (Load, RunningVM, RunnigVM by Endpoint, started jobs, data transfer, transfer files)
  • The number of VMs is dynamically adjusted to the current workload.
  • Automatic selection of the VMs types depending on the tasks constraints.
  • No necessary to install any pluging in the Cloud Managers, works with standard interfaces.
  • Separated user credentials are used for job running and VM operator credentials for VM management.
  • VMDIRAC developers are willing to collaborate actively with Fedcloud Taskforce.
SlipStream™ http://sixsq.com/products/slipstream.html StratusLab, OpenStack, Abiquo, CloudSigma and Amazon EC2 Supports OCCI via rOCCI-cli integration EC2 via bridge (experimental), proprietary SlipStream REST APIs VM image creation and deployment, contextualisation, VM cluster automated deployments. Medium
  • New release 2.0 scheduled for January 2014, with Autoscaling capabilities.
  • Acts as single interface and point of entrance to multiple clouds. Does not provide a brokering mechanism at application level.
  • It is able to use deploy VMs across different cloud frameworks using common deployment scripts.
  • Can deploy clusters concurrently over multiple clouds.
  • Deployments can be configured according to custom parameters (ex. number of nodes, services to install on the VM, etc...)
  • VM contextualization is performed by Slipstream orchestrator (a separately instantiated VM), using SSH or native RP contextualization methods.
  • Autoscaling capabilities will be implemented in 2014. Autoscaling will act at infrastructure level, using CPU/Disk/Memory metrics (not application metrix, ex. number of jobs in queue).
  • RPs don't need to configure new services. It uses native cloud APIs.
  • Provides EC2 (currently in beta) and SlipStream native REST API interface.
  • User do not need to have cloud credentials to instantiate VMs. Slipstream acts as account broker.
  • Does not provide accounting or cost information.
  • Test interface available on the FedCloud for users, to request access please contact salvatore.pinto@egi.eu

Application Brokers

Name URL Cloud SW Support OCCI Functionalities Effort required Comments
COMPSs http://www.bsc.es/compss   OpenNebula, Amazon, Azure, OpenStack Yes, support of rOCCI servers and OCCI+OVF  Automatic parallelization and orchestration of applications and services, elasticity, auto scaling Medium
  • COMPSs allows the execution of applications on distributed infrastructures (grids, clusters, cloud).
  • Already used for the BioVeL/OpenBio use case.
  • Transparent support to different infrastructures; the user only provides his credentials.
  • Automatic selection of the VMs types dependending on the tasks constraints.
  • The number of VMs is dynamically adjusted to the current workload.
VMDIRAC https://github.com/DIRACGrid/VMDIRAC/wiki   OpenNebula, OpenStack, CloudStack and Amazon EC2 Yes, it supports rOCCI, OpenNebula "OCCI 0.8", Amazon EC2, OpenStack Nova 1.1 and CloudStack 2 APIs. Accounting, monitoring, brokering, scheduling, HEPiX and SSH based VM contextualisation, VM Horizontal Auto-scaling. Medium
  • rOCCI support with X509/VOMS auth.
  • In production: LHCb (OpenStack and OpenNebula) Belle (Amazon), Work in progress France Federated Cloud.
  • Web UI including VM browsing with VM monitoring, and VM Overview Plots (Load, RunningVM, RunnigVM by Endpoint, started jobs, data transfer, transfer files)
  • The number of VMs is dynamically adjusted to the current workload.
  • Automatic selection of the VMs types depending on the tasks constraints.
  • No necessary to install any pluging in the Cloud Managers, works with standard interfaces.
  • Separated user credentials are used for job running and VM operator credentials for VM management.
  • VMDIRAC developers are willing to collaborate actively with Fedcloud Taskforce.