Difference between revisions of "Fedcloud-tf:WorkGroups:Scenario3"

From EGIWiki
Jump to: navigation, search
 
(25 intermediate revisions by 11 users not shown)
Line 1: Line 1:
{{Fedcloud-tf:Menu}} {{Fedcloud-tf:WorkGroups:Menu}} {{TOC_right}}
+
#REDIRECT[[Federated_Cloud_Information_Discovery]]
 
 
== Scenario 3: Integrating information from multiple resource providers  ==
 
 
 
<font color="red">Leader: David Wallom, OeRC</font>
 
 
 
== Scenario collaborators  ==
 
 
 
{| border="1"
 
|-
 
! Role
 
! Institution
 
! Name
 
|-
 
| Scenario leader
 
| OeRC
 
| David Wallom
 
|-
 
| Collaborator
 
| OeRC
 
| Matteo Turilli
 
|-
 
| Collaborator
 
| EGI.eu
 
| Peter Solagna
 
|-
 
| Collaborator
 
| INFN
 
| Elisabetta Ronchieri
 
|}
 
 
 
== Information that should be published by a cloud service  ==
 
 
 
The following are the information identified during the TF F2F meeting:
 
 
 
'''Please add more points edit/comments the list'''
 
 
 
#What is the name of the resource and what type of interface can I use to manage instances on the resource?
 
##What is the endpoint I should contact to interact with the cloud management interface? (E.g. the url of the web-service/portal)
 
#What are the AuthN and AuthZ rules that operate on your cloud?
 
#What instances are already installed on the resource and am I allowed to upload my own instances?
 
#If I am able to upload instances what format of instances does the resource accept?
 
#Is there a data interface available and if so what is it?
 
#What is the overall size of the resource?
 
#Are instance templates defined that limit the choice of instance scales I am able to run?
 
#What type of virtual network can I establish on the resource?
 
#Does the resource support cloud scalability through managed bursting to another external provider?
 
 
 
The following are questions on the dynamic information;
 
 
 
#I have a virtual instance that requires X,Y,Z resources, does your cloud have A&gt;X, B&gt;Y,C&gt;Z resource available?
 
#My instance is short lived is its utilisation of resources going to be captured in the information system such that overprovisioning will/will not occur?
 
#What is the charging scheme and how much will using your cloud cost?
 
 
 
== More information to publish==
 
=== Storage capabilities===
 
In this section the storage capabilities information to be published are analyzed not (necessarily) considering the possibilities in the GLUE2 schema currently available.
 
 
 
==== Relevant information ====
 
The following table contains what is possible to insepct through the OCCI 1.2 spec:
 
 
 
{| border="1"
 
! Attribute
 
!Comment
 
|-
 
|occi.storage.size
 
|Size of the storage resource instance
 
|-
 
|occi.storage.status
 
|Status of the storage instance (online,offline,backup,snapshot..)
 
|}
 
 
 
These attributes are describing an actual instance, which is not going to be published in the information system. What we want to advertise are the capabilities that could be requested to the cloud service.
 
 
 
{| border="1"
 
! Attribute
 
!Comment
 
|-
 
|Max Storage installed in the site
 
|This is the total amount of disk space that the cloud site provides as virtualized storage resource.
 
|-
 
|Max size of a single virtual storage resource
 
|This is the max size of storage
 
|-
 
|Interfaces
 
|How users and VMs can interact with the storage resources. E.g. CDMI.
 
|-
 
|Storage throughput
 
|Max I/O speed allowed to VMs writing/reading to the storage area.
 
|-
 
|Capabilities
 
|This are additional capabilities of a storage service, on top of the create/delete/link. Examples could be: backup or snapshot.
 
|}
 
 
 
''Please add more options in the table'', or participate to the discussion in the FedCloud task force mailing list.
 
 
 
=== Network capabilities ===
 
 
 
The following table contains some of the Network capabilities that could be advertised through the information system:
 
 
 
{| border="1"
 
! Attribute
 
!Comment
 
|-
 
|Internal Bandwith
 
|The maximum bandwith available between the virtual machines in the cloud
 
|-
 
|Outbound bandwith
 
|Bandwith that can be allocated to each virtual machine outside the cloud
 
|-
 
|Average latency
 
|If the VM are deployed on different physical sites the latency between the instances can be higher and affect network performances. Low values of network latency assure that the virtual machine are physically instantiated in the same network.
 
|-
 
|IPv6 enabled
 
|Can the virtual network be configured for IPv6?
 
|-
 
|Virtual private network enabled
 
|Is it possible to set up a virtual private network, in order to increase the security and the isolation of the instantiated machines?
 
|}
 
 
 
== How to render those information in GLUE2  ==
 
 
 
'''Note''': BDII service speaks only GLUE2. The Cloud information need to be squeezed in the current set of GLUE2 Entities. If the schema is extended to include Cloud-specific entities, it needs to be officially approved by OGF and implemented in the various ''glue-schema'' ''glue-validator'' components deployed with the BDII.
 
 
 
=== Use the currently available GLUE2.0 entities  ===
 
 
 
Currently the GLUE2 includes two main conceptual models for Computing Elements and Storage Elements. These elements should be used to model the Cloud capabilities remaining compliant to the current GLUE2.0 schema.
 
 
 
==== Capabilities for cloud services  ====
 
 
 
''Note: '''bold''' capabilities are new, not already in GLUE2 specification. Adding new capabilities do not requires an extension of the GLUE2 schema.<br>'' ''Please:'' add new ''high level'' capabilities if you feel that something is missing. These capabilities are used in the following entities.
 
 
 
{| border="1"
 
|-
 
! Capability
 
! Description
 
|-
 
| '''cloud.VMmanagement'''
 
| This is the '''standard''' capability that every cloud service should publish if it allows to instantiate/suspend/delete virtual machines
 
|-
 
| '''cloud.virtualImagesUpload'''
 
| This is the capability that allows users to upload their own virtual images through the cloud interface
 
|-
 
| security.authentication/security.authorization
 
| I would leave those capability, given that every cloud provider has authentication
 
|-
 
|
 
|
 
|}
 
 
 
==== Computing Service entity description  ====
 
 
 
*This Service is used to describe the computing resource itself, decoupling from the Grid endpoint.
 
*Attributes that need to be provided by the resource providers are in '''bold'''
 
 
 
{| border="1"
 
|-
 
! Attribute
 
! Type
 
! Multiplicity
 
! Description
 
|-
 
| Creation time
 
| ..
 
| ..
 
| ..
 
|-
 
| Validity
 
| ..
 
| ..
 
| ..
 
|-
 
| ID
 
| ..
 
| ..
 
| ..
 
|-
 
| '''Name'''
 
| String
 
| 1
 
| Human readable name. It could be used to fill the information: "what is the name of the resource"
 
|-
 
| OtherInfo
 
| String
 
| n
 
| Placeholder to add information that does not fit into any other attribute. Cloud information that cannot be mapped in other attributes could be added here.
 
|-
 
| '''Capability'''
 
| Capability_t
 
| n
 
| This attribute lists the capabilities available for this service, currently the type ''Capability_t'' does not include specific cloud capabilities. Being an open enum type it can be extended with additional capabilities. Currently some of the already available capabilities are: security.accounting, security.authentication or information.logging. We could consider to add capabilities like "''cloud.vm.uploadImage''" to add the information in the quesiton: "am I allowed to upload my own instances?". To identify cloud services there would be the need to add a new capability, common to all the cloud services regardless of their specific capabilities, like: "cloud.managementSystem" (nb: stupid example). ''Resource providers, in this design stage, could provide just descriptions of the capabilities they would like to publish. I (Peter) will try to group them proposing some labels for the different capabilities.''
 
|-
 
| '''Type'''
 
| ServiceType_t
 
| 1
 
| Type of service in a reverse namespace model, e.g.: org.glite.lb or org.glite.wms. It could be ''org.opennebula'', ''org.stratuslab'' or ''com.cloudsigma''
 
|}
 
 
 
There are, then, a number of more attributes (static and dynamic) that could be used by cloud services, like: StatusInfo,TotalJobs, RunningJobs etc etc. Please note that '''Location''' is a GLUE2 entity that can be linked to the Service entity, this could answer to the ''"Where is located the cloud facility?"'' question.
 
 
 
=== ComputingEndpoint description  ===
 
 
 
Every ComputingService has associated '''one or more''' Computing Endpoint. The endpoint is used to create, control am monitor computational activities.<br>
 
 
 
*Resource providers should provide the information to create one endpoint for each interface they're exposing for the cloud service.
 
 
 
{| border="1"
 
|-
 
! Attribute
 
! Type
 
! Multiplicity
 
! Description
 
|-
 
| CreationTime
 
| ..
 
| ..
 
| I will skip the most general, attributes like OtherInfo and Capability(described above).
 
|-
 
| '''URL'''
 
| URI
 
| 1
 
| Network location of the endpoint.
 
|-
 
| '''Capability'''
 
| Capability_t
 
| 0..n
 
| It's the same field of the Service entity. Some capability could be interface-specific. I would replicate all the general capability also for this instance.
 
|-
 
| '''Technology'''
 
| EndpointTechnology_t
 
| 1
 
| Examples are "webservice" and "corba". We could add "webportal" or something like this to clarify that the endpoint refers to a web application.
 
|-
 
| '''InterFaceName'''
 
| InterFaceName_t
 
| 1 (mandatory)
 
| The interface in the cloud case could be ''OCCI'', ''EC2'', ''jclouds'' or "webinterface". This can answer to the question: "what type of interface can I use to manage instances on the resource?"
 
|-
 
| '''InterfaceVersion'''
 
| ..
 
| ..
 
| No description needed.
 
|-
 
| '''Supported profile'''
 
| URI
 
| *
 
| We can define, here, a set of profiles for the authN/authZ of the users, like ''uri:sec:x509''.
 
|}
 
 
 
==== ExecutionEnvironment  ====
 
 
 
The ExecutionEnvironment class describes the hardware and operating system environment in which a job will run. It could be used to describe the VM images already available in the Cloud service.
 
 
 
{| border="1"
 
|-
 
! Attribute
 
! Type
 
! Multiplicity
 
! Description
 
|-
 
| '''Platform'''
 
| Platform_t
 
| 1
 
| The platform atchitecture, can be: amd64,i386,itanum,powerpc,sparc
 
|-
 
| TotalInstances/used instances
 
| -
 
| -
 
| These attributes are not relevant in a cloud environment, where the execution environment are deployed dynamically.
 
|-
 
| PhysicalCPUs
 
| UInt32
 
| 0..1
 
| The physical CPUs are not relevant - I would say- in a virtualised environment.
 
|-
 
| '''LogicalCPUs'''
 
| UInt32
 
| 0..1
 
| This attribute could be used to express the '''maximum''' number of cores that is possible to instantiate in a single VM of this type (likely it will be common to all the execution environments of the same cloud service).
 
|-
 
| '''MainMemorySize'''
 
| UInt64
 
| 1
 
| Max physical memory that is possible to instantiate on a single VM.
 
|-
 
|
 
*'''OSFamily'''
 
*'''OSName'''
 
*'''OSVersion'''
 
 
 
| (*)
 
| 1
 
| Attributes which define the operating system available. There will be an execution environment for every virtual machine available in the cloud service. We should define some placeholders to create an ExecutionEnvironment ''stub'' to describe the max cores/memory for the virtual machines uploaded by a user.
 
|}
 
 
 
=== Deploy a new set of entities  ===
 
 
 
This is the next step: define cloud specific GLUE entities to extend the GLUE2 schema in order to publish the cloud services in a standard way. <!-- What to model?
 
    What is the name of the resource and what type of interface can I use to manage instances on the resource?
 
        What is the endpoint I should contact to interact with the cloud management interface? (E.g. the url of the web-service/portal)
 
    What are the AuthN and AuthZ rules that operate on your cloud?
 
    What instances are already installed on the resource and am I allowed to upload my own instances?
 
    If I am able to upload instances what format of instances does the resource accept?
 
    Is there a data interface available and if so what is it?
 
    What is the overall size of the resource?
 
    Are instance templates defined that limit the choice of instance scales I am able to run?
 
    What type of virtual network can I establish on the resource?
 
    Does the resource support cloud scalability through managed bursting to another external provider?
 
 
 
The following are questions on the dynamic information;
 
 
 
    I have a virtual instance that requires X,Y,Z resources, does your cloud have A>X, B>Y,C>Z resource available?
 
    My instance is short lived is its utilisation of resources going to be captured in the information system such that overprovisioning will/will not occur?
 
    What is the charging scheme and how much will using your cloud cost?
 
-->
 
 
 
= Technical implementation  =
 
 
 
For a first demo the best technical choice is to go for openldap, which is available in almost all the *nix machines in the world. On top of that, openldap is the server used by the gLite BDIIs, therefore it would be easy to use the same configuration files set-up used for the GRIS or the GIIS.
 
 
 
* Host of the ldap server: '''ldap://fedclouds-is.hellasgrid.gr:2170'''
 
** Backup hosted by CESGA: '''fedclouds-is2.hellasgrid.gr'''
 
*Use the GLUE20.schema in the ''slapd.conf'' file to enable all the GLUE2.0 entities.
 
 
 
== Resource providers to be published for the demo  ==
 
 
 
'''!!! NEW !!!:''' ''IMPORTANT'': <span style="color:#DC143C">Fill the table with the address of the LDAP server set up as an information provider for your testbed.</span>
 
{| border="1"
 
|-
 
! RP Name
 
! Resource Centre
 
! Address of the LDAP server "ldap://hostname:2170"
 
|-
 
|CESNET
 
|CESNET Cloud
 
|ldap://carach5.ics.muni.cz:2170
 
|-
 
|GWDG
 
|GWDG Cloud
 
|ldap://one.cloud.gwdg.de:2170
 
|-
 
|SARA
 
|SARA Cloud
 
|ldap://bdii.cloud.sara.nl:2170
 
|-
 
|
 
|
 
|
 
|-
 
|}
 
 
 
'''Olda table for the CF2012 demo'''
 
{| border="1"
 
|-
 
! RP Name
 
! RP contact name
 
! Resource Centre name to be published (was Site Name)
 
! Country
 
! Capabilities to be published (specify the endpoints supporting the capabilities!)
 
! Other info to publish
 
! VM Manager
 
! V.Images available (OSFamily,OSName,OSVersion)
 
! Max cores
 
! Max CPU speed
 
! Max RAM
 
|-
 
| CESNET
 
| Miroslav Ruda
 
| CESNET Cloud
 
| Czech Republic
 
| cloud.managementSystem, cloud.vm.uploadImage, cloud.data.cdmi
 
|
 
| XEN
 
| 1.) Linux, OpenSUSE, 11.4<br>2.) Linux, Debian, 6.0.3
 
| 24
 
|
 
| 96GB
 
|-
 
| KTH
 
| Zeeshan Ali&nbsp;Shah
 
| KTH-PDC Cloud
 
| Sweden
 
| cloud.managementSystem, cloud.vm.customimage, cloud.data.cdmi
 
|
 
|
 
|
 
|
 
|
 
|
 
|-
 
| GWDG
 
| Piotr Kasprzak
 
| GWDG Cloud
 
| Germany
 
| cloud.managementSystem, cloud.vm.uploadImage
 
|
 
| KVM
 
| 1.) Linux, Scientific Linux, 6.1<br>2.) Linux, Ubuntu, 11.10
 
| 8
 
| 2.4 GHZ
 
| 16GB
 
|-
 
| CYFRONET
 
| Jan Meizner
 
| CYFRONET Cloud
 
| Poland
 
| cloud.managementSystem, cloud.vm.uploadImage
 
|
 
| KVM
 
|
 
| 24
 
|
 
| 48GB
 
|-
 
| CESGA
 
| Alvaro Simon
 
| CESGA Cloud
 
| Spain
 
| cloud.managementSystem, cloud.vm.customimage
 
|
 
| KVM
 
| 1.) Linux, Scientific Linux, 5.5
 
|
 
|
 
|
 
|}
 
 
 
== Distributed implementation ==
 
 
 
Publishing correct information in the information system must be responsibility of the resource provider. To build a decentralized information system there is the need for:
 
* Central information system instance that can pull the information from the resource information provider
 
* Distributed information system to be queried by the central one (one instance for every cloud provider)
 
 
 
 
 
A possible strategy is to base everything on the Top-BDII, which is the currently available technology. Ideally one LDAP server is sufficient for the resource providers. The Top-BDII can be configured to get the data from different LDAP servers, and merge them.
 
 
 
Pros:
 
* No need to develope transport and updating mechanisms
 
* Resource providers need only to produce one LDIF file and load it into an LDAP server.
 
Cons:
 
* Top-BDII showed bad performances in publishing dynamic data (not an issue for testing as long as there are few resource providers)
 
* We must publish 100% GLUE2 compliant information
 
 
 
'''Find [[Fedclouds BDII instructions|here]] some guidelines for the ldap installation.'''
 

Latest revision as of 13:09, 8 June 2015