Difference between revisions of "Fedcloud-tf:WorkGroups:Scenario4"

From EGIWiki
Jump to: navigation, search
(Draft Usage Record)
(Draft Usage Record)
Line 33: Line 33:
 
== Draft Usage Record ==
 
== Draft Usage Record ==
 
To help define what we will account for the existing [https://twiki.cern.ch/twiki/pub/EMI/ComputeAccounting/CAR-EMI-tech-doc-0.8.doc Compute Accounting Record] could be extended.   
 
To help define what we will account for the existing [https://twiki.cern.ch/twiki/pub/EMI/ComputeAccounting/CAR-EMI-tech-doc-0.8.doc Compute Accounting Record] could be extended.   
The following are a set of assumptions I have made related to this draft Usage Record:
 
  
# For each instantiated VM there is one record, this has implications for charging as there is one ‘Charge’ attribute per record it does not fit with a charging model where each of the elements of CPU, Memory, I/O etc. are charged for separately.
+
The following is an updated version of the draft Usage Record as a result of the feedback from our F2F meeting on 24th November, 2011:
 +
 
 +
# Many of the properties defined in the Usage Record may be set to NULL, this is based on the assumption that different Resource Providers may want to publish different sets of data, some may want to charge, some not BUT there should only by one Usage Record, usable by all.
 +
# We concluded thay including a "Charge" property in the Usage Record was unneccessary as this is a Cloud Accounting record it should reflect detail about the VM and its usage so that it may be used by Resource Providers to charge for their service, but the charging mechanism would be Resource Provider specific and shoudl therefore be separated from accounting.
 
# The fields for GlobalUserName, VirtualOrganization, Projectname and Group relate to an existing method of authorizing grid users, if this authorization mechanism were to be used for cloud usage as well then these would provide a method of associating VM usage with a user and then querying usage in a portal could be based on user, VO etc.  
 
# The fields for GlobalUserName, VirtualOrganization, Projectname and Group relate to an existing method of authorizing grid users, if this authorization mechanism were to be used for cloud usage as well then these would provide a method of associating VM usage with a user and then querying usage in a portal could be based on user, VO etc.  
 
# A VM might be in one of three states: started, stopped and suspended.  I think we would expect to gather many VM records whilst a VM is running, so I propose using the existing UR “Status” property for this – it may be possible to extend the implementation of this property to include some VM-specific statuses if these existing ones are not deemed appropriate.
 
# A VM might be in one of three states: started, stopped and suspended.  I think we would expect to gather many VM records whilst a VM is running, so I propose using the existing UR “Status” property for this – it may be possible to extend the implementation of this property to include some VM-specific statuses if these existing ones are not deemed appropriate.
Line 44: Line 46:
  
 
{| border="1"
 
{| border="1"
!Current UR Property
+
!Cloud Usage Record Property
!Current Definition
+
!Definition
!Extended for Cloud Accounting
 
 
|-
 
|-
 
|RecordIdentity
 
|RecordIdentity
 
|Unique identifier of a record (String)
 
|Unique identifier of a record (String)
|(keep the same)
+
|-
 +
|SiteName
 +
|GOCDB SiteName - extend service types in GOCDB to include cloud resource providers
 +
|-
 +
|MachineName
 +
|VM Hostname (String)
 +
|-
 +
|LocalUserID
 +
|Local user name
 +
|-
 +
|LocalGroupID
 +
|Local group name
 
|-
 
|-
 
|GlobalUserName
 
|GlobalUserName
 
|Global identity of user (certificate DN) (String)
 
|Global identity of user (certificate DN) (String)
|(keep the same)
 
 
|-
 
|-
 
|VirtualOrganization
 
|VirtualOrganization
 
|All three may be retrieved from FQAN - use if VOs part of authorization mechanism
 
|All three may be retrieved from FQAN - use if VOs part of authorization mechanism
|(keep the same)
 
 
|-
 
|-
 
|ProjectName
 
|ProjectName
|
 
 
|
 
|
 
|-
 
|-
 
|Group
 
|Group
 
|
 
|
|
 
 
|-
 
|-
|Charge
+
|Status
|Total charge of job (Float)
+
|Completion status (String) completed, started or suspended
 +
|-
 +
|StartTime
 +
|Must be set if Status = started (Timestamp)
 +
|-
 +
|EndTime
 +
|Set to NULL until Status = completed (Timestamp)
 +
|-
 +
|MeasurementTime 
 +
|Set to amount of time used so far (Timestamp)
 +
|-
 +
|SuspendTime
 +
|Set when Status = suspended (Timestamp)
 +
|-
 +
|TimeZone
 
|
 
|
|-
 
|Status
 
|Completion status (String), aborted, completed, failed, held, queued, started, suspended.     
 
|For VM instance - completed, started and suspended could be used
 
 
|-
 
|-
 
|WallDuration
 
|WallDuration
|WallClock time   
+
|WallClock time  Actual time used
|Is this EndTime in the case of VM?
 
 
|-
 
|-
 
|CPUDuration
 
|CPUDuration
 
|CPU time consumed (Duration)
 
|CPU time consumed (Duration)
|(keep the same)
 
 
|-
 
|-
|EndTime
+
|CPUCount
|Completion time (Timestamp) 
+
|Number of CPUs
|Set to NULL until Status = completed
+
|-
 +
|CPUPower
 +
|Power of CPUs
 
|-
 
|-
|StartTime
+
|NetworkType
|Start time of the job (Timestamp) 
+
|
|Must be set if Status = started
 
 
|-
 
|-
|MachineName
+
|NetworkInbound
|Hostname of the LRMS (String)   
+
|Bandwidth
|VM Hostname
 
 
|-
 
|-
|Network
+
|NetworkOutbound
|Amount of network resource used by the job (+ integer)
+
|Bandwidth
|Network resource used by the VM
 
 
|-
 
|-
 
|Memory
 
|Memory
|Amount of physical memory used by the job (+ integer)
+
|Memory allocated to the VM
|Memory associated with the VM
 
|-
 
|TimeDuration
 
|Additional measure of time duration (Duration) 
 
|This could be used for suspended VM to record suspended-endtime
 
|-
 
|SiteName
 
|GOCDB SiteName     
 
|Extend service types in GOCDB to include cloud resource providers
 
|-
 
!Proposed New Property
 
!Definition
 
!
 
 
|-
 
|-
 
|ImageID
 
|ImageID
 
|Every image has a unique ID associated with it
 
|Every image has a unique ID associated with it
|
 
 
|-
 
|-
 
|CloudType
 
|CloudType
 
|Cloud type
 
|Cloud type
|
 
 
|-
 
|-
 
|StorageRecordIdentity
 
|StorageRecordIdentity
 
|Link to associated storage record
 
|Link to associated storage record
|
 
 
|}
 
|}
  

Revision as of 19:14, 29 November 2011

Main Roadmap and Innovation Technology For Users For Resource Providers Media


Workbenches: Open issues
Scenario 1
VM Management
Scenario 2
Data Management
Scenario 3
Information Systems
Scenario 4
Accounting
Scenario 5
Monitoring
Scenario 6
Notification
Scenario 7
Federated AAI
Scenario 8
VM Image Management
Scenario 9
Brokering
Scenario 10
Contextualisation
Scenario 11
Security



Scenario 4: Accounting across Resource Providers

Leader: Alison Packer and John Gordon, STFC

Scenario collaborators

Role Institution Name
Scenario leader STFC Alison Packer
Collaborator OeRC Matteo Turilli
Collaborator

Plan

To account for resource usage across multiple resource providers we need to define:

  1. the elements to be accounted for and how accounting data may be gathered. (This relates to the statement “Resource providers agree and follow the same rules for accounting the resource usage”.)
  2. how accounting data will be published
  3. what is required from this data (for VM users, Resource Providers, VO Managers, others?)

Draft Usage Record

To help define what we will account for the existing Compute Accounting Record could be extended.

The following is an updated version of the draft Usage Record as a result of the feedback from our F2F meeting on 24th November, 2011:

  1. Many of the properties defined in the Usage Record may be set to NULL, this is based on the assumption that different Resource Providers may want to publish different sets of data, some may want to charge, some not BUT there should only by one Usage Record, usable by all.
  2. We concluded thay including a "Charge" property in the Usage Record was unneccessary as this is a Cloud Accounting record it should reflect detail about the VM and its usage so that it may be used by Resource Providers to charge for their service, but the charging mechanism would be Resource Provider specific and shoudl therefore be separated from accounting.
  3. The fields for GlobalUserName, VirtualOrganization, Projectname and Group relate to an existing method of authorizing grid users, if this authorization mechanism were to be used for cloud usage as well then these would provide a method of associating VM usage with a user and then querying usage in a portal could be based on user, VO etc.
  4. A VM might be in one of three states: started, stopped and suspended. I think we would expect to gather many VM records whilst a VM is running, so I propose using the existing UR “Status” property for this – it may be possible to extend the implementation of this property to include some VM-specific statuses if these existing ones are not deemed appropriate.
  5. We need a method of identifying where a specific running VM is – there are many ways to do this, I suggest that the RecordIdentity might contain a concatenation of some of the other fields in the record – e.g. TimeStamp SiteName MachineName ImageID - other suggestions welcome.
  6. We have job accounting and storage accounting systems and their related usage records already - we therefore account for jobs and storage using these existing records/methods.

The following table shows what might be used and some additional elements required for cloud accounting:

Cloud Usage Record Property Definition
RecordIdentity Unique identifier of a record (String)
SiteName GOCDB SiteName - extend service types in GOCDB to include cloud resource providers
MachineName VM Hostname (String)
LocalUserID Local user name
LocalGroupID Local group name
GlobalUserName Global identity of user (certificate DN) (String)
VirtualOrganization All three may be retrieved from FQAN - use if VOs part of authorization mechanism
ProjectName
Group
Status Completion status (String) completed, started or suspended
StartTime Must be set if Status = started (Timestamp)
EndTime Set to NULL until Status = completed (Timestamp)
MeasurementTime Set to amount of time used so far (Timestamp)
SuspendTime Set when Status = suspended (Timestamp)
TimeZone
WallDuration WallClock time Actual time used
CPUDuration CPU time consumed (Duration)
CPUCount Number of CPUs
CPUPower Power of CPUs
NetworkType
NetworkInbound Bandwidth
NetworkOutbound Bandwidth
Memory Memory allocated to the VM
ImageID Every image has a unique ID associated with it
CloudType Cloud type
StorageRecordIdentity Link to associated storage record

Feedback from Resource Providers

VM What is measured
Oxford/Eucalyptus (no longer running) Number of VM instantiated - Type of VM - Amount of time each VM has been running - Number of Elastic Block Storage volumes created - Amount of time the EBS volumes have been owned - Times the EBS volumes have been mounted - Binding between EBS volumes and VMs - Number of OS images loaded in the S3 repository - Amount of time the OS images have been loaded into the S3 repository
OpenNebula (Have reviewed UR)