Difference between revisions of "Fedcloud-tf:WorkGroups:Scenario4"

From EGIWiki
Jump to: navigation, search
(Cloud Accounting Draft Usage Record)
(Cloud Accounting Draft Usage Record)
Line 33: Line 33:
 
== Cloud Accounting Draft Usage Record ==
 
== Cloud Accounting Draft Usage Record ==
 
The current version of the Cloud Accounting Usage Record is here:
 
The current version of the Cloud Accounting Usage Record is here:
 +
 +
==== Previous versions ====
 +
This draft schema attempts to map the Cloud Accounting Usage Record fields to fields used by Venus-C and OpenNebula.
 +
[https://oerc.basecamphq.com/projects/7732005/file/115130463/Current%20Version-CloudAccountingUR.pdf Version 3 - Cloud Accounting Usage Record]
 +
 +
[https://oerc.basecamphq.com/projects/7732005/file/115133635/Draft%20-%20CloudAccountingUR-v2.pdf Version 2 - Cloud Accounting Usage Record]
 +
 +
[https://oerc.basecamphq.com/projects/7732005/file/105129409/Draft-CloudAccountingUsageRecord-v1.pdf Version 1 - Cloud Accounting Usage Record]).
 +
 +
==== Compute Accounting Record ====
 +
To help define what we will account for the existing [https://twiki.cern.ch/twiki/pub/EMI/ComputeAccounting/CAR-EMI-tech-doc-0.8.doc Compute Accounting Record] has been used.
 +
 +
== Publishing Cloud Accounting Records ==
  
 
Records may be retrieved from OpenNebula databases using this script [https://oerc.basecamphq.com/projects/7732005/file/115574771/cloudacc.txt] which creates a file in the message format which can be sent using the SSM.
 
Records may be retrieved from OpenNebula databases using this script [https://oerc.basecamphq.com/projects/7732005/file/115574771/cloudacc.txt] which creates a file in the message format which can be sent using the SSM.
Line 139: Line 152:
 
%%
 
%%
 
</pre>
 
</pre>
==== Compute Accounting Record ====
 
To help define what we will account for the existing [https://twiki.cern.ch/twiki/pub/EMI/ComputeAccounting/CAR-EMI-tech-doc-0.8.doc Compute Accounting Record] could be extended. 
 
 
==== Previous versions: ====
 
This draft schema attempts to map the Cloud Accounting Usage Record fields to fields used by Venus-C and OpenNebula.
 
[https://oerc.basecamphq.com/projects/7732005/file/115130463/Current%20Version-CloudAccountingUR.pdf Version 3 - Cloud Accounting Usage Record]
 
 
[https://oerc.basecamphq.com/projects/7732005/file/115133635/Draft%20-%20CloudAccountingUR-v2.pdf Version 2 - Cloud Accounting Usage Record]
 
 
[https://oerc.basecamphq.com/projects/7732005/file/105129409/Draft-CloudAccountingUsageRecord-v1.pdf Version 1 - Cloud Accounting Usage Record]).
 
  
 
== Publishing Accounting Data ==
 
== Publishing Accounting Data ==

Revision as of 16:49, 15 March 2012

Main Roadmap and Innovation Technology For Users For Resource Providers Media


Workbenches: Open issues
Scenario 1
VM Management
Scenario 2
Data Management
Scenario 3
Information Systems
Scenario 4
Accounting
Scenario 5
Monitoring
Scenario 6
Notification
Scenario 7
Federated AAI
Scenario 8
VM Image Management
Scenario 9
Brokering
Scenario 10
Contextualisation
Scenario 11
Security



Scenario 4: Accounting across Resource Providers

Leader: Alison Packer and John Gordon, STFC

Scenario collaborators

Role Institution Name
Scenario leader STFC Alison Packer
Collaborator OeRC Matteo Turilli
Collaborator BSC Daniele Lezzi

Plan

To account for resource usage across multiple resource providers we need to define:

  1. the elements to be accounted for and how accounting data may be gathered. (This relates to the statement “Resource providers agree and follow the same rules for accounting the resource usage”.)
  2. how accounting data will be published
  3. what is required from this data (for VM users, Resource Providers, VO Managers, others?)

Cloud Accounting Draft Usage Record

The current version of the Cloud Accounting Usage Record is here:

Previous versions

This draft schema attempts to map the Cloud Accounting Usage Record fields to fields used by Venus-C and OpenNebula. Version 3 - Cloud Accounting Usage Record

Version 2 - Cloud Accounting Usage Record

Version 1 - Cloud Accounting Usage Record).

Compute Accounting Record

To help define what we will account for the existing Compute Accounting Record has been used.

Publishing Cloud Accounting Records

Records may be retrieved from OpenNebula databases using this script [1] which creates a file in the message format which can be sent using the SSM.

The SSM client package can be retrieved from here: [2] A short description of how to install and setup SSM is here: [3]

Terminology:

  • A message is one file which is sent and received by the SSM. Usually a message will contain a number of records (eg 1000)
  • A record corresponds to one row in the database. It contains a number of key-value pairs as specified by the tables below
  • The header in each message tells the server which type of records are in that message. You need one header per message, so one header per file.
  • Epoch time is used - number of seconds since 1st January 1970.

Cloud Accounting Message Format:

Header APEL-cloud-message: v0.1

The header only appears once at the top of each message (that is once at the top of each file). It defines the type of record and the schema version.

Key Value Description Mandatory
RecordId string Current time + Site + MachineName Yes
Site string Sitename, e.g. GOCDB Sitename Yes
ZoneName string Location, e.g. EU, AM, AP
MachineName string VM Id Yes
LocalUserId string Local username
GlobalUserName string User's X509 DN
FQAN string User's VOMS attributes
Status string Completion status - started, completed, suspended
StartTime int Must be set if Status = Started (epoch time)
EndTime int Must be set if Status = completed (epoch time)
SuspendTime int Set when Status = completed (epoch time)
TimeZone string Time Zone
WallDuration int Wallclock - actual time used (seconds)
CpuDuration int CPU time consumed (seconds)
CpuCount int Number of CPUs allocates
NetworkType string
NetworkInbound int GB received
NetworkOutbound int GB sent
Memory int Memory allocated to the VM (MB)
Disk int Size allocated to the VM (GB)
StorageRecordId string Link to associated storage record
ImageId string Image ID
CloudType string e.g. OpenNebula

End of record: %%

Example Message

APEL-cloud-message: v0.1
RecordId: 2012-03-14 16:34:45+00:00 STFC-RAL vm-0
SiteName: STFC-RAL
ZoneName: EU
MachineName: vm-0
LocalUserId: 0
LocalGroupId: 0
GlobalUserName: NULL
FQAN: NULL
Status: completed
StartTime: 1331131577
EndTime: 1331131929
SuspendTime: NULL
TimeZone: UTC
WallDuration: NULL
CpuDuration: NULL
CpuCount: NULL
NetworkType: NULL
NetworkInbound: NULL
NetworkOutbound: NULL
Memory: 1024
Disk: NULL
StorageRecordId: NULL
ImageId: NULL
CloudType: OpenNebula
%%
...another cloud record...
%%
...
%%

Publishing Accounting Data

Cloud Accounting System

The software and technology choices for the new APEL system (which receives compute accounting records) will be used to provide a ‘testbed’ so that the resource providers can send records to a Cloud Accounting System at STFC. The software is written in Python and the data is stored in a MySQL database, the component parts of the system are:

  1. SSM
  2. Record loader
  3. MySQL Database

More detail available in the Cloud Accounting System document here: Cloud Accounting System

Work on providing OpenNebula Accounting Records

Contact made with "VENUS" project developers who have developed a system for extracting accounting data. Detail on the Cloud Accounting System (above) provided to them, requested more detail on their implementation.

Feedback from Resource Providers

VM What is measured
Oxford/Eucalyptus (no longer running) Number of VM instantiated - Type of VM - Amount of time each VM has been running - Number of Elastic Block Storage volumes created - Amount of time the EBS volumes have been owned - Times the EBS volumes have been mounted - Binding between EBS volumes and VMs - Number of OS images loaded in the S3 repository - Amount of time the OS images have been loaded into the S3 repository
OpenNebula (Have reviewed UR)