Difference between revisions of "Federated Cloud Accounting"

From EGIWiki
Jump to: navigation, search
m (OpenNebula Accounting Scripts)
 
Line 1: Line 1:
{{Fedcloud_Menu}}
+
#REDIRECT [[Federated_Cloud_Technology#Accounting]]
 
 
{{FedCloud_TF_Menu}} {{TOC_right}}
 
 
 
[[Category:Federated_Cloud]]
 
 
 
= Scope =
 
 
 
Accounting across Resource Providers
 
 
 
= Members =
 
 
 
{| class="wikitable"
 
!Role
 
!Institution
 
!Name
 
|-
 
|Scenario leader
 
|STFC
 
|Stuart Pullinger
 
|-
 
|Collaborator
 
|STFC
 
|John Gordon
 
|-
 
|Collaborator
 
| BSC
 
| Daniele Lezzi
 
|-
 
|Collaborator
 
| CESNET
 
| Boris Parak
 
|}
 
 
 
 
 
= Roadmap =
 
 
 
To account for resource usage across multiple resource providers we need to define:
 
 
 
# the elements to be accounted for and how accounting data may be gathered. (This relates to the statement  “Resource providers agree and follow the same rules for accounting the resource usage”.) '''DONE''' See [[Federated_Cloud_Accounting#Cloud_Accounting_Usage_Record|Cloud Accounting Usage Record]].
 
# how accounting data will be published '''DONE''' [[Federated_Cloud_Accounting#Publishing_Cloud_Accounting_Records|Publishing Cloud Accounting Records]].
 
# how accounting data will be viewed '''DONE''' There is a cloud view in the [http://accounting-devel.egi.eu/cloud.php? EGI Accounting Portal].
 
# what is required from this data (for VM users, Resource Providers, VO Managers, others?) '''DONE''' Requirements implemented in portal views.
 
 
 
 
 
== Meetings to discuss Cloud Accounting Data ==
 
 
 
A team from the Fedcloud task met on 25th July and 1st August 2013 to discuss the data received so far from the Resource Providers and where improvements should be made.  The outcome of these meetings is here: [https://oerc.basecamphq.com/projects/7732005/file/164221704/CloudAccountingMeetings-JulAug2013.pdf]
 
 
 
There have been many other meetings and discussions at FedCloud weekly meetings, F2F, and EGI Conferences.
 
 
 
= Documentation =
 
 
 
== Cloud Accounting Usage Record ==
 
The current version of the Cloud Accounting Usage Record should be used with the latest version of SSM, further information is here: [https://oerc.basecamphq.com/projects/7732005/file/149644970/New-CloudAccountingUR.pdf Production Cloud Accounting Usage Record]
 
 
 
==== Agreed new version ====
 
Version 0.4 of the Cloud Accounting Usage Record was agreed at the FedCloud Face to Face in Amsterdam in January 2015. The definition with comments found [[Media:Cloud_Accounting_Usage_Record_Schema_v0.4-final.pdf|here]] (retrieved from  [https://oerc.basecamphq.com/projects/7732005/file/199381047/Cloud%20Accounting%20Usage%20Record%20Schema%20v0.4-final.pdf here]).
 
 
 
==== Previous versions ====
 
The previous version of the Cloud Accounting Usage Record is here: [https://oerc.basecamphq.com/projects/7732005/file/125408455/Current%20Version-CloudAccountingUR.pdf Previous Cloud Accounting Usage Record v.1]
 
 
 
This draft schema attempts to map the Cloud Accounting Usage Record fields to fields used by Venus-C and OpenNebula.
 
[https://oerc.basecamphq.com/projects/7732005/file/115587067/Draft-CloudAccountingUR-v3.pdf Version 3 - Cloud Accounting Usage Record]
 
 
 
==== Compute Accounting Record ====
 
Many fields in the cloud usage record were aligned with those in [https://twiki.cern.ch/twiki/pub/EMI/ComputeAccounting/CAR-EMI-tech-doc-0.8.doc Compute Accounting Record].
 
 
 
== Publishing Cloud Accounting Records ==
 
 
 
==== Cloud Accounting System ====
 
The software and technology choices for the existing APEL system (which receives grid compute accounting records) have been used to provide a ‘testbed’ so that the resource providers can send records to a Cloud Accounting System at STFC.  The software is written in Python and the data is stored in a MySQL database, the component parts of the system are:
 
 
 
# SSM
 
# Record loader
 
# MySQL Database
 
 
 
More detail available in the Cloud Accounting System document here:
 
[https://oerc.basecamphq.com/projects/7732005/file/111685266/The%20Cloud%20Accounting%20System.pdf Cloud Accounting System]
 
 
 
=== Publishing Records  ===
 
 
 
To publish records to the central Cloud Accounting Database resource providers will need to install and configure SSM 2.0 and also create messages containing records which are sent using SSM. The information below has links to scripts for creating these messages for OpenNebula and Openstack sites. There is also information on the installation of SSM 2.0 below.
 
 
 
===== SSM Package for publishing records  =====
 
 
 
The new version of SSM (2.0) should be retrieved from here [http://apel.github.io/]  
 
 
 
There are versions for SL5 and SL6.
 
 
 
The EGI test message broker network will be switched off in April 2014. The FedCloud will move to using a test queue on the production brokers from 1st April 2014. Sites will need to change 3 values in their SSM configuration file (/etc/apel/sender.cfg) on 1st April:
 
 
 
[broker]
 
...
 
<strike>network: TEST-NWOB</strike>
 
network: PROD
 
...
 
<strike>use_ssl: false</strike>
 
use_ssl: true
 
...
 
[messaging]
 
...
 
<strike>destination: /queue/global.accounting.cloudtest.CENTRAL</strike>
 
destination: /queue/global.accounting.test.cloud.central
 
...
 
 
 
'''1st April Update: Sites are reporting problems using 'use_ssl: true' They receive a Java error message like this:'''
 
 
 
ssm.ssm2 - WARNING - Error message received: java.lang.SecurityException: User name [null] or password is invalid. No user for client certificate: CN=...
 
 
 
'''This reflects a problem with the configuration of the message broker network. Please use 'use_ssl: false' in your configuration until further notice.'''
 
 
 
A full System Administrator's guide is here [https://twiki.cern.ch/twiki/pub/EMI/EMI3APELClient/APEL_SSM_System_Administrator_Guide.pdf]. A more detailed guide is here [https://twiki.cern.ch/twiki/pub/EMI/EMI3APELClient/APEL_SSM_User_Guide.pdf]
 
 
 
===== Terminology:  =====
 
 
 
*A message is one file which is sent and received by the SSM. Usually a message will contain a number of records (eg 1000)
 
*A record corresponds to one row in the database. It contains a number of key-value pairs as specified in the table below
 
*The header in each message tells the server which type of records are in that message. You need one header per message, so one header per file.
 
 
 
==== Cloud Accounting Message Format for use with SSM 2.0  ====
 
 
 
'''Header''' APEL-cloud-message: v0.2
 
 
 
{| width="650" border="1" cellspacing="1" cellpadding="1"
 
|-
 
! scope="col" | Key
 
! scope="col" | Value
 
! scope="col" | Description
 
! scope="col" | Mandatory
 
|-
 
| VMUUID
 
| string
 
| Virtual Machine's Universally Unique IDentifier
 
| Yes
 
|-
 
| SiteName
 
| string
 
| Sitename, e.g. GOCDB Sitename
 
| Yes
 
|-
 
| MachineName
 
| string
 
| VM Id
 
|
 
|-
 
| LocalUserId
 
| string
 
| Local username
 
|
 
|-
 
| LocalGroupId
 
| string
 
| Local groupname
 
|
 
|-
 
| GlobalUserName
 
| string
 
| User's X509 DN
 
|
 
|-
 
| FQAN
 
| string
 
| User's VOMS attributes
 
|
 
|-
 
| Status
 
| string
 
| Completion status - started, completed, suspended
 
|
 
|-
 
| StartTime
 
| int
 
| Must be set if Status = Started (epoch time)
 
|
 
|-
 
| EndTime
 
| int
 
| Must be set if Status = completed (epoch time)
 
|
 
|-
 
| SuspendDuration
 
| int
 
| Set when Status = suspended (seconds)
 
|
 
|-
 
| WallDuration
 
| int
 
| Wallclock - actual time used (seconds)
 
|
 
|-
 
| CpuDuration
 
| int
 
| CPU time consumed (seconds)
 
|
 
|-
 
| CpuCount
 
| int
 
| Number of CPUs allocated
 
|
 
|-
 
| NetworkType
 
| string
 
| Description
 
|
 
|-
 
| NetworkInbound
 
| int
 
| GB received
 
|
 
|-
 
| NetworkOutbound
 
| int
 
| GB sent
 
|
 
|-
 
| Memory
 
| int
 
| Memory allocated to the VM (MB)
 
|
 
|-
 
| Disk
 
| int
 
| Disk allocated to the VM (GB)
 
|
 
|-
 
| StorageRecordId
 
| string
 
| Link to associated storage record
 
|
 
|-
 
| ImageId
 
| string
 
| Image ID
 
|
 
|-
 
| CloudType
 
| string
 
| e.g. OpenNebula, Openstack
 
|
 
|}
 
 
 
'''End of record:'''&nbsp;%%
 
 
 
==== Example Message  ====
 
<pre>APEL-cloud-message: v0.2
 
VMUUID: https://cloud.cesga.es:3202/compute/47f74797-e9c9-46d7-b28d-5f87209239eb 2013-02-25 17:37:27+00:00
 
SiteName: CESGA
 
MachineName: one-2421
 
LocalUserId: 19
 
LocalGroupId: 101
 
GlobalUserName: NULL
 
FQAN: NULL
 
Status: completed
 
StartTime: 1361813847
 
EndTime: 1361813870
 
SuspendDuration: NULL
 
WallDuration: NULL
 
CpuDuration: NULL
 
CpuCount: 1
 
NetworkType: NULL
 
NetworkInbound: 0
 
NetworkOutbound: 0
 
Memory: 1000
 
Disk: NULL
 
StorageRecordId: NULL
 
ImageId: NULL
 
CloudType: OpenNebula
 
%%
 
...another cloud record...
 
%%
 
...
 
%%
 
</pre>
 
 
 
=== OpenNebula Accounting Scripts ===
 
Records may be retrieved from OpenNebula databases using the oneacct-export tool from [https://github.com/EGI-FCTF/oneacct_export] which creates a file in the message format which can be sent using the SSM 2.0.
 
 
 
Current stable version: 0.4.5-1
 
  https://appdb.egi.eu/store/software/oneacct.export/releases/0.4.x
 
 
 
Installation instructions:
 
  https://appdb.egi.eu/store/software/oneacct.export/releases/0.4.x
 
 
 
=== OpenStack Accounting Scripts ===
 
 
 
[https://github.com/IFCA/caso cASO] is a pluggable extractor of Cloud Accounting Usage Records from an OpenStack installation. cASO gets usage information from nova or ceilometer APIs and can generate valid output for Apel SSM or logstash.
 
 
 
* Current version (0.2) is available at PyPi: (https://pypi.python.org/pypi/caso/), you can install it with <code>pip install caso</code>
 
* Check the [http://caso.readthedocs.org/en/latest/ cASO documentation] includes how to install and configure OpenStack for generating the accounting records.
 
* Source code available at [https://github.com/IFCA/caso cASO github repo]
 
* Packages for Ubuntu distributions are build at [https://build.opensuse.org/project/show/home:aloga:cloud:integration OpenSUSE build service home:aloga:cloud:integration project]
 
 
 
== Resource Providers published to Cloud Accounting Database ==
 
=== Using SSM 2.0 ===
 
* Current list of Sites: http://goc-accounting.grid-support.ac.uk/cloudtest/cloudsites2.html
 
* VMs during last hour: http://goc-accounting.grid-support.ac.uk/cloudtest/vmshour2.html
 
* VMs during last day (long list may make your browser choke): http://goc-accounting.grid-support.ac.uk/cloudtest/vms2.html
 
 
 
== Feedback from Resource Providers ==
 
{| border="1"
 
!VM
 
!What is measured
 
|-
 
|Oxford/Eucalyptus (no longer running)
 
|Number of VM instantiated - Type of VM - Amount of time each VM has been running - Number of Elastic Block Storage volumes created - Amount of time the EBS volumes have been owned - Times the EBS volumes have been mounted - Binding between EBS volumes and VMs - Number of OS images loaded in the S3 repository - Amount of time the OS images have been loaded into the S3 repository
 
|-
 
|OpenNebula
 
| (Have reviewed UR)
 
|}
 
 
 
= References =
 
<references/>
 

Latest revision as of 14:41, 26 June 2017