Difference between revisions of "Fedcloud-tf:WorkGroups:Scenario4"

From EGIWiki
Jump to: navigation, search
m (SSM Package for publishing records)
Line 96: Line 96:
'''1st April Update: Sites are reporting problems using 'use_ssl: true' They receive a Java error message like this:'''
'''20th April 2015 Update: The problems using 'use_ssl: true' have been fixed. Please use 'use_ssl: true' in your configuration as shown above.'''
  ssm.ssm2 - WARNING - Error message received: java.lang.SecurityException: User name [null] or password is invalid. No user for client certificate: CN=...
'''This reflects a problem with the configuration of the message broker network. Please use 'use_ssl: false' in your configuration until further notice.'''
A full System Administrator's guide is here [https://twiki.cern.ch/twiki/pub/EMI/EMI3APELClient/APEL_SSM_System_Administrator_Guide.pdf]. A more detailed guide is here [https://twiki.cern.ch/twiki/pub/EMI/EMI3APELClient/APEL_SSM_User_Guide.pdf]
A full System Administrator's guide is here [https://twiki.cern.ch/twiki/pub/EMI/EMI3APELClient/APEL_SSM_System_Administrator_Guide.pdf]. A more detailed guide is here [https://twiki.cern.ch/twiki/pub/EMI/EMI3APELClient/APEL_SSM_User_Guide.pdf]

Revision as of 17:42, 20 April 2015

Main Roadmap and Innovation Technology For Users For Resource Providers Media

Workbenches: Open issues
Scenario 1
VM Management
Scenario 2
Data Management
Scenario 3
Information Systems
Scenario 4
Scenario 5
Scenario 6
Scenario 7
Federated AAI
Scenario 8
VM Image Management
Scenario 9
Scenario 10
Scenario 11

Scenario 4: Accounting across Resource Providers

Leader: Stuart Pullinger, STFC

Scenario collaborators

Role Institution Name
Scenario leader STFC Stuart Pullinger
Collaborator STFC John Gordon
Collaborator BSC Daniele Lezzi
Collaborator CESNET Boris Parak


To account for resource usage across multiple resource providers we need to define:

  1. the elements to be accounted for and how accounting data may be gathered. (This relates to the statement “Resource providers agree and follow the same rules for accounting the resource usage”.) DONE See Cloud Usage Record.
  2. how accounting data will be published DONE Publishing Cloud Accounting Records.
  3. how accounting data will be viewed DONE There is a cloud view in the EGI Accounting Portal.
  4. what is required from this data (for VM users, Resource Providers, VO Managers, others?) DONE Requirements implemented in portal views.

Meetings to discuss Cloud Accounting Data

A team from the Fedcloud task met on 25th July and 1st August 2013 to discuss the data received so far from the Resource Providers and where improvements should be made. The outcome of these meetings is here: [1]

There have been many other meetings and discussions at FedCloud weekly meetings, F2F, and EGI Conferences.

Cloud Accounting Usage Record

The current version of the Cloud Accounting Usage Record should be used with the latest version of SSM, further information is here: Production Cloud Accounting Usage Record

Agreed new version

Version 0.4 of the Cloud Accounting Usage Record was agreed at the FedCloud Face to Face in Amsterdam in January 2015. The definition with comments is here.

Previous versions

The previous version of the Cloud Accounting Usage Record is here: Previous Cloud Accounting Usage Record v.1

This draft schema attempts to map the Cloud Accounting Usage Record fields to fields used by Venus-C and OpenNebula. Version 3 - Cloud Accounting Usage Record

Compute Accounting Record

Many fields in the cloud usage record were aligned with those in Compute Accounting Record.

Publishing Cloud Accounting Records

Cloud Accounting System

The software and technology choices for the existing APEL system (which receives grid compute accounting records) have been used to provide a ‘testbed’ so that the resource providers can send records to a Cloud Accounting System at STFC. The software is written in Python and the data is stored in a MySQL database, the component parts of the system are:

  1. SSM
  2. Record loader
  3. MySQL Database

More detail available in the Cloud Accounting System document here: Cloud Accounting System

Publishing Records

To publish records to the central Cloud Accounting Database resource providers will need to install and configure SSM 2.0 and also create messages containing records which are sent using SSM. The information below has links to scripts for creating these messages for OpenNebula and Openstack sites. There is also information on the installation of SSM 2.0 below.

SSM Package for publishing records

The new version of SSM (2.0) should be retrieved from here [2]

There are versions for SL5 and SL6.

The EGI test message broker network will be switched off in April 2014. The FedCloud will move to using a test queue on the production brokers from 1st April 2014. Sites will need to change 3 values in their SSM configuration file (/etc/apel/sender.cfg) on 1st April:

network: TEST-NWOB
network: PROD
use_ssl: false
use_ssl: true
destination: /queue/global.accounting.cloudtest.CENTRAL
destination: /queue/global.accounting.test.cloud.central

20th April 2015 Update: The problems using 'use_ssl: true' have been fixed. Please use 'use_ssl: true' in your configuration as shown above.

A full System Administrator's guide is here [3]. A more detailed guide is here [4]

  • A message is one file which is sent and received by the SSM. Usually a message will contain a number of records (eg 1000)
  • A record corresponds to one row in the database. It contains a number of key-value pairs as specified in the table below
  • The header in each message tells the server which type of records are in that message. You need one header per message, so one header per file.

Cloud Accounting Message Format for use with SSM 2.0

Header APEL-cloud-message: v0.2

Key Value Description Mandatory
VMUUID string Virtual Machine's Universally Unique IDentifier Yes
SiteName string Sitename, e.g. GOCDB Sitename Yes
MachineName string VM Id
LocalUserId string Local username
LocalGroupId string Local groupname
GlobalUserName string User's X509 DN
FQAN string User's VOMS attributes
Status string Completion status - started, completed, suspended
StartTime int Must be set if Status = Started (epoch time)
EndTime int Must be set if Status = completed (epoch time)
SuspendDuration int Set when Status = suspended (seconds)
WallDuration int Wallclock - actual time used (seconds)
CpuDuration int CPU time consumed (seconds)
CpuCount int Number of CPUs allocated
NetworkType string Description
NetworkInbound int GB received
NetworkOutbound int GB sent
Memory int Memory allocated to the VM (MB)
Disk int Disk allocated to the VM (GB)
StorageRecordId string Link to associated storage record
ImageId string Image ID
CloudType string e.g. OpenNebula, Openstack

End of record: %%

Example Message

APEL-cloud-message: v0.2
VMUUID: https://cloud.cesga.es:3202/compute/47f74797-e9c9-46d7-b28d-5f87209239eb 2013-02-25 17:37:27+00:00
SiteName: CESGA
MachineName: one-2421
LocalUserId: 19
LocalGroupId: 101
GlobalUserName: NULL
Status: completed
StartTime: 1361813847
EndTime: 1361813870
SuspendDuration: NULL
WallDuration: NULL
CpuDuration: NULL
CpuCount: 1
NetworkType: NULL
NetworkInbound: 0
NetworkOutbound: 0
Memory: 1000
Disk: NULL
StorageRecordId: NULL
ImageId: NULL
CloudType: OpenNebula
...another cloud record...

OpenNebula Accounting Scripts

Records may be retrieved from OpenNebula databases using the oneacct-export tool from [5] which creates a file in the message format which can be sent using the SSM 2.0.

Current stable version: 0.2.5-1


Installation instructions:


OpenStack Accounting Scripts


  1. Install the software
  2. Check out the doc (man osssmrc)
  3. Edit /etc/osssmrc
  4. activate the osssm service
Openstack VM statuses
Openstack status Mapped SSM status Description
active started VM is booted and running
build started VM is in building phase, not yet available
deleted completed the VM was just deleted (transient state)
error error the VM is in error state, the VM may still consume resources
hard_reboot started the VM is in a reset phase
migrating started the VM is moving from one host to another
paused paused the VM process was stopped (as with a kill -STOP). It does not consume any CPU more, but still RAM and disk
reboot started the VM is proceeding an ACPI power stop/start
rebuild started performing a shutdown, a re-image, and a reboot
confirming_resize started the VM has been resized and nova waits confirmation from the user to leave it in latest state
rescue started the VM has been put into **rescue** mode (VM disk attached as second drive to a newly booted image)
resize started the VM is being resized (changing its flavor)
revert_resize started the user did not confirm the resize and the VM is falling back to the previous state
password started xxx
verify_resize started openstack is performing checks on the resized VM
shutoff completed the VM is powered off: no more CPU/RAM consumed, still disk
suspended suspended the VM has performed a suspend to disk, it consumes no more CPU/RAM, still disk (plus memory image)
terminated completed xxx
stopped completed the VM has performed a proper shutdown on user request: no more CPU/RAM consumed, still disk
saving started the VM is being snapshotted to create a new image


cASO is a pluggable extractor of Cloud Accounting Usage Records from an OpenStack installation. cASO gets usage information from nova or ceilometer APIs and can generate valid output for Apel SSM or logstash.

Resource Providers published to Cloud Accounting Database

Using SSM 2.0

Feedback from Resource Providers

VM What is measured
Oxford/Eucalyptus (no longer running) Number of VM instantiated - Type of VM - Amount of time each VM has been running - Number of Elastic Block Storage volumes created - Amount of time the EBS volumes have been owned - Times the EBS volumes have been mounted - Binding between EBS volumes and VMs - Number of OS images loaded in the S3 repository - Amount of time the OS images have been loaded into the S3 repository
OpenNebula (Have reviewed UR)