Alert.png The wiki is deprecated and due to be decommissioned by the end of September 2022.
The content is being migrated to other supports, new updates will be ignored and lost.
If needed you can get in touch with EGI SDIS team using operations @ egi.eu.

Difference between revisions of "Fedcloud-tf:WorkGroups:Scenario4"

From EGIWiki
Jump to navigation Jump to search
Line 167: Line 167:
Records may be retrieved from OpenNebula databases using the cloudacc script from [https://github.com/EGI-FCTF/opennebula-cloudacc EGI-FCTF/opennebula-cloudacc] which creates a file in the message format which can be sent using the SSM 2.0.
Records may be retrieved from OpenNebula databases using the cloudacc script from [https://github.com/EGI-FCTF/opennebula-cloudacc EGI-FCTF/opennebula-cloudacc] which creates a file in the message format which can be sent using the SSM 2.0.


Current stable version: 3.8.1-2
Current stable version: 3.8.3
   https://github.com/EGI-FCTF/opennebula-cloudacc/archive/3.8.1-2.tar.gz
   https://github.com/EGI-FCTF/opennebula-cloudacc/archive/3.8.3.tar.gz
   https://github.com/EGI-FCTF/opennebula-cloudacc/archive/3.8.1-2.zip
   https://github.com/EGI-FCTF/opennebula-cloudacc/archive/3.8.3.zip


===== Openstack resource providers =====
===== Openstack resource providers =====

Revision as of 18:48, 19 March 2013

Main Roadmap and Innovation Technology For Users For Resource Providers Media


Workbenches: Open issues
Scenario 1
VM Management
Scenario 2
Data Management
Scenario 3
Information Systems
Scenario 4
Accounting
Scenario 5
Monitoring
Scenario 6
Notification
Scenario 7
Federated AAI
Scenario 8
VM Image Management
Scenario 9
Brokering
Scenario 10
Contextualisation
Scenario 11
Security



Scenario 4: Accounting across Resource Providers

Leader: Alison Packer and John Gordon, STFC

Scenario collaborators

Role Institution Name
Scenario leader STFC Alison Packer
Collaborator OeRC Matteo Turilli
Collaborator BSC Daniele Lezzi

Plan

To account for resource usage across multiple resource providers we need to define:

  1. the elements to be accounted for and how accounting data may be gathered. (This relates to the statement “Resource providers agree and follow the same rules for accounting the resource usage”.)
  2. how accounting data will be published
  3. what is required from this data (for VM users, Resource Providers, VO Managers, others?)

Cloud Accounting Usage Record

A new version of the Cloud Accounting Usage Record should be used with a new version of SSM (Version 2.0), further information is here: New Cloud Accounting Usage Record

Previous versions

The previous version of the Cloud Accounting Usage Record is here: Previous Cloud Accounting Usage Record v.1

This draft schema attempts to map the Cloud Accounting Usage Record fields to fields used by Venus-C and OpenNebula. Version 3 - Cloud Accounting Usage Record

Compute Accounting Record

To help define what we will account for the existing Compute Accounting Record has been used.

Publishing Cloud Accounting Records

Cloud Accounting System

The software and technology choices for the new APEL system (which receives compute accounting records) will be used to provide a ‘testbed’ so that the resource providers can send records to a Cloud Accounting System at STFC. The software is written in Python and the data is stored in a MySQL database, the component parts of the system are:

  1. SSM
  2. Record loader
  3. MySQL Database

More detail available in the Cloud Accounting System document here: Cloud Accounting System

Publishing Records

To publish records to the central Cloud Accounting Database resource providers will need to install and configure SSM 2.0 and also create messages containing records which are sent using SSM. The information below has links to scripts for creating these messages for OpenNebula and Openstack sites. There is also information on the installation of SSM 2.0 below.

SSM Package for publishing records

The new version of SSM (2.0) should be retrieved from here [1]

There are versions for SL5 and SL6.

There are a number of changes from the previous version of SSM, including where configuration files and messages containing records are located. We now have different destination queues; for cloud accounting using SSM 2.0 the destination queue to be set in /etc/apel/dns is:

destination: /queue/global.accounting.cloudtest.CENTRAL
Terminology:
  • A message is one file which is sent and received by the SSM. Usually a message will contain a number of records (eg 1000)
  • A record corresponds to one row in the database. It contains a number of key-value pairs as specified in the table below
  • The header in each message tells the server which type of records are in that message. You need one header per message, so one header per file.

Cloud Accounting Message Format for use with SSM 2.0

Header APEL-cloud-message: v0.2

Key Value Description Mandatory
VMUUID string Virtual Machine's Universally Unique IDentifier Yes
SiteName string Sitename, e.g. GOCDB Sitename Yes
MachineName string VM Id
LocalUserId string Local username
LocalGroupId string Local groupname
GlobalUserName string User's X509 DN
FQAN string User's VOMS attributes
Status string Completion status - started, completed, suspended
StartTime int Must be set if Status = Started (epoch time)
EndTime int Must be set if Status = completed (epoch time)
SuspendDuration int Set when Status = suspended (seconds)
WallDuration int Wallclock - actual time used (seconds)
CpuDuration int CPU time consumed (seconds)
CpuCount int Number of CPUs allocated
NetworkType string Description
NetworkInbound int GB received
NetworkOutbound int GB sent
Memory int Memory allocated to the VM (MB)
Disk int Disk allocated to the VM (GB)
StorageRecordId string Link to associated storage record
ImageId string Image ID
CloudType string e.g. OpenNebula, Openstack

End of record: %%

Example Message

APEL-cloud-message: v0.2
VMUUID: https://cloud.cesga.es:3202/compute/47f74797-e9c9-46d7-b28d-5f87209239eb 2013-02-25 17:37:27+00:00
SiteName: CESGA
MachineName: one-2421
LocalUserId: 19
LocalGroupId: 101
GlobalUserName: NULL
FQAN: NULL
Status: completed
StartTime: 1361813847
EndTime: 1361813870
SuspendDuration: NULL
WallDuration: NULL
CpuDuration: NULL
CpuCount: 1
NetworkType: NULL
NetworkInbound: 0
NetworkOutbound: 0
Memory: 1000
Disk: NULL
StorageRecordId: NULL
ImageId: NULL
CloudType: OpenNebula
%%
...another cloud record...
%%
...
%%
OpenNebula resource providers

Records may be retrieved from OpenNebula databases using the cloudacc script from EGI-FCTF/opennebula-cloudacc which creates a file in the message format which can be sent using the SSM 2.0.

Current stable version: 3.8.3

 https://github.com/EGI-FCTF/opennebula-cloudacc/archive/3.8.3.tar.gz
 https://github.com/EGI-FCTF/opennebula-cloudacc/archive/3.8.3.zip
Openstack resource providers
  1. Install the software
  2. Check out the doc (man osssmrc)
  3. Edit /etc/osssmrc
  4. activate the osssm service

Resources:

Resource Providers published to Cloud Accounting Database

Using SSM 2.0

Older Version - using SSM 1.2

Feedback from Resource Providers

VM What is measured
Oxford/Eucalyptus (no longer running) Number of VM instantiated - Type of VM - Amount of time each VM has been running - Number of Elastic Block Storage volumes created - Amount of time the EBS volumes have been owned - Times the EBS volumes have been mounted - Binding between EBS volumes and VMs - Number of OS images loaded in the S3 repository - Amount of time the OS images have been loaded into the S3 repository
OpenNebula (Have reviewed UR)