VT EGI Pay-for-Use Processes/Instructions

EGI Activity groups

Special Interest groups

Policy groups

Virtual teams

Distributed Competence Centres

EGI Virtual teams:

Main •

Active Projects •

Closed Projects •

Guidelines

Pay-For-Use menu:

Home •

Processes/Instructions •

Service/Price Overview •

Service Management •

Legal/Policy •

Actions •

Meetings •

Further details per step to be provided upon completion of Final Report (19 Dec 2014)

Business Process Overview

Publish Pricing Information

Overview

To test the mechanism for turning your accounting into a monetary charge charging rate needs to be defined. During the pilot the numbers are not legally binding but obviously a ballpark figure would be good as it would give more realistic pilot reports.

To define the charging rate the plan is to use a new feature of GOCDB which is known as 'A Key-Value Pair Property Bag Extensiblity Mechanism'. This allows the definition of arbitrary key-value pairs for a site or any service. If a community agrees a naming scheme for the keys and semantics for the values then they have a flexible and low overhead method of adding more information about anything defined in GOCDB. For anyone interested about more details this is documented https://wiki.egi.eu/wiki/GOCDB/Release4/Development/ExtensibilityMechanism

Instructions

If you do not have access rights to change features of your site please ask the site manager or a site admin to do it for you.

1. Go to GOCDB https://goc.egi.eu;
2. Navigate to your site. (My Sites) at the top of the left hand sidebar.
3. Just above the list of Services you should see a new box called 'Site Extension Properties'
4. At the bottom of this box there is a big green plus titled 'Add Properties' – click on it
5. This should show a window with two fields 'Property Name' and 'Property Value'
6. For 'Property Name' insert the 'Key Name' according to your desired 'Charge Type' (see table below) e.g. P4U_Pilot_Grid_CPU
7. For 'Property Value' insert a real number which will be interpreted as shown in the Price Calculation column in the table below. e.g. Euros per HEPSPEC06 Hour.
8. Click the button 'Add Site Property'
9. If you go back to the site view you will see the 'Key Name' listed. You can edit or delete it.
10. Then add other properties from the table below as appropriate for your site.

Once you have done this we will query GOCDB to pull the values for the pilot sites and the accounting portal will join them with usage data to produce a charging report.

Charge Types / Properties

Charge Type	Key Name	Pricing
Grid CPU	P4U_Pilot_Grid_CPU	Euros/HEPSPEC06 Hour
Cloud CPU	P4U_Pilot_Cloud_Wall	Euros/Hour
Storage	P4U_Pilot_Storage_Use	Euros/GB*month
Cloud Storage	P4U_Pilot_Storage_Use_Cloud	Euros/GB*month
VAT	P4U_Pilot_VAT	Optional VAT rate to be applied to above

How to handle pay-for-use process for providers (e-GRANT)

The goal of the Resource Allocation (RA) process is to reach an agreement between Research Communities and Resource Provider concerning the parameters and conditions of use of resources.

Benefits for Research Communities:

• in a multi-provider EGI environment you have a single point of contact to ask and negotiate resources.

Benefits for Resource Providers:

• you will get direct communication with Research Communities and clarity on expectations
• it will allow more effectively plan the use of resources

The basic parameters of a RA contact are: 

• duration
• amount of computing and /or storage resources.

The Resource Provider offer is expressed in a form of Resource Pools which are declared by Providers using common <a href="https://wiki.egi.eu/wiki/Resource_Allocation_Metrics_Description" title="Resource Allocation Metrics Description">metrics</a>.

The Research Communities use the same metrics while describing their needs (resource request).

 

   <ac:structured-macro ac:macro-id="ca8d927c-c6ea-4986-bfa4-df0d951ba6db" ac:name="toc" ac:schema-version="1"/>
  

Glossary

For basic terms used in Resource Allocation process, please go <a href="https://wiki.egi.eu/wiki/Resource_Allocation_Terminology" title="Resource Allocation Terminology">here</a>

Roles

• <a href="https://wiki.egi.eu/wiki/Resource_Allocation_Terminology#Customer" title="Resource Allocation Terminology">Customer</a> - scientist involved in a science project requesting storage or computing resources in EGI, person signing SLA with Resource Provider
• <a href="https://wiki.egi.eu/wiki/Resource_Allocation_Terminology#Broker" title="Resource Allocation Terminology">Broker</a> - matchmaker and coordinator of RA process
• <a href="https://wiki.egi.eu/wiki/Resource_Allocation_Terminology#Resource_Provider" title="Resource Allocation Terminology">Resource Provider</a> -Site Manager or NGI Manager responsible for resource allocation on sites in their scope

Provider activities

Log in

Click on Login as provider (GOCDB).

==

   <ac:image>
    <ri:attachment ri:filename="P1.png"/>
  </ac:image>
  

   ==

Dashboard

After login to e-Grant you are directed to the dashboard. Here you can administrate yours Pools and handle OLAs.

<ac:image ac:height="250"> <ri:attachment ri:filename="P2.png"/> </ac:image>

arranges all OLAs concerning your site(s)

• DOCUMENTS REQUIRING ACTION - contains SLAs awaiting for your action (negotiation, approval, rejection)
• SLA IN NEGOTIATION - contains SLAs pinned to your pools, but that are on other side (Customer, Broker)
• OUTBOX - contains SLA documents being negotiated and awaiting for Broker's action (negotiation, approval, rejection)
• AGREED SLAs - documents containing allocations approved by you (Provider), but waiting for Broker verification?
• BINDING SLAs - documents with allocation being in place. OLAs change their status automatically from 'Agreed' to 'Binding' after Customers approval of RA Request.
• ACTIVE POOLS - list of active pools
• INACTIVE POOLS - list of inactive pools
• POOLS DRAFTS - list of saved pool drafts
• STARRED - contains OLAs requiring special attention - starred by Provider.

Pools

==== Pool creation

  ====

1. Click “Create new pool” button.

<ac:image> <ri:attachment ri:filename="P4.png"/> </ac:image>

2. Edit Pool properties

<ac:image> <ri:attachment ri:filename="P5.png"/> </ac:image>

Basic info

<ac:image ac:height="250"> <ri:attachment ri:filename="P6.png"/> </ac:image>

• Start date - date when pool becomes available to Customers
• End date - date wen pool will be removed
• Site name - name of the site in which resources will be offered
• Quality of Service - Resource Provider can offer different types of access to resources according to its local policies and the user requirements:
  
  • Level C1: Opportunistic: Resources are not guaranteed and are subject to local availability.
  • Level C2: Time allocation: Resources are available in fair share-like mode for a fixed time period.
  • Level C3: Reserved allocation: Resources are exclusively reserved to the VO and the job will be executed immediately after submission.
• Description of Pool's local policies - Any relevant information about local polices regarding this Pool

 

When ready, click on "SAVE POOL" or "SAVE DRAFT FOR POOL" button.

 

3. Edit pool details (all relevant metrics should be filled in)

According to resources offered by a Site, all metrics describing the resources must be specified.
4 different resource types can be offered (with regards to P4U activity):

• HTC Computing
• HTC Storage
• Cloud Computing
• Cloud Storage

To add a resource type choose "Add resources and metrics" button and list of all resource types will drop-out.

<ac:image ac:height="250"> <ri:attachment ri:filename="P7.png"/> </ac:image>

 

==== General metrics

  ====

Start Date and End Date - when pool will be available for Customers

HTC Computing metric

<tbody> </tbody>

Metric name <a class="sortheader" href="https://wiki.egi.eu/wiki/Resource_Allocation_for_Provider"> <ac:image> <ri:url ri:value=""/> </ac:image> </a>	Description <a class="sortheader" href="https://wiki.egi.eu/wiki/Resource_Allocation_for_Provider"> <ac:image> <ri:url ri:value=""/> </ac:image> </a>
Max job duration [hours]	Maximum execution time for a single job. After this time, the job can be interrupted by site administrator. Example: For grant with Max Job Duration metric equal to 72h job using four slots will be canceled after 72 real time hours (the number of slots used by the task does NOT affect the task time) The user can expect (require) that a task with a duration of less than defined in this metric will not be interrupted.
Min local storage [GB]	The minimum scratch space used in the calculations for each slot involved in job execution. The User can expect that the amount of scratch space specified in this metric will be the minimum available for each slot allocated for the job. The usage of more scratch space than specified may not be possible.
Min physical memory per core [GB]	The minimum RAM that can take a user application (job) on a single slot. The user can expect that if a job submitted uses no more RAM per slot than declared in the metric, it will be executed.
Opportunistic computing time [HEPSPEC-hours]	Computing time offered on opportunistic basis. Time specified in this metric is a subset of time specified in main metric Total computing time.
Supported middlewares	Middlewares supported in Site
Total computing time [HEPSPEC-hours]	Total computing time offered. This metric is obligatory for computing. By default time offered in this metric is guaranteed. In case metric Opportunistic computing time is specified, the amount of guaranteed time allocated is lowered by the amount of opportunistic time.

HTC Storage metrics

<tbody> </tbody>

Metric name <a class="sortheader" href="https://wiki.egi.eu/wiki/Resource_Allocation_for_Provider"> <ac:image> <ri:url ri:value=""/> </ac:image> </a>	Description <a class="sortheader" href="https://wiki.egi.eu/wiki/Resource_Allocation_for_Provider"> <ac:image> <ri:url ri:value=""/> </ac:image> </a>
Opportunistic storage capacity [GB]	The target limit of capacity intended to long-term storage of data, on the basis of guarantees.
Total storage capacity [GB]	By default capacity allocated in this metric is guaranteed. In case metric Opportunistic storage capacity is specified, the amount of guaranteed storage capacity is lowered by the amount of opportunistic capacity.

Cloud Computing metrics

<tbody> </tbody>

Metric name <a class="sortheader" href="https://wiki.egi.eu/wiki/Resource_Allocation_for_Provider"> <ac:image> <ri:url ri:value=""/> </ac:image> </a>	Description <a class="sortheader" href="https://wiki.egi.eu/wiki/Resource_Allocation_for_Provider"> <ac:image> <ri:url ri:value=""/> </ac:image> </a>
Total number of virtual cores [VC]	Total amount of computing capacity available for allocation measured in <i>virtual cores</i>. EGI virtual core is equal to EC2 virtual core (<a class="external free" href="http://aws.amazon.com/ec2/" rel="nofollow">http://aws.amazon.com/ec2/</a>).
Virtual Machines (maximum) [VM]	Maximum number of each type of resource template that may be hosted within your service.  This may take into account the physical characteristics of your facilities.
Small Virtual Machines	- Number of virtual cores < 2 - RAM [GB] < 2 - Scratch/ephemeral storage [GB] < 20
Medium Virtual Machines	- Number of virtual cores  < 4 (AND) - RAM [GB] < 4 (AND) - Scratch/ephemeral storage [GB] < 40
Large Virtual Machines	- Number of virtual cores < 8 (AND) - RAM [GB] < 32 (AND) - Scratch/ephemeral storage [GB] < 80
Extra Large Virtual Machines	- Number of virtual cores >= 8 (OR) - RAM [GB] >= 32 (OR) - Scratch/ephemeral storage [GB] >= 80
Allowed level of oversubscription (0% - 50%)	What is the state level of over provisioning that you are utilising in CPU and possibly memory.

Cloud Storage metrics

<tbody> </tbody>

Metric name <a class="sortheader" href="https://wiki.egi.eu/wiki/Resource_Allocation_for_Provider"> <ac:image> <ri:url ri:value=""/> </ac:image> </a>	Description <a class="sortheader" href="https://wiki.egi.eu/wiki/Resource_Allocation_for_Provider"> <ac:image> <ri:url ri:value=""/> </ac:image> </a>
Capacity [GB]	Total available storage volume made available to the user community, this should also note any over provisioning that is made of storage capability
High availability levels [yes/no]	Facilities or capabilities to provide high availability quality of service to user communities
Connectivity resilience [yes/no]	Does the underlying facility have multiple network routes of entry (another form but more specific of high availability).

Editing Pools

At any time Provider can change the Negotiation scenario (default being Free hand to Broker).

• Negotiation scenerio - defines the role of EGI Broker EGI Resource Provider in RA Process for a given resource Pool
  
  • Free hand to broker: the broker, responsible of matching demand and offer, is free to allocate the resources from one RP Pool according to local criteria which aim to optimize usage of available resources and user demand. The Resource Provider delegates the responsibility of accepting a proposed resource allocation to the Broker.
  • Right to revoke: the broker matches demand and offer and defines a resource allocation proposal. The RP Pool Manager is responsible of accepting or rejecting the resource allocation proposal of the Broker (EGI.eu), in case of no reply to a Broker's proposal after a default time, the resource allocation proposal is considered to be accepted.
  • Negotiation: the broker matches demand and offer and defines a resource allocation proposal. The RP Pool Manager is responsible of accepting or rejecting the resource allocation proposal of the Broker (EGI.eu) and to explicitly accept or reject.

Provider may also want to change metric values of the pool. In this case:

1. Click "Edit" button.

<ac:image> <ri:attachment ri:filename="P5.png"/> </ac:image>

2. Change chosen metric values.

<ac:image> <ri:attachment ri:filename="P8.png"/> </ac:image>

   3. Save

==

   OLA (re)negotiation and confirmation/rejection   ==

 

Provider can negotiate metrics in OLA. (SLA on the pictures)

 <ac:image>
 <ri:attachment ri:filename="Prov12.png"/>
</ac:image>

Provider can choose one of the following actions:

• Reject SLA - finishes negotiation with no resources allocated for Customer's SLA (rejection)
• Negotiate SLA PROPOSAL - allows Provider to make a counter offer. Provider has 7 days from recieving OLA Proposal to conduct this action.
• ALLOCATE RESOURCES - finishes negotiation with resource allocation proposed by Broker (confirmation).
  
  

<ac:image> <ri:attachment ri:filename="Prov13.png"/> </ac:image>

 

In case of NEGOTIATE SLA PROPOSAL Provider can change metric values ans click Save.

<ac:image> <ri:attachment ri:filename="Prov14.png"/> </ac:image>

In case that Provider accepts proposal, after clicking ALLOCATE RESOURCES, proposal is sent to Broker for final verification.

Capacity management

Under construction.

Help and Support

To avoid problems with using e-Grant remember about using HINTS:

Information marks. There is a lot of information marks next to the metrics inbox. When you point with mouse cursor on the mark the hint will show you like on example below:

<ac:image> <ri:attachment ri:filename="p10.png"/> </ac:image>

Contact with us. The staff at EGI.eu will support the applicant at any point of the process, from request creation to accessing the assigned resources. Please send your questions to: <a href="mailto:resource-allocation-support@mailman.egi.eu">resource-allocation-support@mailman.egi.eu</a>