Alert.png The wiki is deprecated and due to be decommissioned by the end of September 2022.
The content is being migrated to other supports, new updates will be ignored and lost.
If needed you can get in touch with EGI SDIS team using operations @ egi.eu.

Difference between revisions of "Resource Pool"

From EGIWiki
Jump to navigation Jump to search
 
(31 intermediate revisions by 2 users not shown)
Line 1: Line 1:
{{Template:Op menubar}} {{TOC_right}}  
{{Template:Op menubar}} {{TOC_right}}  
[[Category:Operations]]
[[Category:ResourceAllocation]]
= Introduction =
In June 2013 the EGI Council approved the implementation of a EGI Resource Allocation Pool to support resource needs of user communities whose request is scientifically peer reviewed. Resource requests from user communities will be reviewed by a Scientific Review Committee (see the [https://documents.egi.eu/document/1472 Terms of Reference]).


= Terminology =
This page provides information to partners interested in contributing resources to the EGI federated pool about resource allocation in EGI.


Through this page you can submit your resource offer to contribute to the federated pool.


'''Pool '''- broad concept of defining/declaring resources availability for EGI Resource Allocation.  
= How can I contribute to the federated EGI resource pool? =
== CALL 1. 2013 QR3 ==
The call for contributions of resource to the EGI federated resource pool CALL1.2013.QR3 is now open to Resource Providers (NGIs and Resource Centres).


'''Pool of resources''' - capacity of resources available for EGI allocation according to defined process under defined conditions. Pool definition may be subject to change by EGI or RP according to a specific policy
<big>
'''Click [https://www.surveymonkey.com/s/2GPHMTK here]''' to submit your resource offer to contribute to the EGI Resource Pool (see the [https://documents.egi.eu/document/1943 pdf] version of the on-line offer form) .


'''DEADLINE for submission: 30 September 2013'''
</big>


= [[Resource_Allocation_Terminology|Terminology]] =


== Resource allocation QoS levels  ==
= Computing: QoS levels  =
For computing resources a Resource Provider can offer different types of access to resources according to its local policies and the user requirements:
* [[Resource_Allocation_Terminology#Level_C1:_Opportunistic | Level C1: Opportunistic]]. Resources are not guaranteed and are subject to local availability.
* [[Resource_Allocation_Terminology#Level_C2:_Time_allocation| Level C2: Time allocation]]. Resources are available in fair share-like mode for a fixed time period.
* [[Resource_Allocation_Terminology#Level_C3:_Reserved_allocation | Level C3: Reserved allocation]]. Resources are exclusively reserved to the VO and the job will be executed immediately after submission.


Source: https://rt.egi.eu/rt/Ticket/Attachment/219099/80552/QoS_RA-TF_v6_AC.docx
= Storage: QoS levels =
For storage resources a Resource Provider can offer different types of access to resources according to its local policies and the user requirements:
* [[Resource_Allocation_Terminology#Level_S1:_Opportunistic | Level S1. Opportunistic]]. Storage space is provided depending on availability of local resources at a time without any guarantees in terms of quota and duration of the allocation.
* [[Resource_Allocation_Terminology#Level_S2:_Guaranteed | Level S2. Guaranteed]]. Storage space is reserved.


=== Opportunistic (basic support level provided by a Pool)  ===
= Resource allocation policies =
For the resources offered to the EGI Pool, the Resource Provider can play different roles in the resource brokerage process according to its local requirements.


*Resources not guaranteed and subject to availability: CPU slots are granted when no other job is submitted by a VO with a higher priority.  
* '''Free hands''': the broker, responsible of matching demand and offer, is free to allocate the resources from one RP Pool according to local criteria which aim to optimize usage of available resources and user demand. The Resource Provider delegates the responsibility of accepting a proposed resource allocation to the Broker.  
**The Pool supports the VO for a fixed period of time. The duration of the agreement has to be declared in the VO Request submitted to the Operator and is subject to negotiation
* '''Right to revoke''': the broker matches demand and offer and defines a resource allocation proposal. The RP Pool Manager is responsible of accepting or rejecting the resource allocation proposal of the Broker (EGI.eu), in case of no reply to a Broker's proposal after a default time, the resource allocation proposal is considered to be accepted.
**Before expiration, the agreement can be renewed and renegotiated directly with the RPP or NGIP
* '''Negotiated''': the broker matches demand and offer and defines a resource allocation proposal. The RP Pool Manager is responsible of accepting or rejecting the resource allocation proposal of the Broker (EGI.eu) and to '''explicitly''' accept or reject.  
**3 different queue levels are proposed
***short: 30 minutes max - for NAGIOS probes or short test
***medium&nbsp;: 12 hours max
***long&nbsp;: 48 hours max
*The opportunistic model is commonly the default model for RPs that wish to support a VO without any formal agreement. Formalizing this model can help VOs to be more aware of who supports them, and may foster resource acknowledgement.


=== Time allocation  ===
The table below illustrates the role of the Broker and of the Resource Provider in the resource allocation process according to the resource allocation policy of choice (one of the options above).  
 
*Resources available in fair share-like mode for a fixed time period: during the agreed fixed period, the VO is allocated a guaranteed minimum number of CPU hours per day/week/month/..., the dayly/weekly/monthly/... period is hereafter called Fine Time Scale (FTS). When the VO reaches this threshold, the allocation mode is converted in Opportunistic until the next FTS slot starts.
**The Pool supports the VO for a fixed period of time. The duration of the agreement has to be declared in the VO Request submitted to the Operator and is subject to negotiation
**After expiration a new Request has to be submitted to the Operator
**Fine Time Scale (FTS) may be declared in the VO Request submitted to the Operator, or may be proposed by the Pool operator, and is subject to negotiation by the Pool
**3 different queue levels are proposed
***short: 30 minutes max - for NAGIOS probes or short test
***medium&nbsp;: 12 hours max
***long&nbsp;: 48 hours max
 
=== Reserved allocation ===
 
*Resources are exclusively reserved to the VOand the job will be executed immediately after submission, unless all the available job slots are already occupied by the VO. The resources made available to the VO are expressed in cores, or job slots.. .
**The duration of the agreement has to be declared in the VO Request together with the amount of resources. Subject to negotiation
**available at any time during a fixed period of time
**Since the resources are fully dedicated to the VO, at least 2 different queue levels are proposed
***short: 30 minutes max - for NAGIOS probes or short test
***production: negotiated with the RPP or NGIP
**The model may be appropriate for well organized large communities, that are able to make 100% use of the dedicated cores during the agreed period.
 
<br>
 
== Resource allocation scenarios ==
 
Source: https://wiki.egi.eu/wiki/Resource_Allocation_Process


{| class="wikitable"
{| class="wikitable"
Line 64: Line 49:
! scope="col" | Actions/steps<br>
! scope="col" | Actions/steps<br>
|-
|-
| in free hand<br>  
| Free hands<br>  
| address requests below a defined threshold  
| address requests below a defined threshold  
| central collector for requests; set up and coordinate the usage of the pool; investigate all the operational aspects; allocate resources;&nbsp; monitor the effectiveness of the pool resources usage<br>  
| central collector for requests; set up and coordinate the usage of the pool; investigate all the operational aspects; allocate resources;&nbsp; monitor the effectiveness of the pool resources usage<br>  
Line 72: Line 57:
| After the pool has been set up, EGI collects requests, defines the required operational steps to support the requests; RPs configure their resources that are part of the pool accordingly
| After the pool has been set up, EGI collects requests, defines the required operational steps to support the requests; RPs configure their resources that are part of the pool accordingly
|-
|-
| right to revoke<br>  
| Right to revoke<br>  
| address request above the threshold  
| address request above the threshold  
| central collector for requests; set up and coordinate the usage of the pool; investigate all the operational aspects; allocate resources;&nbsp; monitor the effectiveness of the pool resources usage<br>  
| central collector for requests; set up and coordinate the usage of the pool; investigate all the operational aspects; allocate resources;&nbsp; monitor the effectiveness of the pool resources usage<br>  
Line 80: Line 65:
| After the pool has been set up, EGI collects requests, defines the required operational steps to support the requests; RPs configure their resources that are part of the pool accordingly
| After the pool has been set up, EGI collects requests, defines the required operational steps to support the requests; RPs configure their resources that are part of the pool accordingly
|-
|-
| negotiated (elastic pools)<br>
| Negotiated 
| address 'heavy' requests  
| address 'heavy' requests  
| central collector for requests; broker for the RPs resources; define all the technical aspects of the requests; open calls to RPs in order to set up the pool&nbsp; through the NGIs<br>  
| central collector for requests; broker for the RPs resources; define all the technical aspects of the requests<br>  
| broker for the RCs resources; provide support to RCs in order to ease the creation of a pool; monitor the effectiveness of the pool resources usage by the requester<br>  
| broker for the RCs resources; provide support to RCs in order to ease the creation of a pool; monitor the effectiveness of the pool resources usage by the requester<br>  
| not at all; it is up to the different RPs (NGIs and/or RCs) to decide how many resources (if any) they want to provide to that specific request/user community<br>  
| it is up to the different RPs (NGIs and/or RCs) to agree on how many resources (if any) they want to provide to that specific request/user community. A broker's proposal is implemented only if explicitly acknowledged by the Resource Providers<br>  
| the call shall look for resources whose allocation could be guaranteed; in addition opportunistic usage is an added value  
| the call shall look for resources whose allocation could be guaranteed; in addition opportunistic usage is an added value  
| for each request EGI investigates all the operational aspects, providing a technical descritption of the request (and recipes if necessary) and sets up a call to RPs (throug NGIs) to provide resources for the temporary (elastic) pool dedicated to that request. Then coordinates the setting up of the pools and informs the requester about the conditions for accessing and using the resources
| for each request EGI investigates all the operational aspects, providing a technical description of the request (and recipes if necessary). Then the broker coordinates the setting up of the pools and informs the requester about the conditions for accessing and using the resources
|}
|}


<br>
<br>


= Contribute resources to the EGI distributed pool =
= Contact information =
For more information about the EGI Pool please contact: operations (at) egi.eu

Latest revision as of 16:33, 23 October 2014

Main EGI.eu operations services Support Documentation Tools Activities Performance Technology Catch-all Services Resource Allocation Security


In June 2013 the EGI Council approved the implementation of a EGI Resource Allocation Pool to support resource needs of user communities whose request is scientifically peer reviewed. Resource requests from user communities will be reviewed by a Scientific Review Committee (see the Terms of Reference).

This page provides information to partners interested in contributing resources to the EGI federated pool about resource allocation in EGI.

Through this page you can submit your resource offer to contribute to the federated pool.

How can I contribute to the federated EGI resource pool?

CALL 1. 2013 QR3

The call for contributions of resource to the EGI federated resource pool CALL1.2013.QR3 is now open to Resource Providers (NGIs and Resource Centres).

Click here to submit your resource offer to contribute to the EGI Resource Pool (see the pdf version of the on-line offer form) .

DEADLINE for submission: 30 September 2013

Terminology

Computing: QoS levels

For computing resources a Resource Provider can offer different types of access to resources according to its local policies and the user requirements:

Storage: QoS levels

For storage resources a Resource Provider can offer different types of access to resources according to its local policies and the user requirements:

  • Level S1. Opportunistic. Storage space is provided depending on availability of local resources at a time without any guarantees in terms of quota and duration of the allocation.
  • Level S2. Guaranteed. Storage space is reserved.

Resource allocation policies

For the resources offered to the EGI Pool, the Resource Provider can play different roles in the resource brokerage process according to its local requirements.

  • Free hands: the broker, responsible of matching demand and offer, is free to allocate the resources from one RP Pool according to local criteria which aim to optimize usage of available resources and user demand. The Resource Provider delegates the responsibility of accepting a proposed resource allocation to the Broker.
  • Right to revoke: the broker matches demand and offer and defines a resource allocation proposal. The RP Pool Manager is responsible of accepting or rejecting the resource allocation proposal of the Broker (EGI.eu), in case of no reply to a Broker's proposal after a default time, the resource allocation proposal is considered to be accepted.
  • Negotiated: the broker matches demand and offer and defines a resource allocation proposal. The RP Pool Manager is responsible of accepting or rejecting the resource allocation proposal of the Broker (EGI.eu) and to explicitly accept or reject.

The table below illustrates the role of the Broker and of the Resource Provider in the resource allocation process according to the resource allocation policy of choice (one of the options above).

Scenario
Scope
EGI role
NGIs role
Review
Usage (guaranteed allocation/opportunistic)
Actions/steps
Free hands
address requests below a defined threshold central collector for requests; set up and coordinate the usage of the pool; investigate all the operational aspects; allocate resources;  monitor the effectiveness of the pool resources usage
provide resources directly; act as broker towards the Resource Centers
no review as long as the pool's resources are enough to satisfy all requests. If a review is needed it is up to EGI.eu only to decide
guaranteed allocation for a subset of the pool's resources; opportunistic for the rest After the pool has been set up, EGI collects requests, defines the required operational steps to support the requests; RPs configure their resources that are part of the pool accordingly
Right to revoke
address request above the threshold central collector for requests; set up and coordinate the usage of the pool; investigate all the operational aspects; allocate resources;  monitor the effectiveness of the pool resources usage
provide resources directly; act as broker towards the Resource Centers + option to withdraw if national/local policies are conflicting
same as above, but  if a review is needed a very lightweight procedure should be defined, involving EGI.eu and representatives of the RPs contributing to the pool
guaranteed allocation for a subset of the pool's resources; opportunistic for the rest After the pool has been set up, EGI collects requests, defines the required operational steps to support the requests; RPs configure their resources that are part of the pool accordingly
Negotiated address 'heavy' requests central collector for requests; broker for the RPs resources; define all the technical aspects of the requests
broker for the RCs resources; provide support to RCs in order to ease the creation of a pool; monitor the effectiveness of the pool resources usage by the requester
it is up to the different RPs (NGIs and/or RCs) to agree on how many resources (if any) they want to provide to that specific request/user community. A broker's proposal is implemented only if explicitly acknowledged by the Resource Providers
the call shall look for resources whose allocation could be guaranteed; in addition opportunistic usage is an added value for each request EGI investigates all the operational aspects, providing a technical description of the request (and recipes if necessary). Then the broker coordinates the setting up of the pools and informs the requester about the conditions for accessing and using the resources


Contact information

For more information about the EGI Pool please contact: operations (at) egi.eu