Alert.png The wiki is deprecated and due to be decommissioned by the end of September 2022.
The content is being migrated to other supports, new updates will be ignored and lost.
If needed you can get in touch with EGI SDIS team using operations @ egi.eu.

Difference between revisions of "Resource Allocation Terminology"

From EGIWiki
Jump to navigation Jump to search
 
(30 intermediate revisions by 2 users not shown)
Line 1: Line 1:
{{Template:Op menubar}}
{{Template:Op menubar}}
{{EGI_RA_menubar}}
{{TOC_right}}
{{TOC_right}}
[[Category:Operations]]
<!--
'''Pool '''- broad concept of defining/declaring resources availability for EGI Resource Allocation.


'''Pool of resources''' - capacity of resources available for EGI allocation according to defined process under defined conditions. Pool definition may be subject to change by EGI or RP according to a specific policy
[[Resource Pool|Back]] to the Resource Pool main page
-->


= Resource Provider =
= Pool =
The Resource Provider (RP) offers access to ICT resources through service abstractions (e.g., computing power, storage, etc.). NGIs and or individual Resource Centres can be Resource Providers to EGI.
The broad concept of defining/declaring resources availability for EGI Resource Allocation.  


= Resource Provider Pool =
= Pool of resources =
The Resource Provider Pool (RP Pool) is defined to be the specific resource capacity available for allocation by one Resource Provier and available to EGI for allocation. The RP pool is a dynamic entity as it can change over time: for example, the capacity offered, the resource technical specifications and the allocation policies can be changed at any point in time by the Resource Provider.
The capacity of resources available for EGI allocation according to defined process under defined conditions. Pool definition may be subject to change by EGI or RP according to a specific policy


= RP Pool Manager =
= Resource Provider  =
The RP Pool Manager is the contact point for a given pool, liaises with the resource broker (EGI.eu) and is responsible of defining the local pool allocation policies.


= EGI Pool =
The Resource Provider (RP) offers [[Glossary#IT_Service|IT Services]] through service abstractions (e.g., computing power, storage, etc.). NGIs and or individual Resource Centres can be Resource Providers to EGI.
The EGI Resource Pool is the federation of one or more RP Pools. EGI federates resources, i.e. in order to satisfy a given demand, resources can be provided by one or multiple federated Resource Providers.  
 
= Resource Provider Pool  =
 
The Resource Provider Pool (RP Pool) is defined to be the specific resource capacity available for allocation by one Resource Provier and available to EGI for allocation. The RP pool is a dynamic entity as it can change over time: for example, the capacity offered, the resource technical specifications and the allocation policies can be changed at any point in time by one Resource Provider.
 
= RP Pool Manager  =
 
The RP Pool Manager is the contact point for a given pool, liaises with the resource broker (EGI.eu) and is responsible of defining the local pool allocation policies. In case of a right to revoke resource allocation model, the Manager is responsible of accepting or refusing a resource allocation proposal submitted by the Broker.
 
= EGI Pool =
 
The EGI Resource Pool is the federation of one or more RP Pools. EGI federates resources, i.e. in order to satisfy a given request, resources can be provided by one or multiple federated Resource Providers.  
 
= Broker  =
 
The broker is the central entity (EGI.eu) responsible of identifying Resource Providers who collectively are able to satisfy a given request. In the proposed process providers are free to choose one of the following models as applicable:
 
*(A) EGI.eu is the Broker acting at a central level, while the NGI is the Broker responsible of appointing Resource Providers at a national level; EGI.eu liaises with the NGI to define the local providers offering resources to meet a given request.
*(B) EGI.eu is the Broker responsible of liaising directly with Resource Centres at a national level.
 
The process is flexible enough to allow the coexistence of (1) and (2). For example EGI.eu can be delegated the role of Broker at a national level for resource allocations of small entity (where small can be defined by the NGI) with the purpose of keeping the matchmaking process easy.
 
= Matchmaker  =
 
Matchmaker is the model that best suites user community requests. EGI.eu will play the Federator as well as the Matchmaker, i.e. is the entity responsible of
 
*processing resource requests,
*negotiating an offer,
*identifying the Resource Providers willing to offer resources that match the request,
*establishing and enforcing an Operational Level Agreement (OLA) with the Resource Providers: the OLA involves the Federator and the Resource Provider.
*establishing and enforcing a Service Level Agreement (SLA) with the Customer: the SLA involves the Customer and the Federator.
*providing monitoring information (collected from the Resource Provider) and periodic service reports.
 
= Customer  =
 
A person signing SLA with Resource Provider, usually representative of a Virtual Organisation.
 
= User  =
 
A person using resources on base of binding SLA (could be Customer itself)
 
<br>
 
= Resource Allocation  =
 
The distribution of resources among competing groups of people or programs. It is used to assign the available resources in an economic way and to satisfy customer's needs.
 
= Service Level Agreement (SLA)  =
 
An agreement between a [https://wiki.egi.eu/wiki/Glossary#Service_Provider Service Provider] and a customer/client. The SLA describes the [https://wiki.egi.eu/wiki/Glossary#Service IT Service], documents service level targets and specifies the responsibilities of the Service Provider and the customer/client.. A single SLA may cover multiple [https://wiki.egi.eu/wiki/Glossary#IT_Service IT Services] or multiple customers/clients.
 
= Operational Level Agreement (OLA)  =
 
An agreement between an [https://wiki.egi.eu/wiki/Glossary#Service_Provider IT Service Provider] (e.g. NGI or Resource Centre) and another part of the same organisation (EGI.eu in its function of Broker). An OLA supports the IT Service Provider’s delivery of [https://wiki.egi.eu/wiki/Glossary#IT_Service IT Services] to [https://wiki.egi.eu/wiki/Glossary#Users Users]. The OLA defines the goods or [https://wiki.egi.eu/wiki/Glossary#Service Services] to be provided and the responsibilities of both parties.<br>
 
Please refer to the [https://wiki.egi.eu/wiki/Glossary EGI Glossary] for the definitions of the terms used.
 
= Computing: Resource allocation QoS levels  =


= Resource allocation QoS levels  =
By providing resources through a RP pool to the EGI Pool, a Resource Provider can decide to offer different quality of service levels.  
By providing resources through a RP pool to the EGI Pool, a Resource Provider can decide to offer different quality of service levels.  


== Opportunistic ==
== Level C1: Opportunistic ==
This is the basic level which can be provided.
 
Resources are not guaranteed and are subject to availability: CPU slots are granted when no other job is submitted by a VO with a higher priority.  
This is the basic level which can be provided. Resources are not guaranteed and are subject to local availability at any time: job slots are granted when no other job is submitted by a VO with a higher priority.  
 
The Pool supports the VO for a fixed period of time. The duration of the agreement has to be declared in the VO Request submitted to the Operator and is subject to negotiation. Before expiration, the agreement can be renewed and renegotiated directly with the Resource Provider. Three different queue levels are proposed, each associated to a maximum waiting time.
 
*'''short''': 30 minutes max - for NAGIOS probes or short test
*'''medium''': 12 hours max
*'''48 hours max'''
 
The opportunistic model is commonly the default model for RPs that wish to support a VO. Formalizing this model can help VOs to be more aware of who supports them, and fosters resource acknowledgement.
 
== Level C2: Time allocation  ==
 
In this intermediate service level, resources are available in fair share-like mode for the duration of the agreement: during this duration, the VO is allocated a guaranteed minimum number of normalized hours per day/week/month/, i.e. a daily/weekly/monthly wall time.
 
The allocated resources can be expressed in '''normalized Wall Clock Time [[HEP SPEC06|HEP-SPEC06]] hours'''.
 
The daily/weekly/monthly/... period is hereafter called Fine Time Scale (FTS). When the VO reaches this threshold, the allocation mode is converted in Opportunistic until the next FTS slot starts.
 
The Pool supports the VO for a fixed period of time. The the duration of the agreement that must be declared in the VO Request submitted to the Operator. The duration is subject to negotiation.
 
Fine Time Scale (FTS) may be declared in the VO Request submitted to the Operator, or may be proposed by the Pool Manager, and is subject to negotiation.
 
Three different queue levels are proposed
 
*'''short''': 30 minutes max - for NAGIOS probes or short tests
*'''medium''': 12 hours max
*'''long''': 48 hours max
 
== Level C3: Reserved allocation  ==
 
Resources are exclusively reserved to the VO and the job will be executed immediately after submission, unless all the available job slots are already occupied by jobs of the VO. The resources made available to the VO are expressed in '''cores''', or '''job slots'''.
 
The duration of the agreement has to be declared in the VO Request together with the amount of resources. Subject to negotiation.
 
In this service, resources are available at any time during a fixed period of time. Since the resources are fully dedicated to the VO, at least 2 different queue levels are proposed:


The Pool supports the VO for a fixed period of time. The duration of the agreement has to be declared in the VO Request submitted to the Operator and is subject to negotiation. Before expiration, the agreement can be renewed and renegotiated directly with the Resource Provider. Three different queue levels are proposed, each associated to a maximum waiting time.
*'''short''': 30 minutes max - for NAGIOS probes or short test  
*'''short''': 30 minutes max - for NAGIOS probes or short test  
*'''medium&nbsp''': 12 hours max
*'''production''': negotiated with the Resource Provider.
*'''48 hours max'''
 
The opportunistic model is commonly the default model for RPs that wish to support a VO. Formalizing this model can help VOs to be more aware of who supports them, and fosters resource acknowledgement.
The model may be appropriate for well organized large communities, that are able to make 100% use of the dedicated cores during the agreed period.  
 
= Storage: Resource allocation QoS levels  =


== Time allocation ==
== Level S1: Opportunistic ==
In this intermediate service level, resources are available in fair share-like mode for a fixed time period: during the agreed fixed period, the VO is allocated a guaranteed minimum number of CPU hours per day/week/month/..., the daily/weekly/monthly/... period is hereafter called Fine Time Scale (FTS). When the VO reaches this threshold, the allocation mode is converted in Opportunistic until the next FTS slot starts.


Storage space is provided depending on availability of local resources at a time without any guarantees in terms of quota and duration of the allocation. Parameters:


**The Pool supports the VO for a fixed period of time. The duration of the agreement has to be declared in the VO Request submitted to the Operator and is subject to negotiation
*overall capacity offered (in GB)
**After expiration a new Request has to be submitted to the Operator
*maximum available per individual request (in GB)
**Fine Time Scale (FTS) may be declared in the VO Request submitted to the Operator, or may be proposed by the Pool operator, and is subject to negotiation by the Pool
**3 different queue levels are proposed
***short: 30 minutes max - for NAGIOS probes or short test
***medium&nbsp;: 12 hours max
***long&nbsp;: 48 hours max


== Reserved allocation ==  
== Level S2: Guaranteed  ==


*Resources are exclusively reserved to the VOand the job will be executed immediately after submission, unless all the available job slots are already occupied by the VO. The resources made available to the VO are expressed in cores, or job slots.. .
Storage space is reserved. Parameters:  
**The duration of the agreement has to be declared in the VO Request together with the amount of resources. Subject to negotiation
**available at any time during a fixed period of time
**Since the resources are fully dedicated to the VO, at least 2 different queue levels are proposed
***short: 30 minutes max - for NAGIOS probes or short test
***production: negotiated with the RPP or NGIP
**The model may be appropriate for well organized large communities, that are able to make 100% use of the dedicated cores during the agreed period.


<br>
*overall capacity offered (in GB)
*maximum available per individual request (in GB)


= Resource allocation scenarios =
= Links  =


Source: https://wiki.egi.eu/wiki/Resource_Allocation_Process
*[https://wiki.egi.eu/w/images/3/33/List_of_possible_3_QoS_levels_v_3.pdf Resource Allocation QoS]
*[https://wiki.egi.eu/wiki/Resource_Allocation_Process Resource Allocation Policies]


{| class="wikitable"
|-
! scope="col" | Scenario<br>
! scope="col" | Scope<br>
! scope="col" | EGI role<br>
! scope="col" | NGIs role<br>
! scope="col" | Review<br>
! scope="col" | Usage (guaranteed allocation/opportunistic)<br>
! scope="col" | Actions/steps<br>
|-
| in free hand<br>
| address requests below a defined threshold
| central collector for requests; set up and coordinate the usage of the pool; investigate all the operational aspects; allocate resources;&nbsp; monitor the effectiveness of the pool resources usage<br>
| provide resources directly; act as broker towards the Resource Centers <br>
| no review as long as the pool's resources are enough to satisfy all requests. If a review is needed it is up to EGI.eu only to decide <br>
| guaranteed allocation for a subset of the pool's resources; opportunistic for the rest
| After the pool has been set up, EGI collects requests, defines the required operational steps to support the requests; RPs configure their resources that are part of the pool accordingly
|-
| right to revoke<br>
| address request above the threshold
| central collector for requests; set up and coordinate the usage of the pool; investigate all the operational aspects; allocate resources;&nbsp; monitor the effectiveness of the pool resources usage<br>
| provide resources directly; act as broker towards the Resource Centers + option to withdraw if national/local policies are conflicting<br>
| same as above, but&nbsp; if a review is needed a very lightweight procedure should be defined, involving EGI.eu and representatives of the RPs contributing to the pool<br>
| guaranteed allocation for a subset of the pool's resources; opportunistic for the rest
| After the pool has been set up, EGI collects requests, defines the required operational steps to support the requests; RPs configure their resources that are part of the pool accordingly
|-
| negotiated (elastic pools)<br>
| address 'heavy' requests
| central collector for requests; broker for the RPs resources; define all the technical aspects of the requests; open calls to RPs in order to set up the pool&nbsp; through the NGIs<br>
| broker for the RCs resources; provide support to RCs in order to ease the creation of a pool; monitor the effectiveness of the pool resources usage by the requester<br>
| not at all; it is up to the different RPs (NGIs and/or RCs) to decide how many resources (if any) they want to provide to that specific request/user community<br>
| the call shall look for resources whose allocation could be guaranteed; in addition opportunistic usage is an added value
| for each request EGI investigates all the operational aspects, providing a technical descritption of the request (and recipes if necessary) and sets up a call to RPs (throug NGIs) to provide resources for the temporary (elastic) pool dedicated to that request. Then coordinates the setting up of the pools and informs the requester about the conditions for accessing and using the resources
|}


= Resources =
<br>[[Category:ResourceAllocation]]
* [Source: https://wiki.egi.eu/w/images/3/33/List_of_possible_3_QoS_levels_v_3.pdf Resource Allocation QoS]

Latest revision as of 16:33, 23 October 2014

Main EGI.eu operations services Support Documentation Tools Activities Performance Technology Catch-all Services Resource Allocation Security


EGI Resource Allocation menu:


Back to the Resource Pool main page

Pool

The broad concept of defining/declaring resources availability for EGI Resource Allocation.

Pool of resources

The capacity of resources available for EGI allocation according to defined process under defined conditions. Pool definition may be subject to change by EGI or RP according to a specific policy

Resource Provider

The Resource Provider (RP) offers IT Services through service abstractions (e.g., computing power, storage, etc.). NGIs and or individual Resource Centres can be Resource Providers to EGI.

Resource Provider Pool

The Resource Provider Pool (RP Pool) is defined to be the specific resource capacity available for allocation by one Resource Provier and available to EGI for allocation. The RP pool is a dynamic entity as it can change over time: for example, the capacity offered, the resource technical specifications and the allocation policies can be changed at any point in time by one Resource Provider.

RP Pool Manager

The RP Pool Manager is the contact point for a given pool, liaises with the resource broker (EGI.eu) and is responsible of defining the local pool allocation policies. In case of a right to revoke resource allocation model, the Manager is responsible of accepting or refusing a resource allocation proposal submitted by the Broker.

EGI Pool

The EGI Resource Pool is the federation of one or more RP Pools. EGI federates resources, i.e. in order to satisfy a given request, resources can be provided by one or multiple federated Resource Providers.

Broker

The broker is the central entity (EGI.eu) responsible of identifying Resource Providers who collectively are able to satisfy a given request. In the proposed process providers are free to choose one of the following models as applicable:

  • (A) EGI.eu is the Broker acting at a central level, while the NGI is the Broker responsible of appointing Resource Providers at a national level; EGI.eu liaises with the NGI to define the local providers offering resources to meet a given request.
  • (B) EGI.eu is the Broker responsible of liaising directly with Resource Centres at a national level.

The process is flexible enough to allow the coexistence of (1) and (2). For example EGI.eu can be delegated the role of Broker at a national level for resource allocations of small entity (where small can be defined by the NGI) with the purpose of keeping the matchmaking process easy.

Matchmaker

Matchmaker is the model that best suites user community requests. EGI.eu will play the Federator as well as the Matchmaker, i.e. is the entity responsible of

  • processing resource requests,
  • negotiating an offer,
  • identifying the Resource Providers willing to offer resources that match the request,
  • establishing and enforcing an Operational Level Agreement (OLA) with the Resource Providers: the OLA involves the Federator and the Resource Provider.
  • establishing and enforcing a Service Level Agreement (SLA) with the Customer: the SLA involves the Customer and the Federator.
  • providing monitoring information (collected from the Resource Provider) and periodic service reports.

Customer

A person signing SLA with Resource Provider, usually representative of a Virtual Organisation.

User

A person using resources on base of binding SLA (could be Customer itself)


Resource Allocation

The distribution of resources among competing groups of people or programs. It is used to assign the available resources in an economic way and to satisfy customer's needs.

Service Level Agreement (SLA)

An agreement between a Service Provider and a customer/client. The SLA describes the IT Service, documents service level targets and specifies the responsibilities of the Service Provider and the customer/client.. A single SLA may cover multiple IT Services or multiple customers/clients.

Operational Level Agreement (OLA)

An agreement between an IT Service Provider (e.g. NGI or Resource Centre) and another part of the same organisation (EGI.eu in its function of Broker). An OLA supports the IT Service Provider’s delivery of IT Services to Users. The OLA defines the goods or Services to be provided and the responsibilities of both parties.

Please refer to the EGI Glossary for the definitions of the terms used.

Computing: Resource allocation QoS levels

By providing resources through a RP pool to the EGI Pool, a Resource Provider can decide to offer different quality of service levels.

Level C1: Opportunistic

This is the basic level which can be provided. Resources are not guaranteed and are subject to local availability at any time: job slots are granted when no other job is submitted by a VO with a higher priority.

The Pool supports the VO for a fixed period of time. The duration of the agreement has to be declared in the VO Request submitted to the Operator and is subject to negotiation. Before expiration, the agreement can be renewed and renegotiated directly with the Resource Provider. Three different queue levels are proposed, each associated to a maximum waiting time.

  • short: 30 minutes max - for NAGIOS probes or short test
  • medium: 12 hours max
  • 48 hours max

The opportunistic model is commonly the default model for RPs that wish to support a VO. Formalizing this model can help VOs to be more aware of who supports them, and fosters resource acknowledgement.

Level C2: Time allocation

In this intermediate service level, resources are available in fair share-like mode for the duration of the agreement: during this duration, the VO is allocated a guaranteed minimum number of normalized hours per day/week/month/, i.e. a daily/weekly/monthly wall time.

The allocated resources can be expressed in normalized Wall Clock Time HEP-SPEC06 hours.

The daily/weekly/monthly/... period is hereafter called Fine Time Scale (FTS). When the VO reaches this threshold, the allocation mode is converted in Opportunistic until the next FTS slot starts.

The Pool supports the VO for a fixed period of time. The the duration of the agreement that must be declared in the VO Request submitted to the Operator. The duration is subject to negotiation.

Fine Time Scale (FTS) may be declared in the VO Request submitted to the Operator, or may be proposed by the Pool Manager, and is subject to negotiation.

Three different queue levels are proposed

  • short: 30 minutes max - for NAGIOS probes or short tests
  • medium: 12 hours max
  • long: 48 hours max

Level C3: Reserved allocation

Resources are exclusively reserved to the VO and the job will be executed immediately after submission, unless all the available job slots are already occupied by jobs of the VO. The resources made available to the VO are expressed in cores, or job slots.

The duration of the agreement has to be declared in the VO Request together with the amount of resources. Subject to negotiation.

In this service, resources are available at any time during a fixed period of time. Since the resources are fully dedicated to the VO, at least 2 different queue levels are proposed:

  • short: 30 minutes max - for NAGIOS probes or short test
  • production: negotiated with the Resource Provider.

The model may be appropriate for well organized large communities, that are able to make 100% use of the dedicated cores during the agreed period.

Storage: Resource allocation QoS levels

Level S1: Opportunistic

Storage space is provided depending on availability of local resources at a time without any guarantees in terms of quota and duration of the allocation. Parameters:

  • overall capacity offered (in GB)
  • maximum available per individual request (in GB)

Level S2: Guaranteed

Storage space is reserved. Parameters:

  • overall capacity offered (in GB)
  • maximum available per individual request (in GB)

Links