Resource Allocation Task Force brainstorming page

Main

EGI.eu operations services

Support

Documentation

Tools

Activities

Performance

Technology

Catch-all Services

Resource Allocation

Security

Introduction

This page is a brainstorming space for Resource Allocation Task Force. It contains ideas and issues for discussion.

Goal

The main goals are:

• Attract new users and allocate them some grid resources
• Provide users with a simplified procedure to find grid resources for their needs and by means of a central unique reference point
• Foster a virtuous cycle among ‘new’ scientific communities, EGI, NGIs, Resource Centers and funding agencies in order to attract new funding to strengthen and expand the European grid infrastructure according to the needs of those scientific communities
• Demonstrate to the national funding agencies, EC (and everybody) that the EGI infrastructure and services are valuable, useful and used by a vast variety of different users

The high level concept

EGI is "a place" where Customer and Provider meet and negotiate resource allocation.

NGI proposals

• NGI IT
• NGI PL

Resource allocation components

Actors

EGI 

• EGI.eu
• supervisor of RA process

Provider

• NGIs that can directly decide on resource allocation on a pool of resources on specific sites
• NGIs that can coordinate resources allocation with their sites
• sites that are identified to allocate resources by passing its NGI. Resource Centres offering resources must be registered in GOCDB as production entities, this ensure that all security policies will be enforced.

Customer

• Regional or international VO
• New or existing VO
• VO should be registered and in production status in Operations Portal

Regional VO:

• Should be up to NGIs to help the VO and allocate resources within NGI
• EGI provides single point of contact for resource allocation for Customers

International VO:

• EGI provides
  • Single point of contact for resource allocation for their Customers
  • Negotiation and signing procedures
  • Monitoring
  • Reporting
    

Terminology

Service Level Agreement (SLA) - An agreement between a Service Provider and a customer/client. The SLA describes the IT Service, documents service level targets and specifies the responsibilities of the Service Provider and the customer/client.. A single SLA may cover multiple IT Services or multiple customers/clients.

Operational Level Agreement (OLA) - An agreement between an IT Service Provider and another part of the same organisation. An OLA supports the IT Service Provider’s delivery of IT Services to Users. The OLA defines the goods or Services to be provided and the responsibilities of both parties.

Please refer to the EGI Glossary for the definitions of the terms used.

Models

Different models can be applied for different requests:

• Mature VO vs New VO
• International or regional VO
• „Big” request vs „Small” request

or NGIs.

Model 1- Open market

Providers’ resources and Customers’ requests are published on an open market (shopping website like for example ebay) – are visible for everyone.

EGI regulates and supervises the market. It determines the rules (e.g. procedures) of the market and acts as a referee in case of agreement violation/negotiations

Pros

• EGI knows what is going on in the infrastructure
• There are clear rules for resource allocation
• In case of agreement violation/negotiations EGI is a referee
  
• Providers knows what are the Customers' current needs
• Customer knows which resources are now available
  

Cons

• For Provider - negotiation process may not be able to be adapted to its needs
• Strong need for a body to match needs to resources

Use cases

• Standard resources' requests

Scientific request review

• EGI decides if the review is needed. If yes then should be done before request is published to the open market.
  

Roles of

• EGI
• NGI
• RP

Model 2 - Broker

Providers’ resources and Customers’ requests are visible to EGI.

EGI matches Providers’ resources to Customers’ requests needs.

Interaction Customer <-> Provider limited.

Provider selection can be done in two ways:

• EGI has up to date information about free resources
• EGI ask Providers to express their interest to support VO - each time request arrives

Pros

• best option for Customer: it gets single point of contact and support from one entity

Cons

• EGI has to get and manage the knowledge what is available in the infrastructure (each site)
• Time consuming for EGI
• Some Providers want to talk with Customers directly
• EGI may not be able to match needs to resources.

Use cases

• big international Customers
• specialised Customer requests
• new international Customers

Scientific request review

• EGI decide is the review is needed. If yes then should be done before request is published to the open market.

Activities

• EGI:
  • collect resource demand from customers and related requirements
  • call for resource offer to create a resource pool
  • establish a resource pool in collaboration with the RPs
  • monitoring
  • accounting collect and provide information about resource usage
  • periodic assessment of usage
  • reporting
  • periodic assessment of acknowledgement of resource usage by VOs
• NGI
• RP

Model 3 - Freedom of choice

Providers’ resources and Customers’ requests transparent - visible for everyone.

The parties have freedom to decide how they want to negotiate resources and under which conditions (no regulation on EGI side).

EGI role is to support Providers and/or Customers on demand. E.g. providing tool for resource allocation, helping in request fulfilment, specialised requests etc.

Pros

• each Provider may want to apply own procedures - pros for Provider
  

Cons

• some Providers may want to apply own procedures - cons for Customer
• Customer has to negotiate with each of the Providers separately
• we can end up with lots of different SLA and negotiation procedures

Use cases

• Regional Customer
  

Scientific request review

• it is up to Provider if it is needed

Role of

• EGI
• NGI
• RP

Issues for discussion

1. SLA
   • who should be the party of the contract
   • what should be included in SLA
2. The roles of NGI, EGI, RC in the process
3. Scientific request review - https://documents.egi.eu/secure/ShowDocument?docid=1472&version=2
4. The acknowledgement for resources use
   • VT_Scientific_Publications_Repository
   • the recommendation about citing EGI in publications is defined here: https://documents.egi.eu/document/1369
     the implementation requires a change in the Grid AUP; the issue has been raised with the Security Coordination Group and David Kelsey is checking with the big customer (WLCG) if/how they are gonna accept it before making the change... work in progress.
     
5. Monitoring of usage
   
6. Provider selection
7. Reporting and evaluation
8. Customer support
9. Reservations/resource pool assurance/guaranteed usage
10. Meeting specific needs of the VO in terms of OS, configuration, software etc.

Requirements received

EGI

1. National VOs need to talk to their NGIs, however, the same processes and interfaces used for EGI international users could be adopted for regional ones.
2. EGI should be a body who investigates, for example, which site could be willing to implement the needed server or software setup or the required storage system etc... and a single point of contact for new users (babysitting)
3. EGI should be responsible of service level management.

Provider

1. Technical review: A technical review of demand is needed complementing the initial scientific peer review (where applicable). RPs must be involved in this, to technically inspect VO requirements to understand the impact on the resource configurations (cluster, storage set-up, network etc.), to understand if the demand complies to the local policies, and to acknowledge that resource allocation is possible for them.
2. Match demand and offer. The resource allocation process should not only allow to address a resource request with an offer, but also vice versa a site with free resources to advertise it.

.

1. Quality of Service. Different levels of Quality of Service could be offered by the federated pool. For example: Level I. Best effort allocation without minimum number of slots allocated (opportunistic usage). Level II. Minimum number of slots allocated at high priority, jobs arriving to that site will enter in execution at the earliest time the cluster occupation permits.
2. Acknowledgement of usage. VOs must periodically acknowledge usage of resources through scientific publication, press releases. The entities to be acknowledged are EGI, the NGIs and the individual RPs contributing resources.

1. The presence of some entity that regulates the process is necessary, e.g. for the enforcement of the security policies, etc. Who decides how the SLA's should behave? We could end up having many different regulations and agreements which, in a shared environment is not very enviable. (Model 2)
   
2. The request and the offer must be matched by someone in the middle who:
   -  knows how many resources are available in the case of a 'static' common pool
   - may know how many resources are available in a dedicated temporary pool by means of a specific call to the NGIs/RPs
3. The internal allocation processes and liaison to individual resource contributors should be made transparent.

Customer

1. Provide a single broker contact point to the user to hide the complexity of a network of heterogeneous RPs and to engage with a single Service Level Agreement, instead of multiple ones with multiple RPs.

1. The broker should support the customer to express technical requirements, where needed.
2. The SLA should be easy enough to be filled in, which means that many service levels should be made optional, as we do now in the VO ID card.
3. The same processes that are being defined for resource allocation, could be generalized to request services, i.e. to allow new user communities to request services to be provided by NGIs, such as application porting.