Resource Centres OLA and Resource infrastructure Provider OLA reports

Introduction

wall of shame

EGI availability and reliability statistics are produced every month for all certified production sites. The current version of the Site-NGI Operational Level Agreement defines the following requirements:

• minimum tolerated availability: 70%,
• minimum tolerated reliabilty: 75%.

For each monthly report, underperforming sites are requested through a GGUS to motivate the poor performance provided.

Suspension procedure: sites which have an availability of less than 50% for three consecutive months will be suspended, i.e. removed from the production infrastructure.

Tools and documentation

• xls availability/reliability report generator (providing access to the database including Nagios results for OPS and SAM results for VOs)
• Availability/reliability computation algorithm
• Availability Computation Engine (ACE)
• OLD: EGEE-III Comments on site availability and reliability statistics

Note: in EGI sites do not need to provide comments anymore. In EGI comments will be solicited and collected through GGUS tickets instead.

Statistics

• June 2010 [1]
• May 2010 [2]
• January 2008 - April 2010 (EGEE league tables)

Suspended sites

• List of suspended sites (2009)

Description of the process

• Generation of statistics

Availability and reliability statistics are automatically generated the first week of the month by GridView in pdf format and placed under [3]. An Excel version is available at [4]

• Preliminary processing

Once the reports are generated, sanity checks are performed by EGI SA1 (Task TSA1.8). After this step is completed, statistics are uploaded into the EGI document server. Links to monthly statistics will be provided on a regular basis at this wiki page.

• Publication

An announcement of the new results is distributed by EGI SA1 (TSA1.8) to the NGI Operations Managers mailing list. COD (TSA1.7) is responsible of supervising statistics by chasing NGIs to chase sites that need to provide comments in case thresholds are not met, and identifies sites eligible for suspension. This phase starts by filing a ticket to the COD Support Unit. The overall comments gathering process is handled through tickets.

• Handling of sites below targets

For a site that misses availability/reliability targets but is not eligible for suspension:

1. a child ticket is opened by the COD team and assigned to the respective NGI, asking for explanation to be given
2. the explanation must be produced within 10 working days since the ticket is received by the site (please see known issues section [5])
3. if the explanation is found satisfactory the ticket is closed
4. conversely if the explanation is not given in due time, or the explanation is found inadequate:
   • the EGI Chief Operations Officer is informed and he/she can decide within 3 working days after the deadline if he/she objects to the site being added to a wiki folow up page listing sites that fail to provide explanations
   • after the 3 days pass, the site is added to the wiki follow up page by COD, unless the EGI Chief Operations Officer has objected or decided to accelerate the process in the previous step
5. the child ticket can then be closed
6. the parent ticket will be closed when all child tickets have been closed.

• Handling of sites that are eligible for suspension

For a site that is eligible for suspension:

1. a child ticket is opened by the COD team assigned to appropriate NGI, notifying that the site will be suspended within 10 working days (please see known issues section [6])
2. after the 10 days period passes, the site is suspended by COD unless the NGI has intervened or the EGI Chief Operations officer objects
3. in the case of NGI intervention, non suspension will occur if both the COD and COO agree on the reasoning provided by the NGI
4. the child ticket closes either when the site is suspended or when suspension is canceled
5. the parent ticket will be closed when all child tickets have been closed

• Removal from the wiki follow up page

Every month, the COD will examine if a site that belonged to the wiki follow up page has managed to meet its targets. In this case, COD will remove the site from the wiki.

Known issues and recommendations to NGIs

1. When reprocessing data, GridView takes into account the Certification status of the site at that moment and not the full Certification status history. Thus newly certified sites will get inaccurate Availability/Reliability figures for the month they were certified. Because the Certification status history is not currently available in the operations tools, until a solution is implemented NGIs should check if they have sites affected by this issue and report it as explanation. More information at [7]