Alert.png The wiki is deprecated and due to be decommissioned by the end of September 2022.
The content is being migrated to other supports, new updates will be ignored and lost.
If needed you can get in touch with EGI SDIS team using operations @ egi.eu.

Difference between revisions of "PROC08 Management of the EGI OPS Availability and Reliability Profile"

From EGIWiki
Jump to navigation Jump to search
(59 intermediate revisions by 7 users not shown)
Line 1: Line 1:
{{Template: Op menubar}}
{{Template:Op menubar}}
{{Template:Doc_menubar}}
{{Template:Doc_menubar}}
[[Category:Operations Manuals]]
[[Category:Operations Procedures]]
[[Category:Deprecated]]
{| style="border:1px solid black; background-color:lightgrey; color: black; padding:5px; font-size:140%; width: 90%; margin: auto;"
| style="padding-right: 15px; padding-left: 15px;" |
|[[File:Alert.png]] This page is '''Deprecated'''; the content has been moved to https://confluence.egi.eu/display/EGIPP/PROC08+Management+of+the+EGI+OPS+Availability+and+Reliability+Profile 
|}
{{TOC_right}}
{{TOC_right}}


{| border="1"
{{Ops_procedures
|-
|Doc_title = Management of the EGI OPS Availability and Reliability Profile
| '''Title'''
|Doc_link = [[PROC08|https://wiki.egi.eu/wiki/PROC08]]
| ''Modification of the set of AVAILABILITY tests''
|Version = 2020-03-02
|-
|Policy_acronym = OMB
|'''Version'''
|Policy_name = Operations Management Board
| 1.0
|Contact_group = operations@egi.eu
|-
|Doc_status = Approved
| '''Document link'''
|Approval_date = 2020-02-20
| ''https://wiki.egi.eu/wiki/PROC08_Modification_of_the_set_of_AVAILABILITY_tests''
|Procedure_statement = This document specifies the procedure for modifying the EGI OPS Availability and Reliability profile.
|-
|Owner = Alessandro Paolini
| '''Last modified'''
}}
| 16:18, 16 March 2011 (UTC)
 
|-
<br>
| '''Policy Group Acronym'''
| ''OMB''
|-
| '''Policy Group Name'''
| ''Operations Management Board''
|-
| '''Contact Person'''
| ''E. Imamagic''
|-
| '''Document Status'''
| ''DRAFT''
|-
| '''Approved Date'''
| ''specify''
|-
| '''Procedure Statement'''
| ''This document specifies the procedure for modifying the set of AVAILABILITY tests, i.e. of those tests whose results affect the computation of the monthly Availability and Reliability statistics.''
|-
|}


----
= Overview  =


= Overview =
A change in the profile is needed every time a new Nagios test needs to be added/removed to/from the profile, in order to have its results included/removed in/from Availability and Reliability monthly statistics. A change in the OPS Availability and Reliability profile affects the computation of the monthly Availability and Reliability statistics of all EGI Resource Infrastructures and Resource Centres.


The purpose of this document is to clearly describe the procedure for modifying the set of [[Availability_and_reliability_tests|AVAILABILITY tests]], i.e. of those tests whose results affect the computation of the monthly Availability and Reliability (A/R) statistics.
= Definitions  =


= Definitions =
*The key words '''Profile''', '''Metric''', '''Probe''' and '''Test''' are defined in the [https://wiki.egi.eu/wiki/ARGO#ARGO_tests ARGO] page.
*List of Availability and Reliability tests: [https://poem.egi.eu/poem/admin/poem/profile/2/change/ ARGO_MON_CRITICAL].


The key words '''Profile''', '''Metric''', '''Probe''' and '''Test''' are defined in the [[SAM Tests]] page.  
Please refer to the [[Glossary|EGI Glossary]] for the definitions of the terms used in this procedure.<br>


The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", “MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119.
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", “MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119.


= Scope =
= Scope =


This procedure is applicable to the set of AVAILABILITY tests which are run under OPS VO and its range is global, as they are applied to all EGI Resource Centres. These tests are used in the official EGI [[External_tools|ACE]] profile used for generating monthly A/R reports.
This procedure is applicable to the EGI OPS Availability and Reliability profile. Any change applied is global, as it has effects on all EGI Resource Centres. The ARGO compute engine (CE) uses profiles to generate monthly Availability and Reliability reports.  


This procedure does not apply to availability/reliability statistics calculated for other VOs (e.g. user communities, national operations VOs).
This procedure is NOT applicable to VO-specific Availability and Reliability profiles used by non-OPS VOs (e.g. user communities, national operations VOs, etc.).  


This procedure does not apply to modifications which have already been agreed with the SAM team:
= Entities involved in the procedure =
* including CREAM-CE results
* switching from the old SAM CA probe to the new one
* switching from the old SAM ARC probes to Nagios ones.


= Entities involved in the procedure =
*'''Applicant'''. The Applicant submits a request for changing the EGI OPS profile. Anybody is allowed to submit the request. The request is submitted to the respective Operations Centre, who after acceptance, forward it to the Operations Management Board [[OMB|OMB]] for discussion.
*'''Operations Centre'''. The entity associated to EGI that is responsible of delivering local operational services to a Resource Infrastructure Provider. In order to contribute resources to EGI a Resource Infrastructure Provider must be associated to an Operations Centre.
*'''Resource Infrastructure Operations Manager'''. Represents the respective Resource Infrastructure within the OMB.
*'''EGI&nbsp;Operations team'''. Team coordinating EGI Infrastructure.
*'''ARGO Team'''. The ARGO Team is responsible of scheduling, integrating and releasing probes.


* '''Applicant'''. The Applicant submits a request for adding a new AVAILABILITY test (or tests). Anybody is allowed to submit the request for modifying the set of AVAILABILITY tests. The request is submitted to the respective Operations Centre, who after acceptance, forward it to the Operations Management Board [[OMB|OMB]] for discussion.
= Pre-requirements  =
* '''Operations Centre'''. The entity associated to EGI that is responsible of delivering local operational services to a Resource Infrastructure Provider. In order to contribute resources to EGI a Resource Infrastructure Provider must be associated to an Operations Centre.
* '''Resource Infrastructure Operations Manager'''. Represents the respective Resource Infrastructure within the OMB.
* '''Chief Operations Officer''' (COO). COO is the chairman of the Operations Management Board [[OMB|OMB]].
* '''SAM Product Team'''. The SAM Product Team is responsible of scheduling, integrating and releasing probes.


= Pre-requirements =
*This procedure requires usage of the [http://argo.egi.eu ARGO] system for generating monthly availability and reliability statistics. Detailed information about ARGO system can be found on the following link: [http://argoeu.github.io/overview/ http://argoeu.github.io/overview/].
*If the request of change includes the addition of new tests, each test MUST first be integrated with the Operations Dashboard, i.e. being an OPERATIONS test included in the [https://poem.egi.eu/ui/metricprofiles/ARGO_MON_OPERATORS ARGO_MON_OPERATORS] profile is a necessary condition to be included in the [https://poem.egi.eu/ui/metricprofiles/ARGO_MON_CRITICAL ARGO_MON_CRITICAL] profile for A/R computation (see procedure [[PROC06|PROC06]]). This should ensure that the new probe has been tested on the production sites, excluding the occurrence of anomalies not strictly depending on the sites.
*EGI software teams are responsible for the development and testing of new probes.


This procedure requires usage of the [[External_tools|ACE]] system for generating monthly availability and reliability statistics. The procedure is not applicable to the GridView system which is currently used. The critical feature which ACE supports and GridView lacks is definition of multiple profiles for availability and reliability statistics.
= Steps  =
<!-- Detailed information about ACE system can be found on the following link: https://tomtools.cern.ch/confluence/display/SAM/ACE.-->


If the request of change includes the addition of new tests, each test MUST first go through the following steps:
{| class="wikitable"
* integration of the probe in the SAM release (see procedure [[PROC07_Adding_new_probes_to_SAM|PROC07]]);
|-
* integration of the test in the Operations Dashboard (i.e. being an OPERATIONS test is a necessary condition to be an AVAILABILITY test) (see procedure [[PROC06_Setting_a_Nagios_test_status_to_OPERATIONS|PROC06]]).
! Step  
Two procedures above assure that the new tests are included in the SAM release, deployed on all Resource Infrastructure SAM instances and accepted by Operations Centre operators.
! Action on  
 
= Steps =
 
{| border="1" cellspacing="0" cellpadding="5" align="center"
! Step
! Action on
! Action
! Action
|-
|-
| 1
| 1  
| Applicant
| Applicant  
| Sends a change request to the attention of the respective own Operations Centre. The request is submitted through a [https://gus.fzk.de/pages/ticket.php GGUS ticket].  
| Sends a change request to the attention of the respective own Operations Centre. The request is submitted through a [https://gus.fzk.de/pages/ticket.php GGUS ticket].  
Use the "Affected ROC/NGI" to address the ticket to the appropriate Operations Centre.
Use the "Affected ROC/NGI" to address the ticket to the appropriate Operations Centre. Template:  
Template:
<pre>Subject: Request for adding/removing XXX(,YYY,...) test(s) to/from the EGI OPS A/R Profile
<pre>
Subject: Request for adding/removing XXX(,YYY,...) test(s) from the set of AVAILABILITY tests


We would like to request adding/removing XXX(,YYY,...) test(s) from the set of AVAILABILITY tests
We would like to request adding/removing XXX(,YYY,...) test(s) to/from from the EGI OPS Profile


Prerequisite data:
Prerequisite data:
* name of SAM test(s):
* name of ARGO test(s):
* name of service on which the test runs:
* name of service on which the test runs:
* link to documentation page:
* link to documentation page:
Line 105: Line 82:
</pre>
</pre>
|-
|-
| 2
| 2  
| Operations Centre
| Operations Centre  
| The Operations Centre processes the request specified in the GGUS ticket for acceptance/rejection.  
| The Operations Centre process the request specified in the GGUS ticket for acceptance/rejection.  
Motivations for rejection need to specify in the GGUS ticket.  
Motivations for rejection need to specify in the GGUS ticket.  
In case of acceptance, a [https://rt.egi.eu/rt/index.html RT ticket] is opened in queue '''noc-managers''' to forward the request for discussion in the OMB.
Template:
<pre>
Subject: Request for adding/removing XXX(,YYY,...) test(s) from the set of AVAILABILITY tests


We would like to request adding/removing XXX(,YYY,...) test(s) from the set of AVAILABILITY tests.  
In case of acceptance, a GGUS ticket is opened to EGI Operations Support Unit to forward the request for discussion in the OMB. Template:
<pre>Subject: Request for adding/removing XXX(,YYY,...) test(s) to/from the EGI OPS Profile
 
We would like to request adding/removing XXX(,YYY,...) test(s) to/from the EGI OPS Profile.  
Please see details in GGUS ticket _link to Applicant's GGUS ticket_.
Please see details in GGUS ticket _link to Applicant's GGUS ticket_.
</pre>
</pre>
|-
|-
| 3
| 3  
| COO and Resource Infrastructure Operations Manager
| EGI Opertions team&nbsp; and Resource Infrastructure Operations Manager  
| COO schedules a presentation of the change requested at the next possible OMB meeting. The relevant Resource Infrastructure Operations Manager presents the request during the meeting. The Applicant is invited to attend the meeting. Only one request will be processed at a time as the impact of a change needs to be assessed. Requests are processed depending on their priority, as agreed by the [[OMB|OMB]].
| EGI Opertions team schedules a presentation of the change requested at the next possible OMB meeting. The relevant Resource Infrastructure Operations Manager presents the request during the meeting. The Applicant is invited to attend the meeting. Only one request will be processed at a time as the impact of a change needs to be assessed. Requests are processed depending on their priority, as agreed by the [[OMB|OMB]].
|-
|-
| 4 (*)
| 4
| COO
| EGI Opertions team
| Opens a GGUS ticket requesting the creation of a new ACE profile with a modified set of AVAILABILITY tests to the SAM/Nagios 3rd level SU.
| Opens a GGUS ticket to "Monitoring (ARGO)" requesting the addition (or removal) to the ARGO_MON_CRITICAL profile.  
|-
|-
| 5 (*)
| 5  
| ACE team
| ARGO team  
| ACE team creates the new ACE profile.
| Implements the change on the agreed date (generally, the first day of the month).
|-
|-
| 6
| 6  
| SA1.8 staff
| EGI Opertions team
| For the following '''one month''' two availability and reliability reports are generated. SA1.8 staff compares the figures and presents them at the next OMB meeting with a recommendation for acceptance or rejection.
| Before closing the ticket, verifies there are no anomalies with the new A/R reports and report back to the NGI Managers by email or to the next OMB meeting.
|-
|-
| 7
| 7  
| OMB
| EGI Opertions team
| If the availability and reliability statistics generated with the new profile are satisfactory OMB approves the modification.
| Broadcasts the modification to all relevant parties (i.e. Operations Centres and Resource Centres) through the next Monthly Broadcast. Closes the GGUS ticket&nbsp;
|}
 
= Revision History  =
 
{| class="wikitable"
|-
! Version
! Authors
! Date
! Comments
|-
| <br>
| T. Ferrari
| 18/10/2011
| removed obsolete information: "This procedure does not apply to modifications which have already been agreed with the SAM team: including CREAM-CE probes, switching from the old SAM CA probe to the new one, switching from the old SAM ARC probes to Nagios ones."
|-
| <br>
| M. Krakowian
| 19 August 2014
| Change contact group -&gt; Operations support
|-
|-
| 8 (*)
| <br>
| COO
| P. Daoglou and P. Korosoglou
| Opens a child GGUS ticket requesting that the new availability and reliability profile becomes the official for EGI.
| 30 September 2015
| Updates in procedure. Replacement of ACE references with ARGO.
|-
|-
| 9
|  
| COO
| Alessandro Paolini
| Broadcasts the modification to all relevant parties (i.e. Operations Centres and Resource Centres). Closes the GGUS ticket (parent and child).
| 2016-06-08
| Change contact group -&gt; Operations  
|-
|
| Alessandro Paolini
| 2019-12-17
| updating some old links and names; updating some steps (it seems it is no more necessary creating a "testing" A/R report before approving the changes)
|}
|}


(*) - These steps depend on the procedure for creating new profiles which will be defined by the ACE team once the ACE is in production. Steps defined here have been provided by the ACE team. This procedure will be updated if any change occurs.
[[Category:Operations_Procedures]]
 
= Revision History =
<!-- this section will track changes introduced in the document AFTER it is officially approved by OMB -->

Revision as of 14:37, 4 August 2021

Main EGI.eu operations services Support Documentation Tools Activities Performance Technology Catch-all Services Resource Allocation Security


Documentation menu: Home Manuals Procedures Training Other Contact For: VO managers Administrators
Alert.png This page is Deprecated; the content has been moved to https://confluence.egi.eu/display/EGIPP/PROC08+Management+of+the+EGI+OPS+Availability+and+Reliability+Profile


Title Management of the EGI OPS Availability and Reliability Profile
Document link https://wiki.egi.eu/wiki/PROC08
Last modified 2020-03-02
Policy Group Acronym OMB
Policy Group Name Operations Management Board
Contact Group operations@egi.eu
Document Status Approved
Approved Date 2020-02-20
Procedure Statement This document specifies the procedure for modifying the EGI OPS Availability and Reliability profile.
Owner Alessandro Paolini



Overview

A change in the profile is needed every time a new Nagios test needs to be added/removed to/from the profile, in order to have its results included/removed in/from Availability and Reliability monthly statistics. A change in the OPS Availability and Reliability profile affects the computation of the monthly Availability and Reliability statistics of all EGI Resource Infrastructures and Resource Centres.

Definitions

  • The key words Profile, Metric, Probe and Test are defined in the ARGO page.
  • List of Availability and Reliability tests: ARGO_MON_CRITICAL.

Please refer to the EGI Glossary for the definitions of the terms used in this procedure.

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", “MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119.

Scope

This procedure is applicable to the EGI OPS Availability and Reliability profile. Any change applied is global, as it has effects on all EGI Resource Centres. The ARGO compute engine (CE) uses profiles to generate monthly Availability and Reliability reports.

This procedure is NOT applicable to VO-specific Availability and Reliability profiles used by non-OPS VOs (e.g. user communities, national operations VOs, etc.).

Entities involved in the procedure

  • Applicant. The Applicant submits a request for changing the EGI OPS profile. Anybody is allowed to submit the request. The request is submitted to the respective Operations Centre, who after acceptance, forward it to the Operations Management Board OMB for discussion.
  • Operations Centre. The entity associated to EGI that is responsible of delivering local operational services to a Resource Infrastructure Provider. In order to contribute resources to EGI a Resource Infrastructure Provider must be associated to an Operations Centre.
  • Resource Infrastructure Operations Manager. Represents the respective Resource Infrastructure within the OMB.
  • EGI Operations team. Team coordinating EGI Infrastructure.
  • ARGO Team. The ARGO Team is responsible of scheduling, integrating and releasing probes.

Pre-requirements

  • This procedure requires usage of the ARGO system for generating monthly availability and reliability statistics. Detailed information about ARGO system can be found on the following link: http://argoeu.github.io/overview/.
  • If the request of change includes the addition of new tests, each test MUST first be integrated with the Operations Dashboard, i.e. being an OPERATIONS test included in the ARGO_MON_OPERATORS profile is a necessary condition to be included in the ARGO_MON_CRITICAL profile for A/R computation (see procedure PROC06). This should ensure that the new probe has been tested on the production sites, excluding the occurrence of anomalies not strictly depending on the sites.
  • EGI software teams are responsible for the development and testing of new probes.

Steps

Step Action on Action
1 Applicant Sends a change request to the attention of the respective own Operations Centre. The request is submitted through a GGUS ticket.

Use the "Affected ROC/NGI" to address the ticket to the appropriate Operations Centre. Template:

Subject: Request for adding/removing XXX(,YYY,...) test(s) to/from the EGI OPS A/R Profile

We would like to request adding/removing XXX(,YYY,...) test(s) to/from from the EGI OPS Profile

Prerequisite data:
* name of ARGO test(s):
* name of service on which the test runs:
* link to documentation page:
* motivation (which part of the infrastructure will be improved with the new test
 or description of users' problems which will be avoided in future - provide list
 of GGUS tickets is possible)
2 Operations Centre The Operations Centre process the request specified in the GGUS ticket for acceptance/rejection.

Motivations for rejection need to specify in the GGUS ticket.

In case of acceptance, a GGUS ticket is opened to EGI Operations Support Unit to forward the request for discussion in the OMB. Template:

Subject: Request for adding/removing XXX(,YYY,...) test(s) to/from the EGI OPS Profile

We would like to request adding/removing XXX(,YYY,...) test(s) to/from the EGI OPS Profile. 
Please see details in GGUS ticket _link to Applicant's GGUS ticket_.
3 EGI Opertions team  and Resource Infrastructure Operations Manager EGI Opertions team schedules a presentation of the change requested at the next possible OMB meeting. The relevant Resource Infrastructure Operations Manager presents the request during the meeting. The Applicant is invited to attend the meeting. Only one request will be processed at a time as the impact of a change needs to be assessed. Requests are processed depending on their priority, as agreed by the OMB.
4 EGI Opertions team Opens a GGUS ticket to "Monitoring (ARGO)" requesting the addition (or removal) to the ARGO_MON_CRITICAL profile.
5 ARGO team Implements the change on the agreed date (generally, the first day of the month).
6 EGI Opertions team Before closing the ticket, verifies there are no anomalies with the new A/R reports and report back to the NGI Managers by email or to the next OMB meeting.
7 EGI Opertions team Broadcasts the modification to all relevant parties (i.e. Operations Centres and Resource Centres) through the next Monthly Broadcast. Closes the GGUS ticket 

Revision History

Version Authors Date Comments

T. Ferrari 18/10/2011 removed obsolete information: "This procedure does not apply to modifications which have already been agreed with the SAM team: including CREAM-CE probes, switching from the old SAM CA probe to the new one, switching from the old SAM ARC probes to Nagios ones."

M. Krakowian 19 August 2014 Change contact group -> Operations support

P. Daoglou and P. Korosoglou 30 September 2015 Updates in procedure. Replacement of ACE references with ARGO.
Alessandro Paolini 2016-06-08 Change contact group -> Operations
Alessandro Paolini 2019-12-17 updating some old links and names; updating some steps (it seems it is no more necessary creating a "testing" A/R report before approving the changes)