PROC06 Setting Nagios test status to operations

• Title: Procedure for setting Nagios test critical for COD
• Document link: https://wiki.egi.eu/wiki/Operations:Setting_Nagios_tests_critical_procedure
• Last modified: 19.11.2010
• Version: 0.4
• Policy Group Acronym: GOO/COD
• Policy Group Name: Grid Operations Oversight/Central Operator on Duty
• Contact Person: Małgorzata Krakowian, Marcin Radecki
• Document Status: REVIEW
• Approved Date:
• Procedure Statement:The purpose of this document is to clearly describe the actions and the relative steps to be undertaken for setting Nagios tests critical. A Nagios test is set to critical to enable the operations dashboard to display an alarm in case the test fails.

Procedure for setting Nagios test critical for COD DRAFT

The purpose of this document is to clearly describe the actions and the relative steps to be undertaken for setting Nagios tests critical for COD. As a result the operations dashboard displays an alarm in case the test fails.

This procedure only applies for tests run under OPS VO and its range is global, applies for all Operations Centres in EGI project.

Revision history

Version	Authors	Date	Comments
0.4	Marcin Radecki, Małgorzata Krakowian	19.11.2010	Preparing for final version
0.3	Małgorzata Krakowian	9.11.2010	Add comments from Tiziana Ferrari
0.2	Małgorzata Krakowian	23.09.2010	Add comments from discussion in Amsterdam EGI TF.
0.1	Małgorzata Krakowian		First draft

Comments (to be removed in final version)

HC: nagios people at NGI should be allowed to play with the new test before it is made critical. Point is to check if the test itself will not make any harm to the service instances in the region.

Answer: The test is examined by the Nagios team before including in official nagios package, so it should not make any harm to regional nagios instance. The test is run for some time in all production instances (see prerequisites below). This period can be considered a time for validation.

TF: comment to step 4: is this (percentage of passing sites) sufficient as a check? sites may fail to pass, but the probe could be fine.

Answer: Step 4 is to avoid massive generation of alarms in dashboard. Usually, before making the test critical COD asks RODs to try to get the test to OK status or report if not possible. We believe that if the RODs are not able to correct the test in Step 4 they will also have problems after making the test critical which means a big workload for COD.

Setting Nagios tests critical request

Request

• Everyone is allowed to submit the request for making the test critical for COD.
• The request should be submitted to the Chief Operations Officer.

Prerequisites

• The Nagios test needs to satisfy quality criteria in agreement with the UMD roadmap.
• The test needs to be properly documented, and its correct functionality need to be proven (one month of successful running o=in the production infrastructure).
• The test needs to run for at least 1 month on all regional instances (it must be present in official nagios package to provide data for validation).

Quality criteria needs to be provided by SA1

Validation

The general idea is that tickets must be closed before being able to move on to the next step.

Steps:

Step	Action on	Action
1	COO	Opens a GGUS ticket to COD to start the process.
2	COD	Checks the status of the Nagios probe to see if it meets the specified quality criteria (to be defined).
3	COD	COD contacts the OMB to request the approval of the new critical test. Date is specified (at least 1 month in future)
4	NGIs	Request to the ROD teams to try making the test OK. 75% OK in total (entire EGI) is understood as threshold for passing to the next step. If not possible to proceed, report problems to OMB.
5	COD	The announcement about new critical test is broadcast by COD. (This broadcast should be sent to VO managers and NOC/ROC managers) See the template below for an indication of the message content. Subject: Dear All, We would like to announce that test XXX will become critical XXX Best regards,
6	COD	Add the test to critical tests list. https://wiki.egi.eu/wiki/Operations:Operations_tests
7	Operational Portal	Mark the test as a critical in Operational Portal. This step will be removed when COD gets an access to manage operations tests list in Operational Portal. See Requirements to be implemented
8	COD	Final check. Close parent ticket

Requirements to be implemented

• COD is responsible to manage OPS critical test list from operational portal side. https://rt.egi.eu/rt/Ticket/Display.html?id=482