Difference between revisions of "FAQ Regional Operator on Duty"

From EGIWiki
Jump to: navigation, search
Line 37: Line 37:
 
In case of problems with accounting it is not recommended to suggest downtime at the second step of the escalation process for this test. Accounting service is not a functionality which is critical for users but it still need to be follow up.
 
In case of problems with accounting it is not recommended to suggest downtime at the second step of the escalation process for this test. Accounting service is not a functionality which is critical for users but it still need to be follow up.
  
= Temporarily unattended site =
+
= Watch out for flapping states =
  
If the site administrator has a scheduled (vacation, conference, meeting) or unscheduled absence (illness), and the site has no back up administrator they should:
+
You may want to wait for a second test to be run before closing an alarm which is in an OK state. This ensures that the OK result for that tests is stable. The waiting period is, of course, dependent on how long the test takes and how frequently it is checked.
 
 
* notify their ROD team about the length of the absence (or estimated length) including the starting date.
 
* The site administrator (or ROD team) should put the site into an AT RISK (WARNING) downtime, specifying absence as the reason for this downtime. This keeps the site active.  
 
* ROD makes a note of this and in the case of a problem at the site can put the site into unscheduled downtime.
 
* When the site administrator returns, they can either end the downtime, or, in the case of an incident, fix the problem and then end the downtime.
 

Revision as of 12:00, 25 October 2012

Main EGI.eu operations services Support Documentation Tools Activities Performance Technology Catch-all Services Resource Allocation Security



How to handl issues during weekends and public holidays?

Due to the fact that weekends and public holidays are not considered working days it is noted that ROD teams do not have any responsibilities during these days. RODs should ensure that in these days tickets do not expire and alarms will not age above 72h.

What to do with alarms when node is not in production and is part of production sit?

It often happens that testing nodes on production sites are set as non-production. In such case Nagios monitoring system will send information about all nodes. As a result ROD will see on their dashboard alarms for non-production node. If it necessary to monitor such testing node it is recommended to put such non-production node in downtime.

What to do when a sites have multiple alarms/ticket?

When opening a ticket against a site with existing tickets ROD should consider that these problems may be linked or dependant on pending solutions. In such case ROD should use masking mechanism to gather and assigne alarms to one ticket rather than open a ticket for each alarm.

If the problem is different but maybe linked the expiry dates for each ticket should be synchronized to the latest date.

How to handle issues for site/node in downtime?

Handling tickets for site/node in downime

When a ticket has been raised against a site that subsequently enters downtime time, the expiry date on the ticket can be extended.

Sites that are in downtime will still have monitoring switched on and therefore may appear to be failing tests but no alarms on Operations Portal will be raised against them. ROD must take care that when opening tickets to ensure that they don't open tickets against sites in downtime.

Handling alarms for site/node in downime

It often happens that a failure occurred generating a lot of alarms and then site manager decided to put site in Downtime. Getting these alarms OK may take more than 72h when the issue is escalated to COD.
ROD should not create a ticket for sites/nodes in Downtime and is not obligated to deal with such alarms but it is recommended to close these alarms to avoid being escalated to COD. In such case as a reson of closing NON-OK alarm ROD should put link to the downtime in GOC DB.

Site in downtime for more than a month

If a site is in DOWNTIME for more than a month then it is advised that the site should go to the uncertified state.

What to do in case of accounting issue?

In case of problems with accounting it is not recommended to suggest downtime at the second step of the escalation process for this test. Accounting service is not a functionality which is critical for users but it still need to be follow up.

Watch out for flapping states

You may want to wait for a second test to be run before closing an alarm which is in an OK state. This ensures that the OK result for that tests is stable. The waiting period is, of course, dependent on how long the test takes and how frequently it is checked.