The wiki is deprecated and due to be decommissioned by the end of September 2022.
The content is being migrated to other supports, new updates will be ignored and lost.
If needed you can get in touch with EGI SDIS team using operations @ egi.eu.
PROC01 EGI Infrastructure Oversight escalation
Workflow and escalation procedures (improved)
Key changes:
- The procedure has 3 parts:
- escalation for operational problem at site
- escalation for security problem at site
- escalation for operational problem with ROD
- (for 1 part) NGI manager is asked for suspension instead of COD performing it, step 3 lasting 4 days shortened to 3 days, 1 remaining days moved to step 4. Total procedure time unchanged.
- (for 2 part) part was added
- (for 3 part) part was added
Escalation for operational problem at site
This section introduces a critical part of operations in terms of sites' problems detection, identification and solving. The escalation procedure is a procedure that Operators must follow whenever any problem related to a site is detected. The main goal of the procedure is to track the problem follow-up process as a whole and keep the process consistent from the time of detection until the time when the ultimate solution is reached.
Moreover, the procedure is supposed to introduce a hierarchical structure and responsibility distribution in problem solving which should lead to significant improvement in the quality of the production grid service. Consequently, minimizing the delay between the steps of the procedure is of utmost importance. The regular procedure the operators follow can be considered in four phases.
- submitting problems into the problem tracking tool after they are detected using monitoring tools or by a task created by an operations team (COD or ROD);
- updating the task when a site state changes which can be detected either by a comparison of the monitoring information with the current state of the task in the problem tracking tool, or by input from an operations team (COD or ROD);
- closing tickets or escalating outdated tickets when deadlines are reached in the problem tracking tool;
- initiate last escalation step and/or communication with site administrators and regional operations team.
Below are the detailed steps of the escalation procedure if no response is received for the notification of a problem or the problem has been unattended for:
| Step [#] | Max. Duration [work days] | Resp. Unit | Escalation procedure |
| 1 | 3 | ROD | When an alarm appears on the ROD dashboard (>24 hours old): 1st mail to site admin |
| 2 | 3 | ROD | 2nd mail to site admin (optionally: a phone call to site, just to make sure that communication channel is working); At the end of this period escalate to COD |
| 3 | 3 | COD | Ticket escalated to COD, COD should act on the ticket by sending email to the NGI manager informing that he/she should make the site react on the ticket or suspend the site within 3 days and if NGI will not react COD will suspend the site on the 4th day. Inform also ROD and site again. |
| 4 | 1 | COD | If no response is obtained from either the site, ROD or NGI manager COD sends an e-mail asking NGI manager to suspend the site. If no response after 1 working day COD performs site suspension. |
NB : Theoretically, the whole process could be covered in a 2 week period. This is in line with suspension procedure proposed for not replying to low availability/reliability figures. Most often a site solves the problem well before operators need to escalate issue to COD or the site is suspended on the spot for security reasons.
Comments (not part of the procedure):
MR: The below refers to inability to solve the issue, not reply. We do not have to chase sites for not solving the problem as if they cannot solve the problem it affects the availability - low availability will be tracked in other way.
NB: After the first 3 days, at the 2nd escalation step, if the site has not solved its problem, ROD should suggest to the site to declare downtime until they solve the problem and the NGI should be notified. If they do not accept the downtime then COD will proceed with the regular escalation procedure at the agreed deadlines.
Escalation for security problem at site
Below are the detailed steps of the escalation procedure if security issue has been identified at site and security team (which? TODO put name) decided to suspend the site:
| Step [#] | Max. Duration [work days] | Resp. Unit | Escalation procedure |
| 1 | 3 | COD? - Ron to check if correct | Security team (which-TODO) sends a ticket to COD asking for site suspension??. COD should ask NGI manager. |
| 2 | 1 | COD? | If no response is obtained from either the site, ROD or NGI manager COD sends an e-mail asking NGI manager to suspend the site. If no response after 1 working day COD performs site suspension. |
Escalation for operational problem with ROD
The procedure applies only in case when ROD is not handling issues on operational dashboard according to operational procedures.
| Step [#] | Max. Duration [work days] | Resp. Unit | Escalation procedure |
| 1 | 3 | COD | Contact the ROD with CC to NGI manager. Content of the message:
|
| 2 | 3 | COD | Contact NGI manager with CC to ROD. Content of the message:
|
| 3 | (without delay) | COD | Contact COO with CC NGI manager. Content of the message:
|
The precondition to stop escalation is that all issues not handled according to procedure disappeared from COD dashboard. All communication is done via GGUS ticket.
(Old version of) Workflow and escalation procedure (as presented on OMB in Amsterdam during ETF)
This section introduces a critical part of operations in terms of sites' problems detection, identification and solving. The escalation procedure is a procedure that Operators must follow whenever any problem related to a site is detected. The main goal of the procedure is to track the problem follow-up process as a whole and keep the process consistent from the time of detection until the time when the ultimate solution is reached.
Moreover, the procedure is supposed to introduce a hierarchical structure and responsibility distribution in problem solving which should lead to significant improvement in the quality of the production grid service. Consequently, minimizing the delay between the steps of the procedure is of utmost importance. The regular procedure the operators follow can be considered in four phases.
- submitting problems into the problem tracking tool after they are detected using monitoring tools or by a task created by an operations team (COD or ROD);
- updating the task when a site state changes which can be detected either by a comparison of the monitoring information with the current state of the task in the problem tracking tool, or by input from an operations team (COD or ROD);
- closing tickets or escalating outdated tickets when deadlines are reached in the problem tracking tool;
- initiate last escalation step and/or communication with site administrators and regional operations team.
Below are the detailed steps of the escalation procedure if no response is received for the notification of a problem or the problem has been unattended for:
| Step [#] | Max. Duration [work days] | Resp. Unit | Escalation procedure |
| 1 | 3 | ROD | When an alarm appears on the ROD dashboard (>24 hours old): 1st mail to site admin |
| 2 | 3 | ROD | 2nd mail to site admin (optionally: a phone call to site, just to make sure that communication channel is working); At the end of this period escalate to COD |
| 3 | 4 | COD | Ticket escalated to COD, COD should in that week, act on the ticket by sending email to the NGI manager, ROD and site for immediate action otherwise site suspension will happen. |
| 4 | - | COD | (IF no response is obtained from either the site, ROD or NGI manager) COD performs site suspension. |
NB : Theoretically, the whole process could be covered in a 2 week period. This is in line with suspension procedure proposed for not replying to low availability/reliability figures. Most often a site solves the problem well before operators need to escalate issue to COD or the site is suspended on the spot for security reasons.
Comments (not part of the procedure):
MR: The below refers to inability to solve the issue, not reply. We do not have to chase sites for not solving the problem as if they cannot solve the problem it affects the availability - low availability will be tracked in other way.
NB: After the first 3 days, at the 2nd escalation step, if the site has not solved its problem, ROD should suggest to the site to declare downtime until they solve the problem and the NGI should be notified. If they do not accept the downtime then COD will proceed with the regular escalation procedure at the agreed deadlines.