imported>Krakow |
|
| Line 1: |
Line 1: |
| {{Template: Op menubar}} | | {{Template: Op menubar}} {{Template:Doc_menubar}} |
| {{Template:Doc_menubar}} | |
|
| |
|
| [[Category:Operations Manuals]]
| | {{DeprecatedAndMovedTo|new_location=https://docs.egi.eu/providers/operations-manuals/man04_tool_intervention_management}} |
| {{TOC_right}}
| |
| {{Ops_procedures | |
| |Doc_title = Tool Intervention Management | |
| |Doc_link = [[MAN04|https://wiki.egi.eu/wiki/MAN04]]
| |
| |Version = 19 August 2014
| |
| |Policy_acronym = OMB
| |
| |Policy_name = Operations Management Board
| |
| |Contact_group = operations-support@mailman.egi.eu
| |
| |Doc_status = Approved
| |
| |Approval_date = 01 August 2011
| |
| |Procedure_statement = This manual provides information on how to manage central operational tool unscheduled downtimes.
| |
| }} | |
|
| |
|
| ----
| | [[Category:Operations_Manuals]] |
| | |
| = Tool Intervention Management =
| |
| | |
| The purpose of this document is to describe the intervention in case of unscheduled failure of central operational tool.
| |
| | |
| = Scope =
| |
| | |
| This manual only applies to unscheduled downtimes of central operational tools. The list of central operational tool is available [[Tools|here]].
| |
| | |
| Note: Scheduled downtimes are management according to existing procedures ([[MAN02|MAN02]]).
| |
| | |
| = Announcements =
| |
| | |
| All announcements should be sent with the Operations Portal Broadcast tool.
| |
| | |
| When using Operations Portal Broadcast tool the following groups should be included:
| |
| * LCG Rollout Mailing List
| |
| * Operators Mailing lists
| |
| * OSG Mailing list
| |
| * Tool Admins Mailing List
| |
| * WLCG Tier 1 contacts
| |
| * NGI managers
| |
| * VO managers
| |
| * VO users
| |
| * Site administrators
| |
| * Operation tools
| |
| | |
| '''Notice:''' Individual notification templates together with targets are predefined in Operations Portal Broadcast tool. Administrators are advised to use such predefined templates.
| |
| | |
| = Procedure =
| |
| | |
| In the following sections several relevant scenarios are covered.
| |
| | |
| == Case 1: short "undetected" downtime ==
| |
| | |
| '''Description:''' Service fails and recovers before administrator manages to react (e.g. short power or network outage).
| |
| | |
| '''Action:''' Administrator announces the failure by using the following template:
| |
| <pre>
| |
| Subject:
| |
| [SERVICE_NAME] unscheduled downtime
| |
| | |
| Message:
| |
| Dear all,
| |
|
| |
| [SERVICE_NAME] experienced unscheduled downtime between [START] and [END].
| |
| | |
| [DETAILED_FAILURE_DESCRIPTION]
| |
| | |
| Apologies for any inconvenience caused.
| |
|
| |
| Best Regards
| |
| [SERVICE_TEAM]
| |
| </pre>
| |
| | |
| == Case 2: long "detected" outage ==
| |
| | |
| Service fails and administrator detects the problem. The problem takes at least '''1h''' time to recover. In the sections below individual situations are described.
| |
| | |
| === 1. Outage ===
| |
| | |
| '''Description:''' Service failure is detected.
| |
| | |
| '''Action:''' Administrator announces the failure by using the following template:
| |
| <pre>
| |
| Subject:
| |
| [SERVICE_NAME] outage
| |
| | |
| Message:
| |
| Dear all,
| |
|
| |
| [SERVICE_NAME] is experiencing unscheduled downtime.
| |
|
| |
| [ADDITIONAL_INFO]
| |
|
| |
| Apologies for any inconvenience caused.
| |
|
| |
| Best Regards
| |
| [SERVICE_TEAM]
| |
| </pre>
| |
| | |
| === 2. Extended downtime ===
| |
| | |
| '''Description:''' Service recovery is delayed. Update should be sent at least every '''24h'''.
| |
| | |
| '''Action:''' The administrator announces that recovery is taking longer by using the following template:
| |
| <pre>
| |
| Subject:
| |
| [SERVICE_NAME] extended outage
| |
| | |
| Message:
| |
| Dear all,
| |
|
| |
| Outage of [SERVICE_NAME] is extended.
| |
| | |
| [ADDITIONAL_INFO]
| |
|
| |
| Apologies for any inconvenience caused.
| |
|
| |
| Best Regards
| |
| [SERVICE_TEAM]
| |
| </pre>
| |
| | |
| '''Note:''' In this template [ADDITIONAL_INFO] should indicate the estimated time of recovery.
| |
| | |
| === 3. Recovery ===
| |
| | |
| '''Description:''' Service is recovered.
| |
| | |
| '''Actions:''' At the time of recovery the administrator announces the recovery by using the following template:
| |
| <pre>
| |
| Subject:
| |
| [SERVICE_NAME] recovery
| |
| | |
| Message:
| |
| Dear all,
| |
|
| |
| [SERVICE_NAME] is back online.
| |
| | |
| [ADDITIONAL_INFO]
| |
|
| |
| Best Regards
| |
| [SERVICE_TEAM]
| |
| </pre>
| |
| | |
| === 4. Post mortem analysis ===
| |
| | |
| '''Description:''' Service failure required further time to investigate the source of the problem. This action is required only if the post mortem analysis is needed.
| |
| | |
| '''Actions:''' The administrator announces the post mortem analysis of failure by using the following notification template:
| |
| <pre>
| |
| Subject:
| |
| [SERVICE_NAME] outage analysis
| |
| | |
| Message:
| |
| Dear all,
| |
|
| |
| [SERVICE_NAME] experienced unscheduled downtime between [START] and [END].
| |
| | |
| [DETAILED_FAILURE_DESCRIPTION]
| |
|
| |
| Best Regards
| |
| [SERVICE_TEAM]
| |
| </pre>
| |
| | |
| == Revision History ==
| |
| | |
| {| border="3"
| |
| |-
| |
| ! Version
| |
| ! Authors
| |
| ! Date
| |
| ! Comments
| |
| |-
| |
| |
| |
| | M. Krakowian
| |
| | 19 August 2014
| |
| | Change contact group -> Operations support
| |
| |}
| |
The wiki is deprecated and due to be decommissioned by the end of September 2022.