EGI Infrastructure operations oversight

EGI.eu Operations Oversight Pages

EGI Grid Operations oversight of the e-Infrastructure is a co-ordination task for ensuring that GRID monitoring across EGI runs smoothly. This team communicates among the 3 groups - Operations and e-Infrastructure Oversight (OE); Operational Documentation (OD); and "Coordination of interoperations between NGIs and with other Grids".

The Operations oversight team works with the Tool Developers (and particularly the OTAG group), NGIs and their Operations Teams (ROD). There are regular phone meetings for the co-ordinators and others working in the tasks. The OE co-ordinators also organise face to face meetings for the ROD teams 3 to 4 times a year.

Co-ordinators:

Ron Trompert (Chair), Marcin Radecki, Luuk Uljee

Deputy:

Malgorzata Krakowian

Contact:

• There are 3 mailing lists used for different cases:
  • manager-central-operator-on-duty AT mailman.egi.eu - for COD managerial issues like suggesting changes in procedures, tools. COD managers are recipients of this list.
  • central-operator-on-duty AT mailman.egi.eu - for reporting COD day-to-day issues like problems with tools or Nagios tests. COD shifters are recipients of this list.
  • all-central-operator-on-duty AT mailman.egi.eu - for contacting all ROD teams in NGIs. Each ROD team is a recipient of this list.

---

COD offical web pages

• Operations Support Metrics
• Operations tests list
• Availability and reliability tests list

Procedures used in COD activity

In this section were collected all procedures in force for COD

• COD Operational Procedures - old EGEE wiki page with the most recent version of the procedure
• New NGI creation process coordination
• Availability and reliability monthly statistics procedure
• Operations Centre decommission process coordination
• Procedure for setting Nagios test an operations test
• COD escalation procedure

COD shifters daily work instructions

In this section are collected all work instructions containing detailed information specifying exactly what steps are to be followed to carry out an activity.

Dealing with the GGUS tickets assigned to COD

• COD shifter is obliged to check the current status of all GGUS tickets assigned to COD: Link to all GGUS tickets assigned to COD
• If the ticket is waiting for COD action then he/she should perform the action
• In case of a request for:
  • ROD certification - see New ROD team certification work instructions
  • New NGI creation - see New NGI creation process coordination
    • In case where COD is also the Integration Process Coordinator, COD is responsible for the whole procedure.
  • Operations Centre decommission see Operations Centre decommission process coordination
    • COD validates the request and removes ROD information from all-operators mailing list
  • Setting Nagios test to an operations test see Procedure for setting Nagios test to an operations test
    • COD is responsible for coordinating the whole process.
• If the shifter doesn't know what kind of action should be taken, he/she should contact COD managers

Availability/reliability reports

• Handling availability/reliability reports: Availability and reliability work instruction
  • AR reports metrics
  • COD escalation procedure

Issues on Operational portal dashboard

• Escalation for operational problems with ROD - work instruction

Handover

• At the end of the shift a handover should be submitted (send to COD) via Handover tool in the Operational Portal
  • Problems on the dashboard which will pass to next week: the ggus id of the ticket and when next escalation step should be taken
  • GGUS tickets assigned to COD: for each ticket its last status and the action taken by the shifter should be provided
  • Other issues: problems with tools etc.

Internal area

• Phone confs Meetings and Agenda

• List of typical problems with RODs

NOTE: all procedures should contain the following template: https://wiki.egi.eu/wiki/PDT:Procedure_Template

Procedures

Approved

• New NGI creation process coordination

• COD escalation procedure

• Operations Centre decommission process coordination

• Procedure for setting Nagios test an operations test

OTAG topics

Operational Portal: Dashboard

• RT tickets
• COD interaction with Dashboard team (draft)
• COD topic to be discussed on OTAG meeting
• Collection of dashboard requirements regarding COD work (draft)

GOC DB

• Collection of GOC DB requirements regarding COD work (draft)

Pages in draft state

• Improvements to Availability Calculation Procedure (draft)

• A/R fixing procedure (draft)