Operations and Operations Support

Main

EGI.eu operations services

Support

Documentation

Tools

Activities

Performance

Technology

Catch-all Services

Resource Allocation

Security

EGI Infrastructure Operations Oversight menu:

Home •

EGI.eu Operations Team •

Regional Operators (ROD)

Introduction

COD team is a small team responsible for coordination of RODs, provided on a global layer. COD represents the whole ROD structure in terms of technical requirements for operations tools as well as on political level.

The purpose of this page is to collect all materials needed by COD team to perform the Grid operations oversight activities.

People and contact

COD team is formed from Dutch and Polish team and includes COD managers (people responsible for managerial issues) and COD shifters (people performing day-to-day COD work)

COD managers: 

Ron Trompert (Chair), Marcin Radecki, Luuk Uljee, Tadeusz Szymocha, Magda Szopa

COD shifters: 

Tadeusz Szymocha, Magda Szopa, Ron Trompert, Luuk Uljee, Maarten van Ingen, Ernst Pijper, Alexander Verkooijen

People behind the names

There are 2 mailing lists used for different cases:

• manager-central-operator-on-duty AT mailman.egi.eu - for COD managerial issues like suggesting changes in procedures, tools. COD managers are recipients of this list.
• central-operator-on-duty AT mailman.egi.eu - for reporting COD day-to-day issues like problems with tools or Nagios tests. COD shifters are recipients of this list.

COD Duties

• COD managers
  • representing RODs/COD in OTAG, OMB and Operations meetings - collecting requirements and improvements proposals from RODs concerning operations tools and procedures
  • suspending Resource Centres in case of operational issues
  • taking part in OLA task force
  • writing new procedures - in case of need COD is taking part in procedures creation process
  • preparing ROD newsletters - informing RODs about recent and upcoming developments related to Grid Oversight
  • preparing ROD metrics reports - providing an overview of operations support process in grid infrastructure.
• COD shifters
  • escalation of operational problems with RODs
  • dealing with GGUS tickets assigned to COD
  • process coordination of:
    • creation and decommission of Operations Centre
    • setting a Nagios test to an operations test
    • getting explanations for low availability and reliability metrics

COD shifters work instructions

In this section are collected all work instructions containing detailed information specifying exactly what steps are to be followed to carry out an activity.

Action	Description	Related procedures
GGUS tickets assigned to COD	COD shifter is obliged to check the current status of all GGUS tickets assigned to COD see Link to all GGUS tickets assigned to COD If the ticket is waiting for COD action then he/she should perform the action In case of a request for: ROD certification see New ROD team certification work instructions Creation of a new NGI see Creation of a new Operations Centre process coordination see work instruction In case where COD is also the Integration Process Coordinator, COD is responsible for the whole procedure. Operations Centre decommission see Operations Centre decommission process coordination COD validates the request and removes ROD information from all-operators mailing list Setting a Nagios test to an operations test see Procedure for setting a Nagios test to an operations test COD is responsible for coordinating the whole process. If the shifter doesn't know what kind of action should be taken, he/she should contact COD managers	Creation of a new Operations Centre process coordination Operations Centre decommission process coordination Procedure for setting Nagios test an operations test
Availability/reliability reports	Handling availability/reliability reports: Availability and reliability work instruction AR reports metrics	COD escalation procedure Availability and reliability monthly statistics procedure
Operational portal dashboard issues	COD dashboard link	COD escalation procedure
Handover	COD dashboard link At the end of the shift a handover should be submitted (send to COD) via Handover tool in the Operational Portal Problems on the dashboard which will pass to next week: the ggus id of the ticket and when next escalation step should be taken GGUS tickets assigned to COD: for each ticket its last status and the action taken by the shifter should be provided Other issues: problems with tools etc.

Work Instructions

• New ROD team certification work instructions
• New Opertions Centre creation work instruction
• Availability and reliability work instruction
• Core services report work instruction
• Escalation procedure in case of unresponsive NGI

Events

• EGI indico page with COD meeting agendas.
• All open actions can be found from COD actions

Resources

• Document server: ROD newsletter
• Document server: Operations Support Metrics
• Operations Procedures
• Youtube channel

ROD and COD Performance

Definition of ROD performance index

Oct 2011 to date

• Please provide a link here

Definition of Operations Support metrics

May 2010-Sep 2011

• Operations Support metrics

Until April 2010

• EGEE-III Operations Support metrics

Nagios tests

• Operations tests list : list of Nagios probes generating alarms for visualization in the Operations Dashboard
• Availability and reliability tests list: list of Nagios probes whose results are used for Availability and Reliability computation

OTAG topics

Operational Portal: Dashboard

• RT tickets
• COD interactions with Dashboard team (draft)
• COD topics to be discussed on OTAG meeting

GOC DB

• Collection of GOC DB requirements regarding COD work (draft)

Pages in draft state

• Improvements to Availability Calculation Procedure (draft)

• A/R fixing procedure (draft)
  

• UNKNOWN issue
  
• UNKNOWN issue Internal
  
• ROD performance index
  
• Candidate Suspended Sites List
• Tickets > 30 days Work Instruction