Alert.png The wiki is deprecated and due to be decommissioned by the end of September 2022.
The content is being migrated to other supports, new updates will be ignored and lost.
If needed you can get in touch with EGI SDIS team using operations @ egi.eu.

Difference between revisions of "EGI Infrastructure operations oversight"

From EGIWiki
Jump to navigation Jump to search
(Document management of Asana tasks and weekly summary)
Line 42: Line 42:
* Checking status of and managing tickets within [https://app.asana.com/0/650884401877956/board Ops support Asana board]
* Checking status of and managing tickets within [https://app.asana.com/0/650884401877956/board Ops support Asana board]
** When a ticket is being handled it should be moved to the '''In progress''' column, and once finished marked as '''completed''' and moved to the '''postmortem''' column so that it can be reviewed at next EGI Operations team meeting, once reviewed it should be moved to the the '''Done''' column.
** When a ticket is being handled it should be moved to the '''In progress''' column, and once finished marked as '''completed''' and moved to the '''postmortem''' column so that it can be reviewed at next EGI Operations team meeting, once reviewed it should be moved to the the '''Done''' column.
* At the end of each week the OD should record the status of the week in the [https://app.asana.com/0/650884401877956/progress progress section] of [https://app.asana.com/0/650884401877956/board Ops support Asana board].
* At the end of each week the OD should record the status of the week in the [https://app.asana.com/0/650884401877956/progress progress section] of [https://app.asana.com/0/650884401877956/board Ops support Asana board], (set it '''on track''', '''at risk''' or '''delayed'''.
** Once done the '''Weekly summary for next OD''' recurrent task should be closed and the next generated task should be added to the next OD.
** Once done the '''Weekly summary for next OD''' recurrent task should be closed and the next generated task should be added to the next OD.



Revision as of 17:24, 5 August 2019

Main EGI.eu operations services Support Documentation Tools Activities Performance Technology Catch-all Services Resource Allocation Security


EGI Infrastructure Operations Oversight menu: Home EGI.eu Operations Team Regional Operators (ROD) 



Introduction

EGI Infrastructure operations oversight activity is provided by:

  • EGI Foundation Operations team
  • Regional Operators on Duty (ROD) teams

Oversight activity over the NGI infrastructures is needed for detecting problems, coordinating the diagnosis, and monitoring the problems during the entire lifecycle until resolution. Oversight of the NGI is based on monitoring of status of services operated by sites, opening of tickets and their follow up for problem resolution. EGI.org supports and actively controls the overall status of services and sites, opening of tickets for requesting problem fixing, and tackling of residual problems not successfully distributed to NGI’s.

EGI Foundation Operations team

EGI.eu Operations team is the central team responsible for EGI Production Infrastructure. It is also responsible to provide:

  1. Coordination of activities with the Operations Management Board and the User Community Board.
  2. Central Technical Support to site administrators, NGI operators and new user communities. This includes
    • technical support to the EGI Foundation operations activities
    • technical support to ROD teams through target training activities
    • coordination of technical working groups
    • technical support to new resource centres in their certification phase when requested by the Operations Centre because of lack of sufficient local expertise
    • certification and technical support for new infrastructures being integrated by providing assistance and training about EGI operations services, policies and procedures, and developing documentation as needed
  3. Resource Allocation
    • defining service management processes for resource allocation and other EGI.eu operations services
    • training, communicating, adapting, enforcing these at an NGI level
    • defining requirements for the operations tools that generate from the provisioning of these new services

EGI Foundation Operator on Duty (OD)

EGI Foundation Operator on Duty (OD) is a person in the central EGI Foundation Operations team primarily responsible for responding to tickets. This duty is rotated among the team using a rota and ensures that everyone in the team is given hands-on experience dealing with tickets and coordinating operations activities. OD duties include:

  • ensuring a timely response to emails sent to: operations@egi.eu
  • verifying the status of GGUS tickets that are marked as URGENT or have not been attended to in a timely manner. Useful filters are: tickets opened against Operations SU and open top priority tickets in the EGI scope and also open very urgent tickets in the EGI scope.
    • Note that the target initial response times for these are 1 day; for lower priorities it is 5 days.
    • Note that to be able to edit tickets, you need "GGUS supporter" role. See GGUS registration
  • (NEW) ensuring that there are no tickets created more than 3 years ago in the system; those tickets are very likely unattended and need to be addressed properly; use this search to get very old tickets ordered by creation date (oldest first)
    • in ISRM process there two metrics (one for incidents older than 1 year, and one for service requests older than 3 years) that ideally should be zero. They are collected every 4 months and a report listing the tickets still in open status is created (look in the reports section of ISRM page): follow-up the tickets in the list.
  • check the ROD Dashboard on the Operations Portal especially for long-standing critical alarms, and checking with NGIs of their status as necessary
  • check for new tickets in the RT Change Management queue, especially changes that need the CAB to be convened urgently, and if so, inform the CHM Manager
  • Manage Security Vulnerability Handling tickets following WI07_Security_Vulnerability_handling and specifically checking the vulnerability-handling queue on rt
  • Checking status of and managing tickets within Ops support Asana board
    • When a ticket is being handled it should be moved to the In progress column, and once finished marked as completed and moved to the postmortem column so that it can be reviewed at next EGI Operations team meeting, once reviewed it should be moved to the the Done column.
  • At the end of each week the OD should record the status of the week in the progress section of Ops support Asana board, (set it on track, at risk or delayed.
    • Once done the Weekly summary for next OD recurrent task should be closed and the next generated task should be added to the next OD.

Regional Operators on Duty (ROD)

Regional Operators on Duty (ROD) is a team responsible for solving problems on the infrastructure within NGI according to agreed procedures. They ensure that problems are properly recorded and progress according to specified time lines. They ensure that necessary information is available to all parties. The team is provided by each NGI and requires procedural knowledge on the process (rather than technical skills) for their work. Depending on how an NGI is organized there might be a number of members in the ROD team who work on duty roster (shifts on a daily or weekly basis), or there may be one person working as ROD on a daily basis and a few deputies who take over the responsibilities when necessary. This latter model is generally more suitable for small NGIs.