Alert.png The wiki is deprecated and due to be decommissioned by the end of September 2022.
The content is being migrated to other supports, new updates will be ignored and lost.
If needed you can get in touch with EGI SDIS team using operations @ egi.eu.

PROC02 Operations Centre creation

From EGIWiki
Revision as of 09:52, 28 June 2011 by Mkrakowi (talk | contribs)
Jump to navigation Jump to search
Main EGI.eu operations services Support Documentation Tools Activities Performance Technology Catch-all Services Resource Allocation Security


Documentation menu: Home Manuals Procedures Training Other Contact For: VO managers Administrators


Operations Centre creation

  • Title: Operations Centre creation
  • Document link: https://wiki.egi.eu/wiki/PROC02
  • Last modified: 14:39, 17 March 2011 (UTC)
  • Version: 2.02
  • Policy Group Acronym: OMB/COD
  • Policy Group Name: Operations Management Board/Central Operator on Duty
  • Contact Person: Małgorzata Krakowian, Marcin Radecki
  • Document Status: APPROVED
  • Approved Date: 26.10.2010
  • Procedure Statement: The purpose of this document is to clearly describe the actions and the relative steps to be undertaken for integrating a Operations Centre into the EGI operational structure.

Overview

The purpose of this document is to clearly describe actions and relative steps to be undertaken for integrating an Operations Centre into the EGI operational structure.

Definitions

  • The Integration Process Coordinator (IPC) is the entity responsible for integrating a new Operations Centre within EGI. The IPC can be the EGEE parent ROC of the Operations Centre (if still operational), or COD.

Validation

First step for validating an Operations Centre is to be politically approved as formed from an official partner of the EGI infrastructure.

After this step, the Operations Centre needs to be technically validated. The Central Operator on Duty team - in charge of EGI oversight - is responsible for performing this validation. However, if the Operations Centre is a part of an ex EGEE-ROC structure which is still operating, and which is willing to perform the validation, then the ROC can perform the validation itself.

Political Validation

CASE 1. If an Operations Centre is already represented within the EGI Council and is ready to move from an EGEE ROC to an operational Operations Centre, we recommend that the Operations Centre political representative within the EGI Council notifies the EGI Chief Operations Officer that the respective Operations Centre is entering its validation cycle. At this point, technical validation can start.

CASE 2. If an Operations Centre is not represented within the EGI Council, and it is willing to be represented there, the Operations Centre needs to submit a request for admission to the Council. After the Operations Centre has been accepted by the Council, CASE 1 applies.

CASE 3. If a new Operations Centre is not represented within the EGI Council and is not interested in being part of it, but would still like to be a consumer of the EGI Global Services, then an MoU must be established with EGI. Once an MoU is in place technical validation can start.

Technical Validation

  1. The Operations Centre Operations manager(s) sends an email to the Chief Operations Officer (COO) that the Council was informed about the creation of a new Operations Centre, and (if applicable) also that the Council has approved it. The Operations Centre Operations manager(s) should also indicate the IPC responsible for the validation in the email.
  1. The Chief Operations Officer opens a GGUS ticket to the IPC to start the validation process.

Start of the integration

  • The integration of a new Operations Centre starts when the COO opens an Operations Centre validation ticket to the IPC (via GGUS).
  • Once the COO ticket is filed, the IPC can start the validation process. In order to trigger the actions described in this document the IPC creates a set of new child tickets that are assigned to the individual partners that are responsible for the various validation steps. Thereby, the integration process should be as transparent as possible to all parties involved. The required actions are described below.


An example/template for the Operations Centre creation ticket is provided here:

Subject: Creation of <Operations Centre name>


Required information for the creation of the Operations Centre:
Management mailing list : management@xxx.org
Operations Centre Operations manager contact data : Person Surname (email) +phone contact,
Deputy: Person Surname (email) +phone contact,

Operations Centre security officer contact data : Person Surname (email) +phone contact,
Operations Centre security mailing list : abuse@xxx.org

ROD team mailing list : ngi-support@xxx.org
Operations Centre nagios monitoring system details : https://mon-ngi.xxx.org/nagios
Mailing list for GGUS tickets if using GGUS directly : xxxticket@xxx.org

The FAQ document for the Operations Centre provided by the GGUS team is in the attached file (GGUS_1800_FAQ_for_NGI_xxx.pdf)
(see extra document provided)

Pre-requisites

Before opening an Operations Centre creation GGUS ticket, the Operations Centre should:

  1. Decide about the Operations Centre name. Name for European Operations Centres should start with "NGI_"
  2. Decide whether to use the Operations Centre's own help desk system or use GGUS directly. If the Operations Centre wants to set up their own system they need to provide an interface for interaction to GGUS with the local ticketing system and follow the recommendations available at https://gus.fzk.de/pages/ggus-docs/interfaces/docu_ggus_interfaces.php.
  3. Set the following contact points:
    1. Management mailing list
    2. Operations Centre operations manager contact data
    3. Operations Centre security officer contact data
    4. Operations Centre security mailing list
    5. ROD team mailing list (including people responsible for monitoring and supporting the Operations Centre infrastructure)
    6. Mailing list for GGUS tickets IF GGUS is used directly for the helpdesk system
  4. All certified Operations Centre sites need to be under Nagios monitoring. The Nagios monitoring infrastructure can be directly operated by the Operations Centre (see https://twiki.cern.ch/twiki/bin/view/EGEE/GridMonitoringNcgYaim and https://twiki.cern.ch/twiki/bin/view/EGEE/ValidateROCNagios), or alternative by a third party Operations Centre.
  5. Fill the FAQ document for the Operations Centre. The template is provided by the GGUS team: https://gus.fzk.de/pages/ggus-docs/DOC/1800_FAQ_for_TEMPLATE.docx
  6. Staff in the Operations Centre that should be granted a management role (manager, deputy manager, security officer) should first register a user account in the GOCDB. The user registration procedure is described in the GOCDB user documentation at https://twiki.cern.ch/twiki/bin/view/EMI/Glite-APELInstallation, section 4.1.1
  7. Staff in Operations Centre is familiar with Operational Procedures
  8. People who are responsible for operations should be subscribed to following mailing lists:
    1. NGI manager:
      noc-managers [at] mailman.egi.eu - registration through https://mailman.egi.eu/mailman/listinfo
    2. ROD team:
      All-operator-on-duty [at] mailman.egi.eu - list which integrate all NGI ROD mailing list. NGI ROD mailing list will be add as a result of ROD certification procedure
    3. Dashboard administrator:
      ops-portal [at] mailman.egi.eu - registration through https://mailman.egi.eu/mailman/listinfo
      additionally administrator need to ask dashboard team (cic-information@in2p3.fr) to declare him in GOC DB.
    4. Nagios administrator:
      tool-admins@mailman.egi.eu - registration through https://mailman.egi.eu/mailman/listinfo
    5. Regional Helpdesk administrator:
      ggus-if-devs [at] cern.ch - mailing list designed for coordination of changes in the interface
    6. Security staff:
    7. Site administrators:
      LCG-ROLLOUT [at] jiscmail.ac.uk - list gathers all site admins and is designed for technical discussions - membership not mandatory but useful
      Subscription is possible through https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=LCG-<wbr></wbr>ROLLOUT&A=1

Creation steps

Some steps of the process can be done in parallel as they are independent, so all steps grouped within the same task number can be performed concurrently (several different child tickets will be created in order to speed up the process). The general idea is that these tickets must be closed before being able to move on to the next step.

Click to see larger picture.


Validation steps:

Step Substep Action on Action
1 IPC Verification of the validity of the request (were all needed data provided?)
2 IPC Create child tickets in the order given as follows:
1 GOCDB Creation of a new Operations Centre entry in the GOCDB (with no site attached).

Include the following into the GOCDB ticket:

  • Operations Centre name: <Operations Centre name>
  • Operations Centre management mailing list: foo@bar.org
  • Operations Centre security mailing list:
  • Operations Centre Operations manager:
  • Operations Centre security officer:
2 Operations Portal Enter the new Operations Centre in the Operations dashboard (also add ROD mailing list)
3 GGUS Create a new support unit in GGUS : <Operations Centre name>

NOTE: (attach the FAQ document for the Operations Centre to this ticket.)

4 COD Certification of new ROD team –

Procedure_to_handle_new_ROD_certification_GGUS_tickets

Include in the GGUS ticket for Operational Documentation:

  • Country
  • Operations Centre acronym
  • ROD email list
3 Dteam VO manager Create a branch/group for Operations Centre and assign DN of people who will be dteam VO representative for the Operations Centre, Operations Centre Group owner and Operations Centre Group manager.

Responsibilities and terminology

VO representative: A person that can approve or deny dteam VO membership requests. This person is selected by the applicant during the registration phase.

Group owner: A person that can approve or deny Group membership requests. In addition he can create subgroups within his Group.

Group manager: A person that can approve or deny Group membership requests.


Following data about each responsible person:

  • Name and Surname
  • DN
  • email

These responsibilities can be assigned to the same person(s).

How to assign the child ticket: assign the ticket to "VO support" after the selection of "dteam" in the concerned VO field

This step is not blocking the process and can be done in parallel. It is required to finish this step before closing parent ticket.

4 The newly created Operations Centre Include the NAGIOS host in the GOCDB as a 'National-Nagios' service.

(This step only applies to Operations Centres running an Operations Centre Nagios instance)

5 [If the Operations Centre was part of a ROC]

IPC

Configure the Operations Centre in the ROC Nagios instance
6 The newly created Operations Centre Final confirmation that the new Operations Centre can start the operations
7 GOCDB
  • [If the Operations Centre was part of a ROC]

GOCDB transfers related sites from the source ROC to the new Operations Centre structure.

NOTE: please indicate the sites moving across to the new NGI in the ticket.

The newly created Operations Centre
  • [If the Operations Centre wasn't part of a ROC]

Newly created Operations Centre can insert new sites

8 The newly created Operations Centre Inform managers of regional VOs to change the VO scope of their VOs from ROC (regional) to the relevant Operations Centre (national). This action require only confirmation from Operations Centre manager that information was passed.

Information which should be pass to VO managers: "The Vo scope can be changes by creating a ticket to CIC portal SU in GGUS."

9 OPTIONAL The newly created Operations Centre All sites should be reconfigured according to the instructions at:

https://wiki.egi.eu/wiki/MAN1_How_to_publish_Site_Information

change the old information from:

 GlueSiteOtherInfo: EGEE_ROC=XXX
 GlueSiteOtherInfo: GRID=EGEE

to:

 GlueSiteOtherInfo: EGI_NGI=<Operations Centre name>
 GlueSiteOtherInfo: GRID=EGI

This step can be performed at any time from this point.

10 The newly created Operations Centre Transfer all open operational tickets to the new Operations Centre in GGUS.
11 Nagios team Include the Operations Centre level Nagios in the central ops-monitor Nagios instance.

(This step only applies to Operations Centres running an Operations Centre Nagios instance)

12 IPC Validation process of the new Operations Centre Nagios, as described at the step 5.3 of:

https://twiki.cern.ch/twiki/bin/view/EGEE/ValidateROCNagios#Validation_Process.

(This step only applies to Operations Centres running an Operations Centre Nagios instance)

13 Nagios team Validation that sites/Operations Centre shown up correctly in Central DBs
14 [If the Operations Centre was part of a ROC]

Nagios team

Migrating alerts from ROC to Operations Centre Nagios instance.

(This step only applies to Operations Centres running an Operations Centre Nagios instance)

15 IPC Final checks by the IPC.

(Were all steps taken and finished properly? Close the parent ticket.)

16 The newly created Operations Centre Final checks should be verified and then the information that the Operations Centre is ready is broadcast by Operations Centre officials.

(This broadcast should be sent to VO managers and NOC/ROC managers)

See the template below for an indication of the message content.

Subject: <Operations Centre name> is operational

Dear All,

We would like to announce that <Operations Centre name> is now fully operational 
and that we have finished its integration procedure. All necessary operational 
teams and tools are established in our Operations Centre and we are ready for production. 
This Operations Centre is visible in all operational tools as <Operations Centre name> 
and is responsible for all <COUNTRY> sites.

Best regards,

Revision history

Version Authors Date Comments
2.04 Malgorzata Krakowian 2011-04-1 Step concerning SAMAP tool was removed due to tool decommission
2.03 Gonçalo Borges 2011-03-31 Operations Centre creation process visualization
2.02 T. Ferrari 2011-03-17 Assignment of procedure number, update of title "Operations Centre creation coordination procedure" to "Operations Centre Creation", small editorial improvements
2.02 Małgorzata Krakowian 2011-01-27 Name change from NGI to Operations Centre
2.01 Dimitris Zilaskos 2010-12-06 Clarification concerning dteam VO branch creation step
2.00 Małgorzata Krakowian, M. Radecki 2010-10-26 Approved by OMB
1.00 Małgorzata Krakowian, M. Radecki, V.Hansper et alt. 2010-08-17 Approved by OMB