Instructions for Production Tools teams

From EGIWiki
Jump to: navigation, search
Main EGI.eu operations services Support Documentation Tools Activities Performance Technology Catch-all Services Resource Allocation Security


Tools menu: Main page Instructions for developers Accounting Repository AppDB GGUS Message brokers ARGO Operations Portal GOCDB
AAI Proxy Metric Portal Licenses OTAGs Accounting Portal e-GRANT Perun Gstat LToS



Contents


Initial activities

Goal: Introducing new Production tool.
Every new operations tool developed needs to contact EGI Operations (operations@egi.eu) and provide following data:

  1. Name of the tool
  2. Support email
  3. Name and contact details of Tool Team leader
  4. Description of the tool - purpose
  5. License

With EGI Operations team help following steps needs to be performed

  1. Creation of url in egi.eu domain
  2. Registration in GOC DB under EGI.eu NGI
  3. Create category in "Requirements" queue in EGI RT tracker and RT dashboard to receive and handle service requests 
  4. Create GGUS Support Unit to receive and handle incidents (define level of quality of support - default: Medium)
  5. Negotiate and sign OLA with EGI.eu
  6. Acknowledge EGI
  7. Create wiki entry for the tool with relevant information
    • Tool name
    • Category, Short description of the tool
    • URL
    • Contact
    • GGUS Support Unit
    • GOC DB entry
    • link to related OLA
    • link to documentation
    • license
    • provider name
    • link to source code
    • Change management: 
      • link to EGI RT dashboard for the tool
      • (if applicable) internal bug/task tracking facilities
      • (if applicable) link to OTAG team
    • Release and Deployment management: 
      • Release schedule
      • Release notes
      • (if applicable) Roadmap
      • URL of test instance
  8. Subscribe to Tool-admins mailing list
  9. Get from EGI access to notification@egi.eu email


Ongoing activities

Change management

Goal: To ensure changes are planned, approved, implemented and reviewed in a controlled manner to avoid adverse impact of changes to services or the customers receiving services.

CM.png

Record

  1. EGI recording system is EGI RT 
  2. EGI user are instructed to submit changes requests in EGI RT.
  3. Team can use internal tracker but keeping consistence between EGI RT and internal tracker is responsibility of the development team.
  4. If tool has other customers than EGI, development team is responsible to inform EGI about submitted change request from other customer.  
  5. Minimum set of statuses of entry
    • New - newly recorded in system
    • Accepted - accepted by OTAG/OMB
    • Rejected - rejected by OTAG/OMB
    • In progress/Open - Development team is working on the request
    • Resolved - released, closing the ticket being responsibility of the development team
    • Stalled - on hold

Classify

  1. All change requests should be classified in consistent manner. Classification in EGI RT is based on Tool the request is related to.
  2. Tool providers should define list of standard change requests (a Change that is recurrent, well known, has been proceduralized to follow a pre-defined, relatively risk-free path, and is the accepted response to a specific requirement or set of circumstances, where authority is effectively given in advance of implementation.). Standard change request doesn't need approval to be implemented.
  3. Emergency change for Operational tools are the highest priority change related to security vulnerability and can be implemented without approval but will be subject of post-review.

Assess and Approve

  1. Each change request should be commented by development team with assessment of work needed to implement change.
  2. Every change which is not a standard change or emergency change should be assess (prioritize)and approved both internally, among the PTs and with OMB.
  3. Where needed dedicated Operations Tool Advisory Group can be set up. The OTAG mandate is to help developers in requirement prioritization and releasing process of operational tools. OTAG provide forums to discuss the tools evolution that meet the expressed needs of the EGI community. It has representation from the all end users groups depending on the tool.
  4. Minimum set of value for prioritization:
    • None (in RT - 0)
    • Low (in RT - 1)
    • Normal (in RT - 2)
    • High (in RT - 3)
    • Immediate/emergency (in RT - 4)

Implement, release and deploy

Phase which take place in Release and Deployment process (see below).

Review

  1. Every release is a subject of post-implementation review. Within one week customers are providing feedback answering to following questions:
    • Were release notes and documentation sufficient?
    • Is the tool working as expected?

Release and deployment management

Goal: To bundle changes to release, so that these changes can be tested and deployed to the production environment together.

RnDM.png

Plan Release

  1. Release schedules should be defined - the frequency of releases (eg. every 1-2 month).
  2. Scope of every release should be defined.
  3. Release should be planned in a away that OMB can be informed in at least one week time in advance.
    • all planned releases should be presented during the monthly OMB meeting prior to their release to production
    • emergency releases should be presented at the first available OMB meeting, post-release.

Build Release

  1. Due to the open-source nature of the developed software, each tool should have/use a publicly available code repository, like SVN, CVS, GIT
  2. All new releases should correspond to a new tag in the SVC
    • Ideally, if intended to be part of the UMD distribution, all tools should be packaged in an OS native format (rpm for RH family, debs Debian)
  3. Distribution
    • source code through SVC system of choice (tar.gz) or as binaries through EGI AppDB, GITHUB repositories
      • if intended to be part of UMD - new releases should be distributed as binaries through the UMD repository

Test Release

  1. Testing of the software, prior to its release, is a responsibility of the development teams.
  2. Where dedicated OTAG exists it can be involved in the testing activity.
  3. Major or particularly important releases should pass acceptance testing performed by customer representatives
  4. When the development phase of a new release is completed an announce should be send to all the PTs and to all the actors that should be involved in the release testing (OTAGs). The announcement should contain:
    • release notes, containing changelog, installation & configuration steps to apply the update, any known issues
    • documentation links
    • detailed test plan
    • all the information needed by the conformance criteria set by the SA2 activity for the software providers.
    • the expected release date and the kind of testing will depend on each specific release and on its importance.
  5. If a test fails a report will be produced and the release sent back to development to restart the cycle. Tests will include a documentation review and a documentation update if needed. The test phase can be performed internally to the development team if no other tools or services are affected.

Document

  1. Development team is responsible for creation and maintenance of documentations, instructions and manuals related to the tool in collaboration with EGI Operations team.
  2. Before each release documentation should be checked and updated when/where needed.

Inform

  1. Information about next release should be communicated
    • during OMB meeting (at least one week before release).
      • One slide information should be send to operations@egi.eu
    • through Monthly Operational broadcast
      • Content of the broadcast should be sent to operations-support@mailman.egi.eu

Deploy Release

  1. For changes of high impact and high risk, the steps required to reverse an unsuccessful change or remedy any negative effects shall be defined.

Review Release

  1. Each release should be monitored for success or failure and the results shall be analysed internally.

Incident and Problem management

Goal: To restore normal/agreed service operations within the agreed time after the occurrence of an incident, and to investigate the root causes of (recurring) incidents in order to avoid future recurrence of incidents by resolving the underlying problem, or to ensure workarounds are available.

Incident and requests management

Support should be provided:

Problems management

  1. In case of recurring incidents which cannot be solved by development team a GGUS ticket should be created to "Operations"  Support Unit. EGI Operations team will coordinate investigation of  the root causes of (recurring) incidents in order to avoid future recurrence of incidents. 
    • Any existing GGUS tickets which may help investigation should be marked as a child to the created ticket.

Planned maintenance windows or interruptions

Planned maintenance should be

Unscheduled interruptions should be managed according to MAN04

Security incidents

All security incidents should be registered according to EGI_CSIRT:Incident_reporting

Monitoring

Development team is responsible for development, maintenance and support of the tool monitoring probes.

Availability and reliability threshold should be defined with EGI Operations team depending on criticality of the service.

Information Security management

Goal: to manage information security effectively through all activities performed to deliver and manage services, so that confidentiality, integrity and accessibility of relevant assets are preserved

The following rules for information security and data protection apply:
•    The Provider must define and abide by an information security and data protection policy related to the service being provided.
•    This must meet all requirements of any relevant EGI policies or procedures and also must be compliant with the relevant national legislation.

Customer relationship management

Goal: Establish and maintain a good relationship with customers receiving service

Development team need to interact with each other inside EGI activities, primarily with OMB, the main customer of the operational tools.

Interactions between Development teams are guaranteed through periodic phone conferences and face to face meetings. A dedicated mailing list is available as well.

Interactions with customers are guaranteed through periodic OMB meetings and (where needed) dedicated Operations Tool Advisory Group. The OTAG mandate is to help developers in requirement prioritization and releasing process of operational tools. OTAG provide forums to discuss the tools evolution that meet the expressed needs of the EGI community. It has representation from the all end users groups depending on the tool.




Personal tools
Namespaces
Variants
Actions
Navigation
Toolbox
Print/export