Alert.png The wiki is deprecated and due to be decommissioned by the end of September 2022.
The content is being migrated to other supports, new updates will be ignored and lost.
If needed you can get in touch with EGI SDIS team using operations @ egi.eu.

Difference between revisions of "Staged Rollout process"

From EGIWiki
Jump to navigation Jump to search
Line 12: Line 12:
= Repository Configuration  =
= Repository Configuration  =


'''This document assumes SL5 x86_64 PPAs verification, for other architectures the actual repositories to use may change'''  
'''This document assumes SL5 x86_64, for other architectures the actual repositories to use may change'''  


=== Base & EPEL  ===
=== Base & EPEL  ===


SL5 distribution is the official supported OS + EPEL repositories.<br>
SL5 distribution is the official supported OS + EPEL repositories.<br>  


If you are using a different OS (Centos, RHEL, other), please mention it in the final staged rollout repository<br>
If you are using a different OS (Centos, RHEL, other), please mention it in the final staged rollout repository<br>  


The yum repositories list to use are the following ones:  
The yum repositories list to use are the following ones:  
Line 79: Line 79:
  enabled=1
  enabled=1


Make sure you have installed yum-priorities, if not you should install it:<br>
Make sure you have installed yum-priorities, if not you should install it:<br>  
<pre>yum install yum-priorities
<pre>yum install yum-priorities
</pre>
</pre>  
=== Product Repository  ===
=== Product Repository  ===


The URL for the staged rollout of each "'''EMI&nbsp;produc'''t"&nbsp;repository can be found at the " <u>Custom Fields</u>" of the RT ticket within attribute RepositoryURL, note that you should clic<span style="text-decoration: underline;"><u></u></span>k "<u>Expand Release Info</u>" that contains the the URL to the repository and a pointer to the documentation. The easiest way of adding the repositoy is using the files available at the '''repofiles''' directory.  
The URL for the staged rollout of each "'''EMI&nbsp;produc'''t"&nbsp;repository can be found at the " <u>Custom Fields</u>" of the RT ticket within attribute RepositoryURL, note that you should clic<span style="text-decoration: underline;"><u></u></span>k "<u>Expand Release Info</u>" that contains the the URL to the repository and a pointer to the documentation. The easiest way of adding the repositoy is using the files available at the '''repofiles''' directory.  


Example config file for CREAM release, has the repository located at:<br>
Example config file for CREAM release, has the repository located at:<br>  


http://repository.egi.eu/sw/stagerollout/emi.cream.sl5.x86_64/1<br>
http://repository.egi.eu/sw/stagerollout/emi.cream.sl5.x86_64/1<br>  


and the repofile is available at:<br>
and the repofile is available at:<br>  


http://repository.egi.eu/sw/stagerollout/emi.cream.sl5.x86_64/1/repofiles/EMI.cream.sl5.x86_64.repo<br>
http://repository.egi.eu/sw/stagerollout/emi.cream.sl5.x86_64/1/repofiles/EMI.cream.sl5.x86_64.repo<br>  


Contents of this file is shown below:  
Contents of this file is shown below:  
Line 112: Line 112:
===== Extras Repository  =====
===== Extras Repository  =====


For verification of EMI 1.0 Release, an extra repository with compatibility packages is also needed for some products, at least WMS and UI. See information available at [https://rt.egi.eu/rt/Ticket/Display.html?id=2243 RT ticket 2243]. Repository file can be obtained from
For verification of EMI 1.0 Release, an extra repository with compatibility packages is also needed for some products, at least WMS and UI. See information available at [https://rt.egi.eu/rt/Ticket/Display.html?id=2243 RT ticket 2243]. Repository file can be obtained from  


http://repository.egi.eu/sw/stagerollout/emi.extras.sl5.x86_64/1/repofiles/EMI.extras.sl5.x86_64.repo.  
http://repository.egi.eu/sw/stagerollout/emi.extras.sl5.x86_64/1/repofiles/EMI.extras.sl5.x86_64.repo.  

Revision as of 16:10, 2 June 2011

Main EGI.eu operations services Support Documentation Tools Activities Performance Technology Catch-all Services Resource Allocation Security



New Software Release Workflow

The full description of the New Software Release Workflow (NSRW) includes all stages from software release by a Technology Provider to release for deployment in the EGI production infrastructure.

The remainder of this document describes the part of the workflow corresponding to the Staged Rollout phase.

Repository Configuration

This document assumes SL5 x86_64, for other architectures the actual repositories to use may change

Base & EPEL

SL5 distribution is the official supported OS + EPEL repositories.

If you are using a different OS (Centos, RHEL, other), please mention it in the final staged rollout repository

The yum repositories list to use are the following ones:

  • SL5 BASE and updates:
[sl-base]
name=SL 5 base
baseurl=http://ftp.scientificlinux.org/linux/scientific/55/$basearch/SL
	 http://ftp1.scientificlinux.org/linux/scientific/55/$basearch/SL
	 http://ftp2.scientificlinux.org/linux/scientific/55/$basearch/SL
	 ftp://ftp.scientificlinux.org/linux/scientific/55/$basearch/SL
#mirrorlist=ftp://ftp.scientificlinux.org/linux/scientific/mirrorlist/sl-base-55.txt
enabled=1
gpgcheck=0
# To use priorities you must have yum-priorities installed
priority=10
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-sl 
       file:///etc/pki/rpm-gpg/RPM-GPG-KEY-sl5 
       file:///etc/pki/rpm-gpg/RPM-GPG-KEY-csieh
       file:///etc/pki/rpm-gpg/RPM-GPG-KEY-dawson 
       file:///etc/pki/rpm-gpg/RPM-GPG-KEY-jpolok 
       file:///etc/pki/rpm-gpg/RPM-GPG-KEY-cern 
       file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-5
[sl-security]
name=SL 5 security updates
baseurl=http://ftp.scientificlinux.org/linux/scientific/55/$basearch/updates/security
        http://ftp1.scientificlinux.org/linux/scientific/55/$basearch/updates/security
        http://ftp2.scientificlinux.org/linux/scientific/55/$basearch/updates/security
        ftp://ftp.scientificlinux.org/linux/scientific/55/$basearch/updates/security
#mirrorlist=ftp://ftp.scientificlinux.org/linux/scientific/mirrorlist/sl-security-55.txt
enabled=1
gpgcheck=0
# To use priorities you must have yum-priorities installed
priority=10
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-sl
       file:///etc/pki/rpm-gpg/RPM-GPG-KEY-sl5 
       file:///etc/pki/rpm-gpg/RPM-GPG-KEY-csieh  
       file:///etc/pki/rpm-gpg/RPM-GPG-KEY-dawson 
       file:///etc/pki/rpm-gpg/RPM-GPG-KEY-jpolok 
       file:///etc/pki/rpm-gpg/RPM-GPG-KEY-cern
       file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-5
  • EPEL
[extras]
name=epel
mirrorlist=http://mirrors.fedoraproject.org/mirrorlist?repo=epel-5&arch=$basearch
protect=0
enabled=1
priority=20
  • EGI trust anchor
[EGI-trustanchors]
name=EGI-trustanchors
baseurl=http://repository.egi.eu/sw/production/cas/1/current/
gpgkey=http://repository.egi.eu/sw/production/cas/1/GPG-KEY-EUGridPMA-RPM-3
gpgcheck=1
enabled=1

Make sure you have installed yum-priorities, if not you should install it:

yum install yum-priorities

Product Repository

The URL for the staged rollout of each "EMI product" repository can be found at the " Custom Fields" of the RT ticket within attribute RepositoryURL, note that you should click "Expand Release Info" that contains the the URL to the repository and a pointer to the documentation. The easiest way of adding the repositoy is using the files available at the repofiles directory.

Example config file for CREAM release, has the repository located at:

http://repository.egi.eu/sw/stagerollout/emi.cream.sl5.x86_64/1

and the repofile is available at:

http://repository.egi.eu/sw/stagerollout/emi.cream.sl5.x86_64/1/repofiles/EMI.cream.sl5.x86_64.repo

Contents of this file is shown below:

# EGI Software Repository - REPO META (releaseId,repositoryId,repofileId) - (2206,239,166)

[EMI.cream.sl5.x86_64]
name=EMI.cream.sl5.x86_64
baseurl=http://repository.egi.eu/sw/stagerollout/emi.cream.sl5.x86_64/1/
enabled=1
protect=1

Issues

Package signing

GPG keys are not provided by repofiles and signature checks of packages cannot be used currently, a solution for this issue is being investigated. In the meantime, make sure to use gpgcheck=0 in the yum repo file.

Extras Repository

For verification of EMI 1.0 Release, an extra repository with compatibility packages is also needed for some products, at least WMS and UI. See information available at RT ticket 2243. Repository file can be obtained from

http://repository.egi.eu/sw/stagerollout/emi.extras.sl5.x86_64/1/repofiles/EMI.extras.sl5.x86_64.repo.

[EMI.extras.sl5.x86_64]
name=EMI.extras.sl5.x86_64
baseurl=http://repository.egi.eu/sw/stagerollout/emi.extras.sl5.x86_64/1/
enabled=1
protect=1

Staged Rollout Workflow

The staged rollout start when the "RolloutProgress" custom field of RT ticket in the queue sw-rel changes from "InVerification" to "StageRollOut", this means that the SW component as passed the verification process, i.e. was "ACCEPTED" in the verification phase.

It is assumed that the staged rollout repository is fixed for any given major release of any SW, this is publicly available in the repository and EGI wiki for Early Adopters. This state change triggers the following actions:

  1. The SW packages are moved from the InVerification repository to the StageRollOut repository.
  2. The sw-rel ticket creates a new child ticket in the staged-rollout queue. All the Staged Rollout process occurs in this queue/ticket.
    1. Advisable to change to the RT ticket tab "Jumbo".
    2. The "Owner" of the "staged-rollout" ticket is "(Nobody)".
    3. The "Status" of the ticket is "New".
    4. Change the ownership of the ticket to the "Staged Rollout Manager" responsible for that MW component: ARC, gLite, UNICORE, Globus, Operational Tools.
  3. (Staged Rollout Manager) The staged rollout manager is notified and takes the actions below. Advisable to change to the RT ticket tab "Jumbo".
    1. The staged-rollout ticket should contain the links to:
      1. Release notes, contains information about install/upgrade and configuration.
      2. Bugs or issues fixed.
      3. Documentation
      4. Staged rollout repository (needed by the EA teams).
    2. The status of the ticket is "Open".
  4. (Staged Rollout Manager) The queue staged-rollout has a custom field (dropped down) where the staged rollout managers select all the EA teams to assign the staged rollout test.
  5. (Staged Rollout Manager) When the "Save changes" button is pressed an automatic notification mail is sent to all EA . The mail notification should contain the links to release notes, installation/upgrade information, configuration, documentation and the repository URL. All EA teams are added to the "Administrative Cc" field.
  6. (EA teams) Each EA team has to acknowledge the reception of the notification within 1 working day. Either by replying to the mail sent by the RT or directly in the ticket with: <accept|reject> <NGI>-<Site-name><big</big>
  7. (Staged Rollout Manager) The staged rollout manager will check the ticket and if there are no EAs accepting the staged rollout test, it will pool the early-adopters-XXX.mailman.egi.eu mailing lists, and other (s)he see’s fit to get other EA sites.
  8. (EA teams) (NOTE this point needs more clarification) The EA team has the option to put the service node into downtime in the GOCDB, special beta tag. This tag may be set only during the staged rollout phase, if/when the component is release into production, this tag (or downtime), should be removed from the gocdb.
  9. (EA teams) The EA teams do the staged rollout: install/upgrade, configure, and some tests as they see fit.<big</big>
  10. (EA teams) If the EA finds problems or issues, either they are clarified within the ticket by the staged rollout managers and other EA teams, OR , a GGUS ticket should be opened. If a GGUS ticket is opened, this ticket will be assigned to the DMSU support unit, which will then route it to the technology providers as they see fit.
    1. (EA teams) A link to the GGUS ticket should be created in the staged rollout ticket.
  11. (EA teams) The service should ideally be exposed to production load/environment and users. This period lasts between 5 to 7 days, but may be extended depending on the cases or components under test.
  12. (EA teams) Each EA team should fill the staged rollout templates after the last point, and attach it to the staged-rollout ticket:
    1. (EA teams) The report should contain as much information as possible, specially the correctness of the release notes, test that have been preformed, and possible metrics when the service is exposed to production (like number of jobs per day, or number of transfers, what VOs are configured for that service, etc.)
    2. (EA teams) The name of the file should be:
      1. ea-<NGI>-<Site-name>-<MW stack>-<component>-<version>.EXT (doc, docx, odt)
  13. (Staged Rollout Manager) The staged rollout managers:
    1. Collect all reports and produce a summary report.
    2. Create a document in the EGI Doc server with a given ID, which will contain all reports for the staged rollout of that component, as well as the summary report.
    3. Insert a link in the RT ticket to the Dec server ID of the reports.
    4. Select from the custom field"Outcome" the outcome < ACCEPT | REJECT >.
    5. Set the ticket status to resolved.
  14. (Staged Rollout Manager) When the staged rollout child ticket is set to resolved the parent ticket is notified, and it gets the outcome and the doc server ID of the reports.

Resources

  • The Staged Rollout process is supported by the RT ticketing system (see tickets in the staged-rollout queue).
  • Report templates: each Early Adopter has to fill in a report (see the template on DocumentDB.

Naming conventions

  • The EA team names are SSO groups containing the members of each team:
    • ea-< NGI >-< Site-name >
  • The custom fields provided in the staged-rollout queue are:
    • Drops down box containing all EA teams, on the EGI SSO: possibility to select several teams, the button Save Changes notifies those teams and they will be added to the AdminCC field.
    • Outcome of the staged rollout, drop down box with: < ACCEPT | REJECT >.
  • The title of the ticket is of the form:
    • “Staged Rollout < SW stack-MajorVersion > < COMPONENT > < VERSION > < OS > < ARCH >”
    • Examples:
      • Staged Rollout CA 1.38
      • Staged Rollout EMI-1.0 SE-DPM_mysql 1.8.0-1 SL5 x86_64
SW Stack EA SSO group Staged Rollout Managers
gLite (+dCache) https://www.egi.eu/sso/groupView/early-adopters-glite Mario David, Michaela Barth
ARC https://www.egi.eu/sso/groupView/early-adopters-arc Christian Ulrik Søttrup, Sergio Maffioletti
UNICORE https://www.egi.eu/sso/groupView/early-adopters-unicore
Globus https://www.egi.eu/sso/groupView/early-adopters-globus
Operational Tools https://www.egi.eu/sso/groupView/early-adopters-opstools Daniele Cesini



  • The name of the file should be:
    • Filename: ea-< NGI >-< Site-name >-< MW stack >-< component >-< version >.doc(odt)
  • The staged rollout manager will provide a summary and the name of the file is:
    • Filename of summary: summary-< MW stack >-< component >-< version >.doc(odt)

All reports and the summary will be in: https://documents.egi.eu with a given ID, that will be referenced in the respective rt ticket.

  • The description in the document database should have the following naming convention:
    • Staged rollout < MW stack > < component > < version >

Back Staged-Rollout