Alert.png The wiki is deprecated and due to be decommissioned by the end of September 2022.
The content is being migrated to other supports, new updates will be ignored and lost.
If needed you can get in touch with EGI SDIS team using operations @ egi.eu.

Difference between revisions of "UMD:Provisioning:Proposal SR"

From EGIWiki
Jump to navigation Jump to search
Line 150: Line 150:


Given that EGI aims to have quarterly UMD releases, justifies why we have grouped the EMI releases on a quarterly basis. It can be seen that despite the large number of products, only a small fraction has been updated more than once in a given quarter. Note also that not all EMI products have been included into UMD. Note also that the number of products with 2 or more updates per quarter has decreased, as seen in the table below.  
Given that EGI aims to have quarterly UMD releases, justifies why we have grouped the EMI releases on a quarterly basis. It can be seen that despite the large number of products, only a small fraction has been updated more than once in a given quarter. Note also that not all EMI products have been included into UMD. Note also that the number of products with 2 or more updates per quarter has decreased, as seen in the table below.  
One further point is that EMI plans to start doing longer update cycles of 1 month. As such the number of updates per quarter would be reduced to three while the ''probability'' of having more then 1 update of a given product will decrease.
== Technical changes  ==
This section highlights some of the technical changes with respect to the current workflow, and is subject (as the rest of the proposal) to further comments and improvement and additions:
#The triggering of the process is performed by SA2 the same day that the technology provider announces their release. Instead of being triggered by the technology provider.
#The release.xml is produced automatically when SA2 triggers the process. Instead of being produced by the technology provider.
#The release.xml contains only the package dependencies of that update. Other packages in production will not be included in the release.xml. Instead of containing the full set of package dependencies.
#The release.xml is already split per ppa, since SA2 and SA1.3 has the knowledge (or can have this knowledge) of which components are internal dependencies and which ones are products. This would make the “bouncer” redundant.
#The full chain of the SW workflow is done in the RT queue ''sw-rel'', first the verification (as is done now) and after the staged rollout. The custom fields necessary for staged rollout would be mostly ''moved'' to the ''sw-rel'' queue. This simplifies the process and eases the communication between verifiers and early adopters. All issues, GGUS URLs and verification reports IDs would be directly available to the EAs.
#There are 2 new repositories that contain all new packages. These new repositories are static and well publicised especially to SA1. There is no splitting of products into different volatile repositories.
== Conclusions/Summary  ==
We are fully aware of the week points, but also think that if we manage to cover properly those week point the advantages far surpass the disadvantages, while having a much more flexible provisioning process, and also more adaptable to a future distribution in EPEL.


{| align="center" cellspacing="0" cellpadding="5" border="1"
{| align="center" cellspacing="0" cellpadding="5" border="1"
|-
|-
! style="width: 10%;" | Quarter 1  
! style="width: 33%;" | Quarter 1  
! style="width: 25%;" | Quarter 2  
! style="width: 33%;" | Quarter 2  
! style="width: 25%;" | Quarter 3
! style="width: 33%;" | Quarter 3
|-
|-
| CREAM – 3 updates  
| CREAM – 3 updates  
Line 179: Line 162:
|-
|-
| ARGUS – 2 updates  
| ARGUS – 2 updates  
| ARC Clients – 2 updates
| ARC Clients – 2 updates  
|
|  
|-
|-
| LB – 2 updates  
| LB – 2 updates  
Line 198: Line 181:
|  
|  
|}
|}
One further point is that EMI plans to start doing longer update cycles of 1 month. As such the number of updates per quarter would be reduced to three while the ''probability'' of having more then 1 update of a given product will decrease.
== Technical changes  ==
This section highlights some of the technical changes with respect to the current workflow, and is subject (as the rest of the proposal) to further comments and improvement and additions:
#The triggering of the process is performed by SA2 the same day that the technology provider announces their release. Instead of being triggered by the technology provider.
#The release.xml is produced automatically when SA2 triggers the process. Instead of being produced by the technology provider.
#The release.xml contains only the package dependencies of that update. Other packages in production will not be included in the release.xml. Instead of containing the full set of package dependencies.
#The release.xml is already split per ppa, since SA2 and SA1.3 has the knowledge (or can have this knowledge) of which components are internal dependencies and which ones are products. This would make the “bouncer” redundant.
#The full chain of the SW workflow is done in the RT queue ''sw-rel'', first the verification (as is done now) and after the staged rollout. The custom fields necessary for staged rollout would be mostly ''moved'' to the ''sw-rel'' queue. This simplifies the process and eases the communication between verifiers and early adopters. All issues, GGUS URLs and verification reports IDs would be directly available to the EAs.
#There are 2 new repositories that contain all new packages. These new repositories are static and well publicised especially to SA1. There is no splitting of products into different volatile repositories.
== Conclusions/Summary  ==
We are fully aware of the week points, but also think that if we manage to cover properly those week point the advantages far surpass the disadvantages, while having a much more flexible provisioning process, and also more adaptable to a future distribution in EPEL.

Revision as of 16:06, 9 February 2012

EGI Software Provisioning Process: New Process Proposal

Introduction

The SW provisioning to EGI through UMD is described in MS409 (TSA1.3) and MS508 (TSA2.3). This has been technically implemented and proven to work during the year 2011, where EMI1 products and it’s updates, IGE1 products and it’s updates, have gone through the process and included in UMD. Also SAM/Nagios updates and IGTF CA releases undergo the same process and are available in the EGI repositories.

This document will summarize very briefly the current SW provisioning workflow while a detailed description can be found in both MS409 and MS508. We will propose changes to this process, where the reasons will be given below, and the points of change with respect to the current process will be highlighted.

The proposal described in this document should be presented to the SA2 and after to the EGI Operations Management Board (OMB) for gathering comments and changes and after for voting. If approved by the OMB, the aim is to implement this new SW provisioning process for the EMI 2 release by the end of April 2012.

Current SW provisioning workflow

Currently the technology providers submit a GGUS ticket with an xml file called “release.xml”, containing a full description of their release including the meta-packages that we call products and it's dependencies. This was thought as the way to have the dependency list needed to move around the products independently of each other, after that, each product is split per repository for the verification and staged rollout, and in the end it is merged again into the UMD production repositories, a “base” repository with the initial UMD release and an “updates” repositories for the subsequent updates.

Some assumptions, facts and issues with the current process are given below:

  • The process assumes that the technology provider produces the “release.xml”, and this is effort we ask external entities to do in order to trigger the process. EMI will not be providing the “release.xml” for EMI2 onwards.
  • The products and components are publicly available in the repositories of the technology providers the moment they do the release, while it may take some time to produce the “release.xml”, this is an initial delay that may take days as we experienced in the past.
  • After that, the process is quite rigid in that a first step of Verification will be performed by SA2.3. After successful verification it will pass to staged-rollout.
  • The previous point means that a given update from the technology provider will be included in a UMD at least 1 month later, moreover only the components that underwent the full cycle with success will be included in UMD. This should not change, and it may even be extended if the new proposal is approved (see next section).
  • As such there is a rather long time lag between the release of the technology provider and the release in UMD, and some operation managers (site administrators) and users have “commented” about this long time lag.
  • EGI/NGI resource infrastructure providers (sites) have been using the EMI repositories at least as much or more then the UMD repositories to download the middleware.

New proposal for the sw provisioning

We start by noting that EMI's future aims to distribute the MW through the EPEL repositories, and they are already heading that way. IGE is also distributing the Globus packages through the EPEL.

UMD repositories

In this proposal we suggest the following structure for the UMD repositories:

  • base repository: containing the packages of a major UMD release. Already exists in the current structure.
  • updates repository: containing all updates or minor releases of a given major UMD release. Several versions of the packages are accumulated here. Also already in place.
  • testing repository (New repository): contains all packages from the moment they are released by the technology providers. All packages would be fetched from the Technology Providers repositories, the moment they are available in the technology provider's repository. Verification and Staged Rollout would be preformed with packages in this repository. Products with successful verification and staged rollout are moved to the rc repository. This repository is static, known in advance and publicly available not only to the SA2.3 verifiers and Early Adopter sites but also to the production infrastructure. Several versions of the packages maybe accumulated here.
  • rc repository: Release Candidate repository will serve to cache packages/products that underwent successfully the verification and staged rollout steps. The packages in this repository form the release candidate for the next UMD release. The packages in this repository are moved to the base or updates repository at the time of the UMD release. This repository is static, known in advance and publicly available to the production infrastructure. Only one version of each package maybe in this repository at any given time. This repository is empty after each UMD release.
  • Packages that have proven to break the service and for which there is no possible workaround, should be removed from the testing or rc repositories.

This new structure does not split packages needed for a product (product dependencies) into different repositories during verification and staged rollout.

It also assumes that more than one version of a given product maybe in the testing repository, the distinction of which one is at a given stage of the SW workflow relies on the release.xml.

This structure is similar to the OS repositories like RHEL/SL/Centos and Debian.

The table below contains the triggers and action on the SW packages to be moved through the several UMD repositories.

Repository Trigger Action Comment
testing New release of SW SA2: Pull packages from the Technology Provider’s repository If successful, SA2 produces one release.xml per PPA.
rc Staged rollout done Move packages from testing to rc Staged rollout manager publishes all reports and products are release candidates for the next UMD release
base/updates Products to production Move packages from rc to base/updates Date of the UMD release.

New SW provisioning workflow

The workflow description follows:

  • When the technology providers produce a release or update, those packages would be pulled to the testingttt repository, by mirroring or some other method.
  • SA2.3 triggers this process by automatically creating the release.xml.
  • Verification officially starts the moment the packages are in this repository, it doesn't mean that the actual work starts since this will depend on the availability of the verifiers.
  • The verifier open an RT ticket in the sw-rel queue, it has the exact same fields as there are in the current implementation, and will do that when he starts the verification.
  • Alternatively, SA2 or TSA1.3 just opens all tickets corresponding to all products of that release.
  • Verification concentrates on the documentation, release notes, and what special bugs or vulnerabilities the staged rollout should pay special attention to. It may consist of the actual deployment just in case of major release or some special update of a given product.
  • Verifiers write the verification report and publish it in the docDB with links to the corresponding RT ticket.
  • Staged rollout may start after the verification is done.
  • The Early Adopters may choose more freely when to do a staged rollout of any given component, TSA1.3 will announce that a new release is available for staged rollout and which products are included in this release.
  • The RT staged-rollout queue becomes redundant. Certain custom fields in this queue could be merged or added to the sw-rel queue. This is already a simplification, and would allow the close communication between the verifiers and the Early Adopters. The full workflow happens in the sw-rel queue.
  • When SA1.3 ends the staged rollout, it triggers the movement of the product and its dependencies into the rc repository.
  • EGI will do UMD releases/updates every quarter containing all products in the rc repository.
  • The rc repository is freezed 1 week before the next UMD release date. This is the only period where this repository is freezed and no packages may be moved here. Even if staged rollout activities occurs during this period and are finished, the actual movement from the testing repository will have to wait until all is done with the actual preparation of the next UMD release.
  • At any given time, SA2 or SA1.3 may decide to schedule a release containing urgent updates or security vulnerability fixes.

RT queue “sw-rel” fields

Below are the proposed fields for the sw-rel RT queue, most of them implemented and well known while only the last one come from the staged-rollout queue:

  1. Subject: PPA
  2. Status: |new|open|….. default status of RT tickets.
  3. Queue: sw-rel is fixed.
  4. Owner: | Verifier | Staged rollout manager |
  5. CommunicationStatus: shows the status of the movement of packages between repositories.
  6. RolloutProgress:
    1. Submitted
    2. Unverified
    3. In Verification
    4. StagedRollout
    5. ReleaseCandidate
    6. Production
    7. Waiting for response
    8. Not published
    9. Rejected
    10. Ignored
  7. TP: The technology provider.
  8. UMDRelease: The UMD major release version.
  9. ReleaseMetadata: The release.xml.
  10. QualityCriteriaVerificationReport: the document database ID of the verification report.
  11. StageRolloutReport: the document database ID of the staged rollout reports.
  12. RelatedGGUSTickets: URL’s of GGUS ticket opened during verification and staged rollout.
  13. PassedQC: ACCEPT | REJECT
  14. PassedSR: ACCEPT | REJECT
  15. Failed against mandatory documentation QC: tick box for value = TRUE.
  16. EATeams: Early Adopter teams notified for the staged rollout.

Staged rollout reports

A request/suggestion has been made to EGI IT, to implement a web application where the staged rollout reports could be filled, i.e. turned into a web form. Early Adopter team members could fill the fields and submit when they are done, producing a pdf that TSA1.3 can put into the docDB.

A mail has been sent to all Early Adopters, and feedback is being gathered. Already the answers we got are in favour of this implementation.

Pros and Cons

Advantages

  • All repositories are known apriori by SA1/operations (base, updates, rc and testing).
    • base and updates are the stable/production repositories.
    • testing is an unstable repository. As such any site can, if he wishes so, to enable the testing repository because he wants the new version, or because he is an Early Adopter and will do the test of a given component.
  • The long time lag between release in EMI or other technology provider, and the availability in the UMD repositories disappear, since the new versions will be immediately available in the UMD repositories.
  • It's perceived as a clean solution for all sites since they will point all services to all UMD repositories, and only the base and updates repositories are enabled by default. The testing repository is disabled, but again it will be an easy matter of just enabling it for whatever purpose.
  • rc contain the successful candidates for the new UMD official release.
  • The staged rollout may occur in the timeframe between the Verification is completed until 1 week before the UMD release date. The deadline of the staged rollout reports occurs at this moment. If they don't arrive on that deadline, the product will simply not be in that release, but will stay anyway in the UMD repositories for whoever wants them.

Disadvantages, corner cases and issues

  • The dependencies have to be known, this has to be automatically extracted from the rpms and debs. There should be a close track of the products and components that are at any given moment in the testing repo to know exactly what the verifiers are verifying and the Early Adopters are testing.
  • The release.xml should only have the dependencies that are in the testing repository at any given time. If some package depends on another package that is in the production repositories, then that dependency should be fetched from the production repository and is not included in the release.xml.
  • An UMD release every 3 months, corresponds roughly to 4 EMI updates (they are done every three weeks). Analysing EMI updates there are not many products that where updated more then twice (currently in update 11 of EMI). One possibility would be to treat this in a case-by-case basis, or put in place some method to solve these issues.
  • Having product version x.y in the testing repository under the verification or staged rollout, and if a new version of that product is included in the testing repository, then 2 things could happen:
    1. Either we finish the current verification and staged rollout and pass those packages into the rc repository.
    2. Or shift the attention, i.e. do verification and staged rollout of the new version if there is still time left until the freeze date for the next UMD release.
  • Having the release.xml for each product or a similar list is essential and has to be created automatically.

EMI statistics

In order to evaluate the previous points, some statistics follows. The figure gathers the EMI major release plus it’s 10 updates (at the time of writing), totalizing 11 releases. The figure shows, for each of the 36 products (we took out the internal components), the number of updates plus the major release of that product for each quarter. One quarter corresponds roughly to 4 EMI updates.

Given that EGI aims to have quarterly UMD releases, justifies why we have grouped the EMI releases on a quarterly basis. It can be seen that despite the large number of products, only a small fraction has been updated more than once in a given quarter. Note also that not all EMI products have been included into UMD. Note also that the number of products with 2 or more updates per quarter has decreased, as seen in the table below.

Quarter 1 Quarter 2 Quarter 3
CREAM – 3 updates ARC Infosys – 2 updates StoRM – 2 updates
ARGUS – 2 updates ARC Clients – 2 updates
LB – 2 updates ARC CE – 2 updates
UNICORE UVOS – 2 updates dCache – 2 updates
MPI – 2 updates WMS – 2 updates
WMS – 2 updates

One further point is that EMI plans to start doing longer update cycles of 1 month. As such the number of updates per quarter would be reduced to three while the probability of having more then 1 update of a given product will decrease.

Technical changes

This section highlights some of the technical changes with respect to the current workflow, and is subject (as the rest of the proposal) to further comments and improvement and additions:

  1. The triggering of the process is performed by SA2 the same day that the technology provider announces their release. Instead of being triggered by the technology provider.
  2. The release.xml is produced automatically when SA2 triggers the process. Instead of being produced by the technology provider.
  3. The release.xml contains only the package dependencies of that update. Other packages in production will not be included in the release.xml. Instead of containing the full set of package dependencies.
  4. The release.xml is already split per ppa, since SA2 and SA1.3 has the knowledge (or can have this knowledge) of which components are internal dependencies and which ones are products. This would make the “bouncer” redundant.
  5. The full chain of the SW workflow is done in the RT queue sw-rel, first the verification (as is done now) and after the staged rollout. The custom fields necessary for staged rollout would be mostly moved to the sw-rel queue. This simplifies the process and eases the communication between verifiers and early adopters. All issues, GGUS URLs and verification reports IDs would be directly available to the EAs.
  6. There are 2 new repositories that contain all new packages. These new repositories are static and well publicised especially to SA1. There is no splitting of products into different volatile repositories.

Conclusions/Summary

We are fully aware of the week points, but also think that if we manage to cover properly those week point the advantages far surpass the disadvantages, while having a much more flexible provisioning process, and also more adaptable to a future distribution in EPEL.