Alert.png The wiki is deprecated and due to be decommissioned by the end of September 2022.
The content is being migrated to other supports, new updates will be ignored and lost.
If needed you can get in touch with EGI SDIS team using operations @ egi.eu.

Difference between revisions of "2016-bidding/UMD and CMD infrastructure"

From EGIWiki
Jump to navigation Jump to search
Line 12: Line 12:
*Integration with RT, a new product release (the tuple Product, Platform, Architecture) is associated with a RT ticket, which tracks the status of the product in the software provisioning process.  
*Integration with RT, a new product release (the tuple Product, Platform, Architecture) is associated with a RT ticket, which tracks the status of the product in the software provisioning process.  
*Submission of new products with XML.  
*Submission of new products with XML.  
*Repository Back-end: responsible unit for handling the movement of packages between repositories, validating the individual product releases submissions, building accumulative as well as per-product YUM/APT repositories (multiple per OS/Arch case) and the other automations needed to perform the UMD operations. It also provide a RESTful API for external integrations (e.g. with the UMD portal/frontend)  
*Repository Back-end: responsible unit for handling the movement of packages between repositories, validating the individual product releases submissions, building accumulative as well as per-product YUM/APT repositories (multiple per OS/Arch case) and the other automations needed to perform the UMD and CMD operations. It also provide a RESTful API for external integrations (e.g. with the UMD and CMD portal/frontend)  
*Composer: a web-based interface for bundling versioned software products that have successfully passed the UMD verification process, into a robust UMD release ready to be deployed either to the production or the candidate repositories.  
*Composer: a web-based interface for bundling versioned software products that have successfully passed the UMD and CMD verification process, into a robust UMD and CMD release ready to be deployed either to the production or the candidate repositories.  
*ReleaseXML editor: a web interface to create a new entry in the UMD release process, it is connected with the RT and the bouncer  
*ReleaseXML editor: a web interface to create a new entry in the UMD and CMD release process, it is connected with the RT and the bouncer  
*Repositories: the following repositories must be maintained for every operating system and major release supported:  
*Repositories: the following repositories must be maintained for every operating system and major release supported:  
**Untested: contains the packages to be installed during the verification  
**Untested: contains the packages to be installed during the verification  
Line 20: Line 20:
**Base: contains the packages released in the first major release  
**Base: contains the packages released in the first major release  
**Update: contains the packages released in the update releases  
**Update: contains the packages released in the update releases  
**Release Candidate: it is generated before a UMD release, to simulate the production repositories after the UMD release under preparation. This is used to test the installability of the newly released components, as well as the products already in production.  
**Release Candidate: it is generated before a UMD and CMD release, to simulate the production repositories after the UMD and CMD release under preparation. This is used to test the installability of the newly released components, as well as the products already in production.  
*The processes to move products between repositories and to create releases must be as automated as possible.  
*The processes to move products between repositories and to create releases must be as automated as possible.  
*The task must provide statistics about the repository usage in terms of downloads, aggregated by packages and time.  
*The task must provide statistics about the repository usage in terms of downloads, aggregated by packages and time.  
Line 32: Line 32:


== Coordination==
== Coordination==
The task must coordinate with the UMD quality assurance task as well as EGI Operations when necessary, and with the AppDB provider to support the community repositories.
The task must coordinate with the UMD and CMD quality assurance task as well as EGI Operations when necessary, and with the AppDB provider to support the community repositories.


== Operations ==
== Operations ==

Revision as of 17:09, 20 October 2016

Main EGI.eu operations services Support Documentation Tools Activities Performance Technology Catch-all Services Resource Allocation Security


EGI Core services menu: Services PHASE I Services PHASE II Services PHASE III Bids Payments Travel procedure Performance



Go back to the EGI Core Activities Bidding page.

  • Service name: UMD Software provisioning infrastructure

Introduction

The software provisioning infrastructure provides the technical tools to support the UMD release process, used for UMD (Unified Middleware Distribution) and CMD (Cloud Middleware Distribution) from pulling packages from the developers repositories to the build of a release. CMD follow exactly the same process as for UMD; the only difference is in the content (CMD contains cloud-oriented software) and in the release cycle (months for CMD against years for UMD).

Technical description

The software provisioning infrastructure is composed by the following components:

  • Integration with RT, a new product release (the tuple Product, Platform, Architecture) is associated with a RT ticket, which tracks the status of the product in the software provisioning process.
  • Submission of new products with XML.
  • Repository Back-end: responsible unit for handling the movement of packages between repositories, validating the individual product releases submissions, building accumulative as well as per-product YUM/APT repositories (multiple per OS/Arch case) and the other automations needed to perform the UMD and CMD operations. It also provide a RESTful API for external integrations (e.g. with the UMD and CMD portal/frontend)
  • Composer: a web-based interface for bundling versioned software products that have successfully passed the UMD and CMD verification process, into a robust UMD and CMD release ready to be deployed either to the production or the candidate repositories.
  • ReleaseXML editor: a web interface to create a new entry in the UMD and CMD release process, it is connected with the RT and the bouncer
  • Repositories: the following repositories must be maintained for every operating system and major release supported:
    • Untested: contains the packages to be installed during the verification
    • Testing: contains the packages to be installed during staged rollout
    • Base: contains the packages released in the first major release
    • Update: contains the packages released in the update releases
    • Release Candidate: it is generated before a UMD and CMD release, to simulate the production repositories after the UMD and CMD release under preparation. This is used to test the installability of the newly released components, as well as the products already in production.
  • The processes to move products between repositories and to create releases must be as automated as possible.
  • The task must provide statistics about the repository usage in terms of downloads, aggregated by packages and time.
  • Front-end, the information about UMD and CMD releases (release notes, list of components, configuration) must be available in a web frontend.

Note: the architecture of the internal components is not mandatory, but the services provided must be equivalent.

The software provisioning infrastructure must support multiple distributions, multiple operating system (EL based, and Debian based) and major releases (at least two major releases).

The infrastructure should also support a “Preview” repository where products are quickly released without verification. It is not not an official UMD repository, but it represents a place where products can be made available to service providers more quickly and directly, bypassing the quality assurance steps.

Coordination

The task must coordinate with the UMD and CMD quality assurance task as well as EGI Operations when necessary, and with the AppDB provider to support the community repositories.

Operations

The task must operate all the technical services described before:

  • Repositories (production, testing, untested and RC, community repositories)
  • Repositories back-end (including UMD composer)
  • Web pages (repository front-end, Release XML editor)

The task must support the creation of the distributions and for each distribution the creation of the releases, creating the release candidates and the actual releases.

Support

The activity must provide support through the dedicated GGUS support unit, during working hours.

Service level targets

UMD repositories and web front-end, as well as the community repository, must have 90% availability and reliability on a monthly base. The other components used by the UMD team only must have 75% availability and 90% reliability during working hours.

Load Balancing and high availability configuration (between two or more servers) is required in order to handle the load on the repositories and the front-end service.

Effort

Bids planning a total effort between 10 and 12 Person Months/year would allow these services and activities to be addressed appropriately.