EGI Core activities:2013-bidding Software provisioning infrastructure

From EGIWiki
Jump to: navigation, search
Main EGI.eu operations services Support Documentation Tools Activities Performance Technology Catch-all Services Resource Allocation Security


EGI Core services menu: Services PHASE I Services PHASE II Services PHASE III Bids Payments Travel procedure Performance



Go back to the EGI Core Activities Bidding page.


Go back to the Core EGI activity list.

  • Service name: Software provisioning infrastructure
  • Service category: Software
  • Service type: Operation and maintenance

Introduction

The software provisioning infrastructure provides the technical tools to support the UMD release process from pulling packages from the developers repositories to the build of a release.

Technical description

The software provisioning infrastructure is composed by the following components:

Note: the architecture of the internal components is not mandatory, but the services provided must be equivalent.

  • Integration with RT, a new product release (the tuple Product, Platform, Architecture) is associated with a RT ticket, which tracks the status of the product in the software provisioning process.
  • Bouncer and related tools: a set of tools that starting from the information provided during the submission of the product to UMD (currently release.xml), queries the developers repositories as well as the associated operating system add-ons repositories to determine which packages compose the product and must be downloaded into the UMD repositories.
  • Repository Back-end: responsible unit for handling the movement of packages between repositories, validating the individual product releases submissions, building accumulative as well as per-product YUM/APT repositories (multiple per OS/Arch case) and the other automations needed to perform the UMD operations. It also provide a RESTful API for external integrations (e.g. with the UMD portal/frontend)
  • Composer: a web-based interface for bundling versioned software products that have successfully passed the UMD verification process, into a robust UMD release ready to be deployed either to the production or the candidate repositories.
  • ReleaseXML editor: a web interface to create a new entry in the UMD release process, it is connected with the RT and the bouncer
  • Repositories, the following respositories must be maintained for every operating system and major release supported:
    • Untested: contains the packages to be installed during the verification
    • Testing: contains the packages to be installed during staged rollout
    • Base: contains the packages released in the first major release
    • Update: contains the packages released in the update releases
    • Release Candidate: it is generated before a UMD release, to simulate the production repositories after the UMD release under preparation. This is used to test the installability of the newly released components, as well as the products already in production.
    • The processes to move products between repositories and to create releases must be as automated as possible.
    • The task must provide statistics about the repository usage in terms of downloads, aggregated by packages and time.
  • Community repositories: through AppDB developers must have access to a repository-as-a-service platform to upload their software release and easily push them in the UMD software provisioning process, if necessary.
  • Front-end, the information about UMD releases (release notes, list of components, configuration) must be available in a web frontend.
  • Testbed for the verification of UMD components, and general middleware testing.
    • The testbed currently used is composed by 20 cores and 2GB ram per core. For the future, a similar size is expected.
    • Authorized users must be able to instantiate virtual machines independently, at any time
    • Testbed must provide configuration templates for the most common components

The software provisioning infrastructure must support multiple operating system (SL based, and Debian based) and major releases (at least two major releases).

Operations

The task must operate all the technical services described before:

  • Repositories (production, testing, untested and RC, community repositories)
  • Repositories back-end (including UMD composer)
  • Bouncer
  • Web pages (repository front-end, Release XML editor)
  • Testbed

The task must support the UMD release creation, creating the release candidates and the actual releases.
Release candidates must be tested for the installation of all the components, new and already available in the repositories. The test, possibly automated, must be able to generate a report in few hours (less than one working day, possibly 2-4 hours).

Maintenance

The infrastructure must be adapted to the evolution of EGI infrastructure to support:

  • New operating system (most probably only new SL and Debian releases)
  • New source repositories to pull packages for the UMD provisioning (e.g. as new technology providers contribute to UMD)

Service level targets

Support

The activity must provide support through the dedicated GGUS support unit, during working hours

Availability

UMD repositories and web front-end, as well as the community repository, must have 90% availability and reliability on a monthly base. The other components used by the UMD team only must have 75% availability and 90% reliability during working hours.

Load Balancing and high availaiblity configuration (between two or more servers) is required in order to handle the load on the repositories and the front-end service.