Difference between revisions of "Middleware"

From EGIWiki
Jump to: navigation, search
 
(41 intermediate revisions by 4 users not shown)
Line 1: Line 1:
{{Template:Op menubar}}
+
{{Template:Tech menubar}} {{Template:Middleware_menubar}}  
{{TOC_right}}
 
  
[[Category:Middleware]]
+
This wiki collects all the information concerning Middleware supported in EGI.  
{{Template:Middleware_menubar}}
 
This page provides information on supported software, the related release roadmaps and documentation, together with information about processes to validate and deploy new releases and to phase out unsupported software releases. The Technology Providers that currently signed a Service Level Agreement are the [http://www.eu-emi.eu/ European Middleware Initiative] (EMI) and the [http://www.ige-project.eu/ Initiative for Globus in Europe] (IGE).
 
  
= Unified Middleware Distribution =
+
<br>
The Unified Middleware Distribution (UMD) is the integrated set of software components that EGI makes available from technology providers within the EGI Community. These components are distributed to provide an integrated offering for deployment on the EGI production infrastructure [http://www.egi.eu/deployed-technology/technology-roadmap/ More]
 
  
Subscribe the [http://repository.egi.eu/category/umd_releases/feed UMD Release feed]
 
  
== [http://repository.egi.eu/2012/07/10/release-umd-2-0-0/ UMD 2.0] ==
 
* [[UMD-2:UMD-2.0.0|UMD-2.0.0]] |[[UMD-2:UMD-2.1.0|UMD-2.1.0]] | [[UMD-2:UMD-2.2.0|UMD-2.2.0]] | [[UMD-2:UMD-2.2.1|UMD-2.2.1]] | [[UMD-2:UMD-2.2.2|UMD-2.2.2]] | [[UMD-2:UMD-2.3.0|UMD-2.3.0]] | [[UMD-2:UMD-2.4.0|UMD-2.4.0]]
 
  
== [http://repository.egi.eu/category/umd_releases/distribution/umd_1/ UMD 1.0] ==
+
<br>
* [[UMD-1:UMD-1.0.0|UMD-1.0.0]] | [[UMD-1:UMD-1.1.0|UMD-1.1.0]] | [[UMD-1:UMD-1.2.0|UMD-1.2.0]] | [[UMD-1:UMD-1.3.0|UMD-1.3.0]] | [[UMD-1:UMD-1.4.0|UMD-1.4.0]] | [[UMD-1:UMD-1.5.0|UMD-1.5.0]] | [[UMD-1:UMD-1.6.0|UMD-1.6.0]] | [[UMD-1:UMD-1.7.0|UMD-1.7.0]] |[[UMD-1:UMD-1.7.1|UMD-1.7.1]] |[[UMD-1:UMD-1.8.0|UMD-1.8.0]] | [[UMD-1:UMD-1.8.1|UMD-1.8.1]] |[[UMD-1:UMD-1.9.0|UMD-1.9.0]] | [[UMD-1:UMD-1.10.0|UMD-1.10.0]]
 
  
== Release schedule ==
+
== UMD3 and UMD4 (main middleware distribution)  ==
* [[UMD_Release_Schedule#Unified_Middleware_Distribution_1 |UMD 1 update schedule]]
 
* [[UMD_Release_Schedule#Unified_Middleware_Distribution_2| UMD 2 update schedule]]
 
  
== Software provisioning dashboard ==
+
Unified Middleware Distribution (UMD) is the integrated set of software components contributed by Technology Providers and packaged for deployment as production quality services in EGI.  
Check the list of products that are in verification, staged rollout or are released in UMD for production on the [https://rt.egi.eu/rt/Dashboards/260/Software%20Provisioning%20Dashboard Software provisionig dashboard]
 
* [[EGI_Verifier_Guideline#UMD_release_schedule| UMD preparation process]]
 
  
== Roadmap ==
+
'''UMD4 is the current supported UMD distribution.''' It supports CentOS7, SL6. Also Ubuntu 16 (Xenial) will be supported in the future, if Technology Providers require that.<br>
'''Latest UMD Roadmap''': [https://documents.egi.eu/document/612 D5.4]
 
  
= Technology Providers=
+
UMD3 will still receive security or critical updates for SL6. UMD3/SL5 and UMD3/Debian are not supported anymore.
  
{| border="1"
+
=== Support for yaim  ===
|- style="background-color: lightgray;"
 
| '''Provider'''
 
| '''Relese Notes/Updates'''
 
| '''Support calendar'''
 
| '''Release plan'''
 
| '''Information'''
 
|-
 
| '''<div id="EMI">[[#EMI|EMI]]</div>'''
 
|
 
  
[http://www.eu-emi.eu/emi-2-matterhorn EMI 2 Matterhorn]
+
With UMD4/CentOS7 yaim is not supported anymore as a tool to configure the distributed middleware. Instead, every product will follow its documented procedure to install, configure, test the product.  
*[https://twiki.cern.ch/twiki/bin/view/EMI/EMI2Updates EMI-2 Updates]
 
[http://www.eu-emi.eu/emi-1-kebnekaise EMI-1 Kebnekaise]
 
*[https://twiki.cern.ch/twiki/bin/view/EMI/EMI1Updates EMI-1 Updates]
 
| [http://www.eu-emi.eu/releases#MajRel EMI Support Calendar]
 
[http://www.eu-emi.eu/releases EMI Release Policy]
 
  
[http://www.eu-emi.eu/support EMI Incident Handling Policy]
+
So for each product, UMD users will point to the product-specific documentation, which will provide puppet recipes, or ansible recipes, or custom scripts or even step-by-step guides (this last is true especially with very simply configurated products). Even yaim can be used as a configuration tool, and indeed you will find it distributed in the upcoming UMD4.4, but this will mean that some team (in this case the VOBOX) is using yaim to configure their products.  
|
 
* [https://savannah.cern.ch/task/?group=emi-releases Release tracker]
 
* [http://go.egi.eu/emi-overall Overview of updates] (released and to be released)
 
|
 
* [https://twiki.cern.ch/twiki/pub/EMI/DeliverableDSA11/EMI-DSA1.1-1277556-Software_Maintenance_Support_Plan-v1.0.pdf EMI Software Maintenance and Support Plan]
 
* [https://twiki.cern.ch/twiki/bin/view/EMI/DeliverableDNA133 DNA1.3.3 Technical Development Plan] (PY3)
 
* [https://twiki.cern.ch/twiki/bin/view/EMI/DeliverableDJRA113 DJRA1.1.3 Compute Area Work Plan and Status Report] (PY3)
 
<!--** [https://twiki.cern.ch/twiki/bin/view/EMI/EmiJra1T2Compute Status of implementation]-->
 
* [https://twiki.cern.ch/twiki/bin/view/EMI/DeliverableDJRA123 DJRA1.2.3 Data Area Work Plan and Status Report] (PY2)
 
<!--** [https://twiki.cern.ch/twiki/bin/view/EMI/EmiJra1T3DataDJRA123 Status of implementation]-->
 
* [https://twiki.cern.ch/twiki/bin/view/EMI/DeliverableDJRA133 DJRA1.3.3 Security Area Work Plan and Status Report] (PY2)
 
** [https://twiki.cern.ch/twiki/bin/view/EMI/EmiJra1T4Security Status of implementation]
 
* [https://twiki.cern.ch/twiki/bin/view/EMI/DeliverableDJRA143 DJRA1.4.3 Infrastructure Area Work Plan and Status Report] (PY2)
 
** [https://twiki.cern.ch/twiki/bin/view/EMI/InfrastructureArea Status of implementation]
 
* [https://twiki.cern.ch/twiki/bin/view/EMI/EmiDeliverables All deliverables]
 
* EMI [https://twiki.cern.ch/twiki/bin/view/EMI/EMTSrcTemplate#SrcLinks Service Cards]
 
* ''DRAFT'' EMI's CEs [http://www.nordugrid.org/arc/ce/chart.html comparison chart]
 
|-
 
| '''<div id="gLite">[[#gLite|gLite]]</div>'''
 
| [http://glite.cern.ch/ Latest releases]
 
| [http://tinyurl.com/gLitePatches Release Schedule]
 
*[http://glite.cern.ch/support_calendar/ gLite 3.1 and 3.2]
 
* '''gLite 3.1 IS RETIRED''' ([http://glite.cern.ch/R3.1/ gLite 3.1])
 
* gLite 3.2 [http://www.eu-emi.eu/support support policy]
 
* gLite security updates: [https://twiki.cern.ch/twiki/bin/view/EGEE/GLiteSecUpdates process]
 
| <!-- Next releases-->
 
| [http://go.egi.eu/glite32-patches gLite3.2 patches under development]
 
* [http://glite.cern.ch/admin_documentation Installation and configuration]
 
* [http://glite.cern.ch/R3.2/ gLite 3.2]
 
* [http://glite.cern.ch/R3.1/ gLite 3.1]
 
* [https://edms.cern.ch/document/722398/ gLite User Guide]
 
<!--[https://twiki.cern.ch/twiki/bin/view/LCG/WLCGBaselineVersions WLCG baseline clients and services]-->
 
|-
 
| '''<div id="IGE">[[#IGE|IGE]]</div>'''
 
|
 
[http://www.ige-project.eu/downloads/software/releases/210 IGE 2.1.0] release notes
 
  
[http://www.ige-project.eu/downloads/software/releases/200 IGE 2.0.0] release notes
+
=== Why is SL6 both in UMD3 and UMD4?  ===
  
[http://www.ige-project.eu/releases/100 IGE 1.0.0] release notes
+
The UMD3/SL6 and UMD4/SL6: the latter is a mirror of the former excluding products that are reaching or already reached end of life. If you want to install a UI, you can use both, but we advice to use UMD4/SL6.Of course dedicated documentation will be provided with the release in CentOS7.  
| IGE support calendar is being discussed
 
| [http://www.ige-project.eu/downloads/software/releases Releases]
 
  
IGE 3.0 expected in September 2012 ([[File:IGEReleaseSchedule-23-05-2012.pdf |IGE Release plan]], May 2012).
+
== CMD-OS and CMD-ONE (cloud middleware) ==
| [http://www.ige-project.eu/hub/rt Instructions] to submit requirement to IGE
 
[http://rt.ige-project.eu/rt/Search/Results.html?Query=Queue%20%3D%20%27Change%2FRequirements%20Requests%27%20AND%20%28Status%20%3D%20%27new%27%20OR%20Status%20%3D%20%27open%27%20OR%20Status%20%3D%20%27stalled%27%29 IGE Requirements]
 
|-
 
|}
 
  
<!-- ==gLite 3.1 and 3.2 ==
+
The Cloud Middleware Distribution (CMD) distributes OpenStack and OpenNebula integration components developed by Cloud Technology Providers to Resource Centres (i.e. cloud sites).  
* '''Supported middleware'''
 
  
 +
Two different distribution are available, CMD-OS for OpenStack and CMD-ONE for OpenNebula. For further details, please have a look at the Cloud Middleware Distribution wiki https://wiki.egi.eu/wiki/EGI_Cloud_Middleware_Distribution .
  
** [https://twiki.cern.ch/twiki/bin/view/LCG/WLCGBaselineVersions WLCG baseline clients and services]
+
== UMD and CMD ==
* '''Releases'''
 
** [http://glite.cern.ch/ Latest releases] and future [http://bit.ly/22we3i Release Schedule]
 
  * [http://glite.web.cern.ch/glite/packages/latestRelease.asp Latest updates]
 
  
* '''Documentation'''
+
Regular grid components and in general all non cloud-specific components will keep going to UMD. Also clients for the user to interact with the EGI infrastructure are distributed with UMD (storage clients, cloud-related clients).  
** [http://glite.cern.ch/admin_documentation Installation and configuration documentation]
 
* [http://glite.web.cern.ch/glite/documentation/default.asp User guide and general docuementation]
 
** gLite [http://glite.web.cern.ch/glite/packages/R3.2/ 3.2]: User guide and general docuementation
 
** gLite [http://glite.web.cern.ch/glite/packages/R3.1/ 3.1]: User guide and general docuementation
 
** [http://glite.web.cern.ch/glite/documentation/userGuide.asp gLite User Guide]
 
  
* NEW! EMI services [https://twiki.cern.ch/twiki/bin/view/EMI/EMIServicesDocs documentation]
+
At the moment UMD4 is supported, accepting CentOS7 and SL6 packages. '''Ubuntu is not yet enabled'''.  
* NEW! Release notes per component becoming available [[EMI_release_notes]]
 
=== ARC ===
 
* [http://www.nordugrid.org/arc/ Overview]
 
*'''Releases'''
 
** [http://www.nordugrid.org/arc/releases/0_8_3/ Latest release]
 
** [http://wiki.nordugrid.org/index.php/Roadmap Development roadmap]
 
** [http://wiki.nordugrid.org/index.php/Release_management Release management]
 
*'''[http://www.nordugrid.org/papers.html Documentation]'''
 
  
=== UNICORE ===
+
Products in UMD major releases should have a release cycle that is compatible with the UMD one: major releases are supported with updates for at least 1 year, security for 2 years. As this doesn't happen with OpenStack related components, we created CMD for OpenStack.  
*'''Download'''
 
** Download server & central services: [https://sourceforge.net/projects/unicore/files/1%20UNICORE%206%20Servers%20and%20Services/1.1%20Core%20Server/ Core Server] | [https://sourceforge.net/projects/unicore/files/1%20UNICORE%206%20Servers%20and%20Services/1.2%20Workflow%20System/ Workflow System] | [https://sourceforge.net/projects/unicore/files/1%20UNICORE%206%20Servers%20and%20Services/1.3%20Common%20Information%20Service/ Common Information Service]
 
** Download clients: [https://sourceforge.net/projects/unicore/files/2%20UNICORE%206%20Clients,%20Applications,%20and%20APIs/2.1%20UNICORE%20Rich%20Client%20(URC)/ Rich Client] | [https://sourceforge.net/projects/unicore/files/2%20UNICORE%206%20Clients,%20Applications,%20and%20APIs/2.3%20Commandline%20Client%20(UCC)/ Command Line Client] | [https://sourceforge.net/projects/unicore/files/2%20UNICORE%206%20Clients,%20Applications,%20and%20APIs/2.4%20HiLA%20Programming%20API/ HiLA API]
 
* '''[http://unicore.eu/documentation/ Documentation]'''
 
-->
 
  
= YAIM =
+
CMD for OpenNebula has been created to decouple from OpenStack, having a very different release cycle.  
* [https://twiki.cern.ch/twiki/bin/view/LCG/Site-info_configuration_variables YAIM configuration variables]
 
* [https://twiki.cern.ch/twiki/bin/view/LCG/YaimGuide400 YAIM 4 Guide for System Administrators]
 
* [https://twiki.cern.ch/twiki/bin/view/LCG/YaimGuide400#Known_issues YAIM 4 known issues]
 
  
=Software release and deployment process =
+
{| width="800" cellspacing="1" cellpadding="1" border="1"
This section describes the EGI process to deploy new versions of deployed software and operational tools into the EGI production infrastructure.
+
|+ '''Supported middleware distributions summary'''
==Software Provisioning==
+
|-
<!--* [[Middleware:Release_Process]]-->
+
|
* [[EGI_Software_Provisioning|EGI Software Provisioning]]
+
| '''Description'''
* '''ÍMPORTANT'''. [https://rt.egi.eu/rt/Dashboards/260/Software%20Provisioning%20Dashboard Software Provisioning Dashboard] showing by category products still to be verified, under verification, in staged rollout, and ready for being included in the next UMD release
+
| '''Operating Systems supported'''
 +
|-
 +
| '''UMD4'''
 +
| Regular grid components and in general all non cloud-specific components will keep going to UMD. Also clients for the user to interact with the EGI infrastructure are distributed with UMD (storage clients, cloud-related clients).  
 +
| CentOS7, SL6
 +
|-
 +
| '''CMD-OS'''
 +
| OpenStack integration components developed by Cloud Technology Providers to Resource Centres
 +
| CentOS7, Xenial
 +
|-
 +
| '''CMD-ONE'''  
 +
| OpenNebula integration components developed by Cloud Technology Providers to Resource Centres
 +
| CentOS7
 +
|}
  
== Software Deployment ==
+
== Wiki description ==
  
*Learn about [[Staged-Rollout|Staged Rollout]]
+
{|
*Staged Rollout [[Staged-rollout-procedures|workflow]]
+
|-
*Staged Rollout managers:
+
| [[Software Calendars|Software Calendars]]  
**ARC: Christian Ulrik Soettrup
+
| Support Calendar, Decommissioning Calendar, Operations Calendar
**gLite: Mario David
+
|-
**GLOBUS: Mario David
+
| [[Technology Providers|Technology Providers list]]  
**UNICORE: Mathilde Romberg
+
| Information about Technology Providers supporting EGI
**SAM: Daniele Cesini
+
|-
*Early Adopters:
+
| [[UMD Release Schedule|Unified Middleware Distribution (UMD) release schedule]]
**[https://www.egi.eu/earlyAdopters/table Early Adopters]
+
| the planned release schedule for UMD major and minor releases
**[https://www.egi.eu/earlyAdopters/teams Distribution] of Early Adopters per component
+
|-
**[[NO_EA]] List of products without EA, they will not be submitted to the UMD SW provisioning]
+
| [[Software release and deployment process|Release and deployment process]]  
*[https://rt.egi.eu/guest/Dashboards/260/Software%20Provisioning%20Dashboard Software Provisioning Dashboard]  
+
| describes the EGI process to deploy new versions of deployed software and operational tools into the EGI production infrastructure
**[http://www.lip.pt/computing/apps/EGI_EA/index.php Staged Rollout quarterly metrics]  
 
*[[EGI IGTF Release|EGI IGTF Release Process]]
 
*WMS service versions monitoring (from Glue1.3 - TopBDII): [[WMS_Monitor]]
 
*CE's service versions monitoring (from Glue1.3 - TopBDII): [[CE_Monitor]]
 
 
 
= Requirements=
 
== Existing requirements ==
 
 
 
=== OMB Statement of Requirements ===
 
Overview of [[OMB_Requirements| OMB approved requirements]] (static page)
 
 
 
=== Requirements dashboards ===
 
The following table provides access to ''all'' open EGI requirements (UMD requirements and tool requirements), submitted by the User Community and the Operations Community.
 
 
 
 
 
{|border="1"
 
|- style="background-color:darkgray;"
 
|'''Board'''
 
|'''Middleware Requirements'''
 
|'''Tool Requirements'''
 
|'''Nagios Requirements'''
 
|'''Comment'''
 
|-  
 
| Technology Collaboration Board
 
| [[Track_UMD_Requirements |Go to wiki dashboard]]
 
| -
 
| -
 
| This category shows high priority tickets (subset of all tickets) that have already been forwarded to TCB
 
 
|-
 
|-
| Operations Management Boards
+
| [[Middleware Requirements|Middleware and Tool Requirements]]  
| [http://go.egi.eu/omb-requirements OMB Requirements] (RT tickets)
+
| existing requirements, how to submit requirements
| [[Track_Operations_Requirements| Tool Requirements]]
 
| [https://rt.egi.eu/rt/Dashboards/1748/SA1%20Nagios%20requirements OMB Nagios Requirements]
 
| This category shows ALL tickets discussed within the OMB in different technical areas.
 
 
|-
 
|-
| User Community Board
+
| [[EGI Cloud Middleware Distribution|Cloud Middleware Distribution]]  
| [[Track_User_Support_Requirements| Go to wiki dashboard]]
+
| details on the CMD release cycle(still proposal)
| -
 
| -
 
| This category shows tickets discussed within the User Community.
 
 
|}
 
|}
  
== How to submit requirements ==
+
= External resources  =
* [[Nagios_requirements |Nagios test requirements]]
 
* [[Submitting_a_middleware_requirement| Middleware requirements]]
 
  
== More information==
+
*[https://twiki.cern.ch/twiki/bin/view/LCG/WLCGBaselineVersions Baseline versions of services and client tools for WLCG]
* [https://www.egi.eu/indico/getFile.py/access?contribId=1&resId=0&materialId=0&confId=152 EGI Operations requirements gathering process], OMB, 21 Dec 2010
 
* [[EGI Operations Surveys]]
 
* [[Jobs wordir and tempdir]]
 
  
=Documentation=
+
[[Category:Middleware]]
*[[Other_Operations_Documentation#StandardsStandards|Standards]]
 
*[[Documentation]]
 
 
 
=External resources=
 
* [https://twiki.cern.ch/twiki/bin/view/LCG/WLCGBaselineVersions Baseline versions of services and client tools for WLCG]
 

Latest revision as of 18:01, 22 August 2017

Technology Software Component Delivery Software Provisioning UMD Middleware Cloud Middleware Distribution Containers Distribution Technology Glossary


Middleware menu: Home Software Calendars Technology Providers UMD Release Schedule UMD Products Overview UMD Products ID Cards Release and deployment process Middleware Requirements Next middleware release


This wiki collects all the information concerning Middleware supported in EGI.




UMD3 and UMD4 (main middleware distribution)

Unified Middleware Distribution (UMD) is the integrated set of software components contributed by Technology Providers and packaged for deployment as production quality services in EGI.

UMD4 is the current supported UMD distribution. It supports CentOS7, SL6. Also Ubuntu 16 (Xenial) will be supported in the future, if Technology Providers require that.

UMD3 will still receive security or critical updates for SL6. UMD3/SL5 and UMD3/Debian are not supported anymore.

Support for yaim

With UMD4/CentOS7 yaim is not supported anymore as a tool to configure the distributed middleware. Instead, every product will follow its documented procedure to install, configure, test the product.

So for each product, UMD users will point to the product-specific documentation, which will provide puppet recipes, or ansible recipes, or custom scripts or even step-by-step guides (this last is true especially with very simply configurated products). Even yaim can be used as a configuration tool, and indeed you will find it distributed in the upcoming UMD4.4, but this will mean that some team (in this case the VOBOX) is using yaim to configure their products.

Why is SL6 both in UMD3 and UMD4?

The UMD3/SL6 and UMD4/SL6: the latter is a mirror of the former excluding products that are reaching or already reached end of life. If you want to install a UI, you can use both, but we advice to use UMD4/SL6.Of course dedicated documentation will be provided with the release in CentOS7.

CMD-OS and CMD-ONE (cloud middleware)

The Cloud Middleware Distribution (CMD) distributes OpenStack and OpenNebula integration components developed by Cloud Technology Providers to Resource Centres (i.e. cloud sites).

Two different distribution are available, CMD-OS for OpenStack and CMD-ONE for OpenNebula. For further details, please have a look at the Cloud Middleware Distribution wiki https://wiki.egi.eu/wiki/EGI_Cloud_Middleware_Distribution .

UMD and CMD

Regular grid components and in general all non cloud-specific components will keep going to UMD. Also clients for the user to interact with the EGI infrastructure are distributed with UMD (storage clients, cloud-related clients).

At the moment UMD4 is supported, accepting CentOS7 and SL6 packages. Ubuntu is not yet enabled.

Products in UMD major releases should have a release cycle that is compatible with the UMD one: major releases are supported with updates for at least 1 year, security for 2 years. As this doesn't happen with OpenStack related components, we created CMD for OpenStack.

CMD for OpenNebula has been created to decouple from OpenStack, having a very different release cycle.

Supported middleware distributions summary
Description Operating Systems supported
UMD4 Regular grid components and in general all non cloud-specific components will keep going to UMD. Also clients for the user to interact with the EGI infrastructure are distributed with UMD (storage clients, cloud-related clients). CentOS7, SL6
CMD-OS OpenStack integration components developed by Cloud Technology Providers to Resource Centres CentOS7, Xenial
CMD-ONE OpenNebula integration components developed by Cloud Technology Providers to Resource Centres CentOS7

Wiki description

Software Calendars Support Calendar, Decommissioning Calendar, Operations Calendar
Technology Providers list Information about Technology Providers supporting EGI
Unified Middleware Distribution (UMD) release schedule the planned release schedule for UMD major and minor releases
Release and deployment process describes the EGI process to deploy new versions of deployed software and operational tools into the EGI production infrastructure
Middleware and Tool Requirements existing requirements, how to submit requirements
Cloud Middleware Distribution details on the CMD release cycle(still proposal)

External resources