Alert.png The wiki is deprecated and due to be decommissioned by the end of September 2022.
The content is being migrated to other supports, new updates will be ignored and lost.
If needed you can get in touch with EGI SDIS team using operations @ egi.eu.

Difference between revisions of "EGI Quality Criteria Verification"

From EGIWiki
Jump to navigation Jump to search
 
(46 intermediate revisions by 6 users not shown)
Line 1: Line 1:
{{Tech menubar}} {{Tech QA submenu}} {{TOC_right}}  
{{Tech menubar}}
{{SWProv menubar}}
{{TOC_right}}  


== Objective  ==
== Objective  ==
Line 24: Line 26:
*If its necessary Verifiers should include a comment for StageRollout: Configuration changes or minor issues found verifying the product.  
*If its necessary Verifiers should include a comment for StageRollout: Configuration changes or minor issues found verifying the product.  
*If a new QC is necessary and is not included, Verifiers must write a comment to SA2.2 to change current QC.
*If a new QC is necessary and is not included, Verifiers must write a comment to SA2.2 to change current QC.
<br>
== UMD Release Candidate Testing  ==
Before each UMD release the verification team checks the new RC. To perform this task SA2.3 VMs have included the [https://github.com/alvarosimon/RC_tester RC_testing script]. This script is available after each SA2 VM instantiation (into /root/configuration_templates/tools directory). This script is able to detect any package inconsistency after UMD RC repo configuration. The RC testing process is as follows:
*Verifier should instantiate different Linux flavours VM. For UMD case SL5, SL6 and debian.
*Install UMD RC repo files from UMDx RC page.
*Check RC_tester "PRODUCTS" array. '''This array must be updated to include all UMD products!'''
*It is recommended to use ''screen'' program before RC_tester execution. RC tests take about 2h/3h to finish (depends on OS used and the number of products).
*Run ''RC_tester'' and follows its instructions.
*''RC_tester'' generates several logs into log directory, after its execution SA2 verifier should check:
**''installator_OK.log'': List of metapackages installed correctly.
**''installator_ERROR.log'': List of metapackages which contain any issue.
**''&lt;PRODUCT_NAME&gt;_OUTPUT.log'': Complete Product installation log.
**''&lt;PRODUCT_NAME&gt;_postupdate.log'': Info about product update execution. It detects any issue updating current UMD products.
<br>
== Verification Work Effort Metrics  ==
Each verification process and change is registered by EGI RT. A complete report is generated on a daily basis each midnight. The XLS file can be downloaded from here [https://rt.egi.eu/rt/SA2/sa2-sw-rel-verification-metrics.xls SA2 Verification Metrics]
<br>


== Verification Team  ==
== Verification Team  ==
Line 31: Line 58:
== Reference Documents  ==
== Reference Documents  ==


{| cellspacing="0" cellpadding="3" border="1" style="border: 1px solid black;" class="wikitable sortable"
{| border="1" cellspacing="0" cellpadding="3" style="border: 1px solid black;" class="wikitable sortable"
|- align="left" style="background: none repeat scroll 0% 0% Lightgray;"
|- align="left" style="background: none repeat scroll 0% 0% Lightgray;"
! DocDB link  
! DocDB link  
Line 37: Line 64:
! Document
! Document
|-
|-
| [https://documents.egi.eu/document/364 364]  
| [http://egi-qc.github.io/ QCv6]
| 11. 08. 2011
| 10 2013
| UMD Quality Criteria
|-
| [https://documents.egi.eu/document/1700 QCv5]  
| 08. 05. 2013
| UMD Quality Criteria
| UMD Quality Criteria
|-
|-
| [https://documents.egi.eu/document/418 418]  
| [https://documents.egi.eu/document/418 418]  
| 11. 08. 2011
| 08. 05. 2013
| UMD Products service mapping
| UMD Products service mapping
|-
|-
| [https://documents.egi.eu/document/417 417]  
| [https://documents.egi.eu/document/417 417]  
| 11. 08. 2011
| 08. 05. 2013
| QC Verification Templates
| QC Verification Templates
|}
|}
Line 52: Line 83:
== Metrics  ==
== Metrics  ==


{| cellspacing="0" cellpadding="3" border="1" style="border: 1px solid black;" class="wikitable sortable"
{| border="1" cellspacing="0" cellpadding="3" style="border: 1px solid black;" class="wikitable sortable"
|- align="left" style="background: none repeat scroll 0% 0% Lightgray;"
|- align="left" style="background: none repeat scroll 0% 0% Lightgray;"
! Metric  
! Metric  
Line 78: Line 109:
|+ Verification Cheat Sheet  
|+ Verification Cheat Sheet  
|-
|-
! width="100" align="left" | Product\Institute  
! width="200" align="left" | Product\Institute  
! width="75" | CESGA  
! width="75" | CESGA  
! width="75" | IFCA  
! width="75" | IFCA  
! width="75" | IFIC  
! width="75" | IFIC  
! width="75" | INFN  
! width="75" | INFN  
! width="75" | IN2P3
! width="75" | JÜLICH  
! width="75" | JÜLICH  
! width="75" | LIP  
! width="75" | LIP  
! width="75" | LRZ
! width="75" | UTC-N
! width="75" | CSC
! width="75" | GRNET
! width="75" | RAL
! width="75" | FZU
! width="300" style="background: none repeat scroll 0% 0% Gray;" | COMMENTS
! width="300" style="background: none repeat scroll 0% 0% Gray;" | COMMENTS
|-
|-
! align="left" | ARC CE  
! align="left" | EMI ARC CE  
|
|
|
|
|
|  
|  
| *  
| *  
|
|
| *
|
|
|
|
|-
! align="left" | EMI ARC compute cli
|
|
|
|  
|  
|  
|  
|  
|  
| *  
| *  
|
|
| *
|
|
|
|  
|  
|-
|-
! align="left" | ARC compute cli
! align="left" | EMI ARC Infosys
|
|
|
|
|
|
| *
|
|  
|  
| *  
| *  
Line 102: Line 171:
|  
|  
|  
|  
| *
|  
|  
|-
|-
! align="left" | ARC Infosys
! align="left" | EMI dCache
|
|
|
|
|
|
|  
|  
| *
|  
|  
|  
|  
|  
|  
| *  
| *  
|
|
|  
|  
|-
|-
! align="left" | dCache
! align="left" | EMI gLite MPI
| *
| *  
| *  
|
|
|
|
|
|
|  
|  
|  
|  
Line 123: Line 205:
|  
|  
|-
|-
! align="left" | gLite MPI
! align="left" | EMI CREAM_torque
| *  
| *  
| *  
| *  
|
|
|
|
|
|
|
|  
|  
|  
|  
Line 132: Line 221:
|  
|  
|-
|-
! align="left" | CREAM_torque
! align="left" | EMI CREAM_lsf
| *
|
|
|  
| *  
| *  
|
|
|
|
|
|  
|  
|  
|  
Line 141: Line 237:
|  
|  
|-
|-
! align="left" | CREAM_lsf
! align="left" | EMI EMIR
|
|
|
|  
|  
|  
|  
Line 148: Line 247:
|  
|  
|  
|  
|
|
|
| *
|  
|  
|-
|-
! align="left" | WMS  
! align="left" | EMI WMS  
| *  
| *  
|  
|  
| *  
| *  
|  
|  
|
|
|
|
|
|
| *
|  
|  
|  
|  
|  
|  
|-
|-
! align="left" | LB  
! align="left" | EMI LB  
| *  
| *  
|  
|  
| *  
| *  
|
|
|
|
|
|
|
|  
|  
|  
|  
Line 168: Line 285:
|  
|  
|-
|-
! align="left" style="background: none repeat scroll 0% 0% Red;" | FTS  
! align="left" | EMI FTS  
|  
|  
|  
|  
Line 175: Line 292:
|  
|  
|  
|  
| No verifiers/EAs available yet.
|  
|
|
|
|
| *
|
|
|-
|-
! align="left" | DPM  
! align="left" | EMI DPM  
| *  
| *  
|
|
|
|
|
|
|
|  
|  
|  
|  
Line 186: Line 317:
|  
|  
|-
|-
! align="left" | LFC_mySQL  
! align="left" | EMI LFC_mySQL  
| *  
| *  
|
|
|
|
|
|
|
|  
|  
|  
|  
Line 195: Line 333:
|  
|  
|-
|-
! align="left" style="background: none repeat scroll 0% 0% Red;" | LFC_Oracle  
! align="left" | EMI LFC_Oracle  
|
|
|
|
| *
|
|
|
|  
|  
|  
|  
Line 202: Line 348:
|  
|  
|  
|  
| No verifiers/EAs available yet. Oracle training required.
|-
|-
! align="left" | StoRM  
! align="left" | EMI StoRM  
|  
|  
| *  
| *  
|
|  
|  
|  
|  
|  
|  
| *  
| *  
|
|
|
|
|
|
|  
|  
|-
|-
! align="left" | APEL  
! align="left" | EMI APEL  
| *  
| *  
|
|
|
|
|
|
|
|  
|  
|  
|  
Line 222: Line 381:
|  
|  
|-
|-
! align="left" | DGAS  
! align="left" | EMI DGAS  
|  
|  
|  
|  
|  
|  
| *  
| *  
|
|
|
|
|
|
|
|  
|  
|  
|  
|  
|  
|-
|-
! align="left" | BDII  
! align="left" | EMI BDII  
|
|  
|  
|  
|  
Line 238: Line 405:
|  
|  
| *  
| *  
|
|
|
|
|
|
|  
|  
|-
|-
! align="left" | UI  
! align="left" | EMI UI  
| *  
| *  
|
|
|
|
|
|
|
|  
|  
|  
|  
Line 249: Line 429:
|  
|  
|-
|-
! align="left" | PX  
! align="left" | EMI PX  
|
|  
|  
|  
|  
Line 256: Line 437:
|  
|  
| *  
| *  
|
|
|
|
|
|
|  
|  
|-
|-
! align="left" | VOMS_mySQL  
! align="left" | EMI VOMS_mySQL  
|
|  
|  
|  
|  
Line 265: Line 453:
|  
|  
| *  
| *  
|
|
|
| *
|
|
|  
|  
|-
|-
! align="left" style="background: none repeat scroll 0% 0% Red;" | VOMS_Oracle  
! align="left" style="background: none repeat scroll 0% 0% Red;" | EMI VOMS_Oracle  
|  
|  
|  
|  
Line 274: Line 468:
|  
|  
|  
|  
| No verifiers/EAs available yet. Oracle training required.
|
|
|
|
|
|
|
| No verifiers available yet. Oracle training required.
|-
|-
! align="left" style="background: none repeat scroll 0% 0% Red;" | HYDRA  
! align="left" style="background: none repeat scroll 0% 0% Red;" | EMI HYDRA  
|
|  
|  
|  
|  
Line 283: Line 485:
|  
|  
|  
|  
| No verifiers/EAs available yet.
|
|
|
|
|
|
| No verifiers available yet.
|-
! align="left" style="background: none repeat scroll 0% 0% Red;" | EMI Wnodes
|
|
|
|
|
|
|
|
|
|
|
|
|
| No verifiers available yet.
|-
|-
! align="left" | ARGUS  
! align="left" style="background: none repeat scroll 0% 0% Red;" | EMI NAGIOS
|
|
|
|
|
|
|
|
|
|
|
|
|
| No verifiers available yet.
|-
! align="left" style="background: none repeat scroll 0% 0% Red;" | EMI Pseudonymity
|
|
|
|
|
|
|
|
|
|
|
|
|
| No verifiers available yet.
|-
! align="left" | EMI ARGUS  
| *  
| *  
|
|
|
|
|
|
|
|  
|  
|  
|  
Line 294: Line 557:
|  
|  
|-
|-
! align="left" | UNICORE TSI  
! align="left" | EMI UNICORE TSI  
|
|  
|  
|  
|  
Line 300: Line 564:
|  
|  
| *  
| *  
|
|
|
|
|
|
|  
|  
|  
|  
|-
|-
! align="left" | UNICORE WS  
! align="left" | EMI UNICORE WS  
|
|  
|  
|  
|  
Line 309: Line 580:
|  
|  
| *  
| *  
|
|
|
|
|
|
|  
|  
|  
|  
|-
|-
! align="left" | UNICORE Client  
! align="left" | EMI UNICORE Client  
|
|  
|  
|  
|  
Line 318: Line 596:
|  
|  
| *  
| *  
|
|
|
|
|
|
|  
|  
|  
|  
|-
|-
! align="left" | UNICORE Registry  
! align="left" | EMI UNICORE Registry  
|
|  
|  
|  
|  
Line 327: Line 612:
|  
|  
| *  
| *  
|
|
|
|
|
|
|  
|  
|  
|  
|-
|-
! align="left" | UNICORE Gateway  
! align="left" | EMI UNICORE Gateway  
|
|  
|  
|  
|  
Line 336: Line 628:
|  
|  
| *  
| *  
|
|
|
|
|
|
|  
|  
|  
|  
|-
|-
! align="left" | UNICORE Hila  
! align="left" | EMI UNICORE Hila  
|
|  
|  
|  
|  
Line 345: Line 644:
|  
|  
| *  
| *  
|
|
|
|
|
|
|  
|  
|  
|  
|-
|-
! align="left" | UNICORE xuudb  
! align="left" | EMI UNICORE xuudb  
|
|  
|  
|  
|  
Line 354: Line 660:
|  
|  
| *  
| *  
|
|
|
|
|
|
|  
|  
|  
|  
|-
|-
! align="left" | UNICORE Uvos  
! align="left" | EMI UNICORE Uvos  
|
|  
|  
|  
|  
Line 363: Line 676:
|  
|  
| *  
| *  
|
|
|
|
|
|
|
|
|-
! align="left" | EMI AMGA
|
|
|
| *
|
|
|
|
|
|
|
|
|  
|  
|  
|  
|-
|-
! align="left" | AMGA
! align="left" | IGE SAGA-SD
|
|
|
|  
|  
|  
|  
Line 374: Line 712:
| *  
| *  
|  
|  
|
|
|
| No verifiers available yet.
|-
|-
! align="left" style="background: none repeat scroll 0% 0% Red;" | SAGA-SD
! align="left" | IGE GRAM5
|
| *
|
|
|
|
|
|
|  
|  
|  
|  
Line 382: Line 732:
|  
|  
|  
|  
| No verifiers/EAs available yet.
|-
|-
! align="left" | GRAM5
! align="left" | IGE GSISSH
|  
|  
| *  
| *  
|
|
|
|
|
|
|
|  
|  
|  
|  
Line 393: Line 749:
|  
|  
|-
|-
! align="left" | GridFTP  
! align="left" | IGE GridFTP  
| *  
| *  
|
|
|
|
|
|
|
|  
|  
|  
|  
Line 402: Line 765:
|  
|  
|-
|-
! align="left" | myProxy  
! align="left" | IGE myProxy  
| *  
| *  
|
|
|
|
|
|
|
|  
|  
|  
|  
Line 411: Line 781:
|  
|  
|-
|-
! align="left" | RLS  
! align="left" | IGE RLS  
| *  
| *  
|
|
|
|
|
|
|
|  
|  
|  
|  
Line 420: Line 797:
|  
|  
|-
|-
! align="left" style="background: none repeat scroll 0% 0% Red;" | GridWay  
! align="left" | IGE GridWay  
| *
|
|
|
|
|
|
|
|  
|  
|  
|  
Line 427: Line 812:
|  
|  
|  
|  
| Training required. IGE Contact i.marin(at)fdi.ucm.es
|-
|-
! align="left" | SAM  
! align="left" | IGE SAM  
|
|
|
|
|
|
|
|
| *
|
|
|
|
|
|-
! align="left" | IGE OGSA-DAI
|
|
|
|
|
|
|
|
| *
|
|
|
|
|
|-
! align="left" | IGE Security
| *
|
|
|
|
|
|
|
|
|
|
|
|
|
|-
! align="left" | IGE Gridsafe
|
|
|
|
|
|
|
| *  
| *  
|  
|  
Line 441: Line 880:
<br>  
<br>  


== Verified Products  ==
[https://www.egi.eu/earlyAdopters/table Early Adopters list]


{| cellspacing="0" cellpadding="3" border="1" style="border: 1px solid black;" class="wikitable sortable"
[https://www.egi.eu/earlyAdopters/teams Early Adopters per UMD product]
|- align="left" style="background: none repeat scroll 0% 0% Lightgray;"
! RT link
! Verification Date
! More information
|-
| [https://rt.egi.eu/rt/Ticket/Display.html?id=460 460]  
| -
| CA update, version 1.37-1
|-
| [https://rt.egi.eu/rt/Ticket/Display.html?id=1125 1125]
| -
| CA update, version 1.38-1
|-
| [https://rt.egi.eu/rt/Ticket/Display.html?id=490 490]
| -
| SAM Update-6
|-
| [https://rt.egi.eu/rt/Ticket/Display.html?id=626 626]
| -
| SAM Update-7
|-
| [https://rt.egi.eu/rt/Ticket/Display.html?id=1281 1281]
| -
| SAM Update-9
|-
| [https://rt.egi.eu/rt/Ticket/Display.html?id=1729 1729]
| 11/04/11
| SAM Update-10
|-
| [https://rt.egi.eu/rt/Ticket/Display.html?id=1811 1811]
| 29/04/11
| SAM Update-10.1
|-
| [https://rt.egi.eu/rt/Ticket/Display.html?id=1984 1984]
| 20/05/11
| SAM Update-11
|-
| [https://rt.egi.eu/rt/Ticket/Display.html?id=2222 2222]
| 01/06/11
| EMI-UI 1.0.0
|}

Latest revision as of 18:51, 9 December 2014

Technology Software Component Delivery Software Provisioning UMD Middleware Cloud Middleware Distribution Containers Distribution Technology Glossary


Software Provisioning menu: Software Provisioning Process UMD Release Process Quality Assurance UMD Staged Rollout




Objective

The main objective of the TSA2.3 is to verify the quality of the software provided by the TP before entering the SR phase and going to production. By doing so we prevent that software that might work enters into the SR and even goes into production but that doesn´t follow the quality criteria defined in TSA2.2. Some of the reasons for doing the verification before the software enters the stage rollout are:

- Check that the bugs reported in the previous release of the software have been corrected (work in collaboration with DMSU) by the TP.

- Software can work well in the SR but might not have all the functionalities required

- Software might not be safe, well documented, or have the necessary installation rules or licenses

The Verification Process

When a new product is available, the TP has to follow the NSRW. Once the software is correctly uploaded to the repository, the release enters into the verification phase. The requirements are that the TP has to provide all the necessary information to the verifier (QCV) so that the QCV can assess that the TP has tested in advance the quality of the software. Depending on the type of release, different actions will be taken by the QCV, Verification process is described in detail in the EGI Verifier Guideline.

QC Verification Reports

RT workflow and Verification Templates links are available in this page. First of all verifiers must check QC service mapping to know which test must be verified for each product. This service mapping is available here QC Verification service mapping and Verification/Executive Summary templates are available here: QC Verification Templates.

Verifiers must fill all required fields for each product and write a Verification process summary into Executive Summary, this summary should include:

  • A short summary of the installation and configuration process: Installation and configuration were successful?, If not explain any issue or bug found. If the product was rejected explain why.
  • If its necessary Verifiers should include a comment for StageRollout: Configuration changes or minor issues found verifying the product.
  • If a new QC is necessary and is not included, Verifiers must write a comment to SA2.2 to change current QC.


UMD Release Candidate Testing

Before each UMD release the verification team checks the new RC. To perform this task SA2.3 VMs have included the RC_testing script. This script is available after each SA2 VM instantiation (into /root/configuration_templates/tools directory). This script is able to detect any package inconsistency after UMD RC repo configuration. The RC testing process is as follows:

  • Verifier should instantiate different Linux flavours VM. For UMD case SL5, SL6 and debian.
  • Install UMD RC repo files from UMDx RC page.
  • Check RC_tester "PRODUCTS" array. This array must be updated to include all UMD products!
  • It is recommended to use screen program before RC_tester execution. RC tests take about 2h/3h to finish (depends on OS used and the number of products).
  • Run RC_tester and follows its instructions.
  • RC_tester generates several logs into log directory, after its execution SA2 verifier should check:
    • installator_OK.log: List of metapackages installed correctly.
    • installator_ERROR.log: List of metapackages which contain any issue.
    • <PRODUCT_NAME>_OUTPUT.log: Complete Product installation log.
    • <PRODUCT_NAME>_postupdate.log: Info about product update execution. It detects any issue updating current UMD products.


Verification Work Effort Metrics

Each verification process and change is registered by EGI RT. A complete report is generated on a daily basis each midnight. The XLS file can be downloaded from here SA2 Verification Metrics


Verification Team

Verifiers team is updated for each UMD release and it is available here: SA2.3 Verifiers List. All verifiers are included in the SSO sw-rel-qc group that includes a mailing list and permissions to modify RT tickets related with the verification process.

Reference Documents

DocDB link Release Date Document
QCv6 10 2013 UMD Quality Criteria
QCv5 08. 05. 2013 UMD Quality Criteria
418 08. 05. 2013 UMD Products service mapping
417 08. 05. 2013 QC Verification Templates

Metrics

Metric Description
M.SA2.4 Number of new releases validated against defined criteria: Measures the workload on the validation team.
M.SA2.5 Mean time taken to validate a releas: Indicates how responsive the team is to validating releases.
M.SA2.6 Number of releases failing validation: Indicates the quality assurance process of the software providers.

SLAs

EGI SLAs negotiated with the TPs

EMI-EGI SLA

Verification engineer skill matrix

Verification Cheat Sheet
Product\Institute CESGA IFCA IFIC INFN IN2P3 JÜLICH LIP LRZ UTC-N CSC GRNET RAL FZU COMMENTS
EMI ARC CE * *
EMI ARC compute cli * *
EMI ARC Infosys * *
EMI dCache *
EMI gLite MPI * *
EMI CREAM_torque * *
EMI CREAM_lsf *
EMI EMIR * *
EMI WMS * * *
EMI LB * *
EMI FTS *
EMI DPM *
EMI LFC_mySQL *
EMI LFC_Oracle *
EMI StoRM * *
EMI APEL *
EMI DGAS *
EMI BDII *
EMI UI *
EMI PX *
EMI VOMS_mySQL * *
EMI VOMS_Oracle No verifiers available yet. Oracle training required.
EMI HYDRA No verifiers available yet.
EMI Wnodes No verifiers available yet.
EMI NAGIOS No verifiers available yet.
EMI Pseudonymity No verifiers available yet.
EMI ARGUS *
EMI UNICORE TSI *
EMI UNICORE WS *
EMI UNICORE Client *
EMI UNICORE Registry *
EMI UNICORE Gateway *
EMI UNICORE Hila *
EMI UNICORE xuudb *
EMI UNICORE Uvos *
EMI AMGA *
IGE SAGA-SD * No verifiers available yet.
IGE GRAM5 *
IGE GSISSH *
IGE GridFTP *
IGE myProxy *
IGE RLS *
IGE GridWay *
IGE SAM *
IGE OGSA-DAI *
IGE Security *
IGE Gridsafe *


Early Adopters list

Early Adopters per UMD product