EGI Quality Criteria Verification
|Technology||Software Component Delivery||Software Provisioning||UMD Middleware||Cloud Middleware Distribution||Containers Distribution||Technology Glossary|
|Software Provisioning menu:||Software Provisioning Process||UMD Release Process||Quality Assurance||UMD||Staged Rollout|
The main objective of the TSA2.3 is to verify the quality of the software provided by the TP before entering the SR phase and going to production. By doing so we prevent that software that might work enters into the SR and even goes into production but that doesn´t follow the quality criteria defined in TSA2.2. Some of the reasons for doing the verification before the software enters the stage rollout are:
- Check that the bugs reported in the previous release of the software have been corrected (work in collaboration with DMSU) by the TP.
- Software can work well in the SR but might not have all the functionalities required
- Software might not be safe, well documented, or have the necessary installation rules or licenses
The Verification Process
When a new product is available, the TP has to follow the NSRW. Once the software is correctly uploaded to the repository, the release enters into the verification phase. The requirements are that the TP has to provide all the necessary information to the verifier (QCV) so that the QCV can assess that the TP has tested in advance the quality of the software. Depending on the type of release, different actions will be taken by the QCV, Verification process is described in detail in the EGI Verifier Guideline.
QC Verification Reports
RT workflow and Verification Templates links are available in this page. First of all verifiers must check QC service mapping to know which test must be verified for each product. This service mapping is available here QC Verification service mapping and Verification/Executive Summary templates are available here: QC Verification Templates.
Verifiers must fill all required fields for each product and write a Verification process summary into Executive Summary, this summary should include:
- A short summary of the installation and configuration process: Installation and configuration were successful?, If not explain any issue or bug found. If the product was rejected explain why.
- If its necessary Verifiers should include a comment for StageRollout: Configuration changes or minor issues found verifying the product.
- If a new QC is necessary and is not included, Verifiers must write a comment to SA2.2 to change current QC.
UMD Release Candidate Testing
Before each UMD release the verification team checks the new RC. To perform this task SA2.3 VMs have included the RC_testing script. This script is available after each SA2 VM instantiation (into /root/configuration_templates/tools directory). This script is able to detect any package inconsistency after UMD RC repo configuration. The RC testing process is as follows:
- Verifier should instantiate different Linux flavours VM. For UMD case SL5, SL6 and debian.
- Install UMD RC repo files from UMDx RC page.
- Check RC_tester "PRODUCTS" array. This array must be updated to include all UMD products!
- It is recommended to use screen program before RC_tester execution. RC tests take about 2h/3h to finish (depends on OS used and the number of products).
- Run RC_tester and follows its instructions.
- RC_tester generates several logs into log directory, after its execution SA2 verifier should check:
- installator_OK.log: List of metapackages installed correctly.
- installator_ERROR.log: List of metapackages which contain any issue.
- <PRODUCT_NAME>_OUTPUT.log: Complete Product installation log.
- <PRODUCT_NAME>_postupdate.log: Info about product update execution. It detects any issue updating current UMD products.
Verification Work Effort Metrics
Each verification process and change is registered by EGI RT. A complete report is generated on a daily basis each midnight. The XLS file can be downloaded from here SA2 Verification Metrics
Verifiers team is updated for each UMD release and it is available here: SA2.3 Verifiers List. All verifiers are included in the SSO sw-rel-qc group that includes a mailing list and permissions to modify RT tickets related with the verification process.
|DocDB link||Release Date||Document|
|QCv6||10 2013||UMD Quality Criteria|
|QCv5||08. 05. 2013||UMD Quality Criteria|
|418||08. 05. 2013||UMD Products service mapping|
|417||08. 05. 2013||QC Verification Templates|
|M.SA2.4||Number of new releases validated against defined criteria: Measures the workload on the validation team.|
|M.SA2.5||Mean time taken to validate a releas: Indicates how responsive the team is to validating releases.|
|M.SA2.6||Number of releases failing validation: Indicates the quality assurance process of the software providers.|
EGI SLAs negotiated with the TPs
Verification engineer skill matrix
|EMI ARC CE||*||*|
|EMI ARC compute cli||*||*|
|EMI ARC Infosys||*||*|
|EMI gLite MPI||*||*|
|EMI VOMS_Oracle||No verifiers available yet. Oracle training required.|
|EMI HYDRA||No verifiers available yet.|
|EMI Wnodes||No verifiers available yet.|
|EMI NAGIOS||No verifiers available yet.|
|EMI Pseudonymity||No verifiers available yet.|
|EMI UNICORE TSI||*|
|EMI UNICORE WS||*|
|EMI UNICORE Client||*|
|EMI UNICORE Registry||*|
|EMI UNICORE Gateway||*|
|EMI UNICORE Hila||*|
|EMI UNICORE xuudb||*|
|EMI UNICORE Uvos||*|
|IGE SAGA-SD||*||No verifiers available yet.|