EGI Verifier Guideline QC5

Technology

Software Component Delivery

Software Provisioning

UMD Middleware

Cloud Middleware Distribution

Containers Distribution

Technology Glossary

Quality Assurance |

Quality Criteria Definition |

Quality Criteria Dissemination |

Quality Criteria Verification |

Verifier Guideline |

Verification Testbed |

Glossary

The main objective of this guideline is to aid new SA2 verifiers to complete Verification process successfully.

Verification Preconditions

When a new product is available, the TP has to follow the Software Provisioning Process. Once the software is correctly uploaded to the repository, the release enters into the verification phase. The requirements are that the TP has to provide all the necessary information to the verifier (QCV) so that the QCV can assess that the TP has tested in advance the quality of the software. Depending on the type of release, different actions will be taken by the QCV.

The verification process starts when the following pre-conditions are met:

1. RT ticket is in state Open.
2. The RolloutProgress is set to Unverified.
3. CommunicationStatus is OK.
4. Owner is set to nobody.

If these conditions are reached then Verification process may be started.

Verification Start

Once the verification ticket meets the preconditions described above, the QCV must perform the following steps:

1. Set RT ticket Owner with the current QCV.
2. Set UMDRelease to the appropriate UMD Release (currently 1) and save the state.
3. Changes RolloutProgress to In Verification to start the actual verification.

Verification Template

Each product has a specific template that includes all QC that the product must comply with. Templates are available at Verification Reports and Executive Summary templates and are created according to the QC products mapping. These documents are updated if new UMD Quality Criteria is released.

The fields to fill in the template are:

• Accepted:
  • Y, when the product meets the criteria
  • N, when the product does not meet the criteria,
  • NA, when the criteria is not applicable for the product (e.g. API documentation for products without a public API)

• Tested:
  • TP, when the criteria was tested by the Technology Provider and the validator trusts the results of the tests.
  • or VLD, when the criteria was tested by the validation team

• Comments: include here any relevant comments or links to more information for the specified criteria.

Criteria Verification

Level of Testing

The product in verification must be installed and tested in the EGI Verification Testbed, however the verification process is different for UMD Minor or Major releases:

• Major releases (may not be backwards compatible):
  • Verifier MUST actively assess all assigned QCs (executing specific tests, reading available documentation, etc).
  • Product installation from scratch (or upgrade if It's supported by the product).

• Minor releases (backwards compatible):
  • Verifiers only checks QCs affected by update changes.
  • Package update installation and verification.
  • Product installation from scratch.

Holding Verification

SOME WORDS HERE ABOUT THE PROCESS OF VERIFICATION HOLDING

Product Acceptance

QCs tests are Mandatory (M) or Optional (O).

• A product is REJECTED if it fails the installation or configuration process.
• A product is REJECTED if it fails ANY Mandatory QC.
• A product is VERIFIED if it pass ALL assigned QCs.
• A product is VERIFIED if it fails ANY Optional QC.

I have the impression that mandatory/optional mixes two aspects of the verification:
a) The verifier's obligation to execute a QC test (mandatory vs. optional), and
b) The verification outcome when a fault is detected (critical vs. non-critical)

Assuming the following definition for CRITICAL/NON-CRITICAL:
A CRITICAL Quality Criterion MUST will cause immediate and unconditional rejection of the tested product if a failure of any type is detected during testing. 
A NON-CRITICAL Quality Criterion may still lead to acceptance of the tested product if a documented(!) workaround to avoid or mitigate the failure exists.

And using the definition of MANDATORY/OPTIONAL:
A MANDATORY Quality Criterion must be assessed or verified for any software release supplied by a Technology Provider. An OPTIONAL Quality Criterion may be
assessed or tested by the verifier.
 
With the definition of ASSESSING/TESTING as:
"A verifier is ASSESSING a Quality Criterion by analysing documentation (release notes, known bugs list, documented changes, test results, etc.) supplied 
by the Technology Provider without actual independent test execution. A Quality Criteron is TESTED if the verifier independently executes the test associated 
to the Quality Criterion using the same sources of documentation, but an test infrastucture independent from the Technology Provider."

Those aspects may provide you with a matrix guide for verifiers to produce reliable and fair outcomes:
'''1) MANDATORY and CRITICAL'''''' QC''': '''MUST''' be assessed by the verifier, and '''MUST''' be tested by the verifier for any software release (whether major, minor or revision release.
'''2) MANDATORY and NON-CRITICAL QC''': '''MUST''' be assessed for minor and revision releases, '''MAY''' be tested for minor releases (depending on other circumstances) a '''MUST''' be tested for major releases.  
'''3) OPTIONAL and NON-CRITICAL QC:''' '''MAY''' be assessed for any release, and '''MAY''' be tested for any release, if time permits.
'''4) OPTIONAL and CRITICAL QC''': This combination does not make sense.

For all cases 1 - 3 the criticality determines the outcome of the verification based on detected failures.

Verification Summary

When the Verification template is finished an Executive Summary is created, following the template available at Verification Reports and Executive Summary templates. This document includes:

• A summary of tests failed and passed (Critical and non critical).
• A list of comments for other teams involved in the Software Provisioning process (SR, QC, )

Publication of Verification Reports

In order to store the verification reports, a new space must be created in the DocumentDB. The new document must comply with the following information:

• Title: QC Verification Report: <PRODUCT_NAME>-<VERSION>.
• Files to include:
  • First file field is used to upload Executive Summary doc. Executive Summary file names should have this nomenclature: QC_Verification--Executive_Summary_<PRODUCT_NAME>_<VERSION>.doc.
  • Second file filed is used to upload QC Verification report. Verification report file names should have this nomenclature: QC_Verification--Report_<PRUDUCT_NAME>_<VERSION>.doc .
• Keywords field: Space separated keyword names, must include Quality, Criteria, Verification, etc.
• Media type must be set to Document.
• status must be set to FINAL.
• View must be set to public.
• Modify should be set to inspire-SA2.
• TOPICS space field must be set to Software Provisioning Verification Reports and WP5.

The generated document DB link must be included into the QualityCriteriaVerificationReport field of the RT ticket of the verification.

End of Verification

Once the reports are stored in the doc DB and linked in the RT ticket, the result of the verification must be set by changing the RolloutProgress field of the ticket. If the product met is accepted then change this field to the "StageRollout" value. This will cause the Rollout teams to continue with the software provisioning process. If the product is not accepted, then change the value to "Rejected", causing the process to stop.