Alert.png The wiki is deprecated and due to be decommissioned by the end of September 2022.
The content is being migrated to other supports, new updates will be ignored and lost.
If needed you can get in touch with EGI SDIS team using operations @ egi.eu.

EGI Verifier Guideline QC5

From EGIWiki
Revision as of 09:52, 26 May 2011 by Asimon (talk | contribs)
Jump to navigation Jump to search
Technology Software Component Delivery Software Provisioning UMD Middleware Cloud Middleware Distribution Containers Distribution Technology Glossary


Quality Assurance | Quality Criteria Definition | Quality Criteria Dissemination | Quality Criteria Verification | Verifier Guideline | Verification Testbed | Glossary





Objective

The main objective of this guideline is to aid new SA2 verifiers to complete Verification process successfully. This guideline includes these topics:

  • How to request and manage a new SA2 testbed machine.
  • How to handle product RT information and install new middleware.
  • How to use QC verification reports:
    • Which QC tests must be checked.
    • How to fill QC reports and Executive Summaries.
  • When and how a product is rejected or accepted.
    • Differences between critical and non critical bugs.
    • Mandatory and Optional QCs.


SA2 Verification Testbed

Each released product MUST be installed in different machines. SA2 Verifiers should request a new virtual machine for verification test submitting an email to grid-admin(at)cesga.es. This email should include:

  • Subject: SA2 Verification Tesbed request.
  • Body:
    • Verifier Name.
    • Product name to verify.
    • OS and hardware requirements (if necessary)
    • Verifier SSH public key (generated id_dsa.pub)

Verification team will submit a confirmation email to connect to the new machine (public IP, hostname, etc).

New VMs have a limited lifetime of 1 week, after this period the VM is destroyed.


Middleware Installation

Verifiers should be able to connect new created VM machines to start verification process, these machines include:

  • NTP configuration.
  • EPEL and EGI-trustanchors repositories.
  • Host certificate (hostcert.pem and hostkey.pem)


Verifier must install a new repo for each product, this information is available at RT ticket field:

  • Custom Fields->RepositoryURL

Create a new repo file into /etc/yum.repos.d, as example:

  • cd /etc/yum.repos.d
  • vi test_sa2.repo

[arc-ce]

name=arc-ce

baseurl=http://admin-repo.egi.eu/sw/unverified/emitest.arc-ce.sl5.x86_64/1/0/0/

enabled=1

#To use priorities you must have yum-priorities installed

priority=30

  • yum update

Now verifier must install the product metapackage which is included into RT release.xml file (Custom Fields->ReleaseMetadata). An example: rt.egi.eu/rt/Download/CustomFieldValue/8712/release-voms-ppa.xml. In this example emi-voms-mysql is the package required for VOMS-Mysql product installation.

  • yum install emi-voms-mysql

Verifier must check missing dependencies or packages. If any issue is found installing the new middleware the Verifier must open a GGUS ticket directly to product Technology Provider.


QC Verification Reports

RT workflow and Verification Templates links are available in: Quality Criteria Verification. First of all verifiers must check QC service mapping to know which test must be verified for each product. This service mapping is available here QC Verification service mapping and Verification/Executive Summary templates are available here: QC Verification Templates.

Verifiers must fill all required fields for each product and write a Verification process summary into Executive Summary, this summary should include:

  • A short summary of the installation and configuration process: Installation and configuration were successful?, If not explain any issue or bug found. If the product was rejected explain why.
  • If its necessary Verifiers should include a comment for StageRollout: Configuration changes or minor issues found verifying the product.
  • If a new QC is necessary and is not included, Verifiers must write a comment to SA2.2 to change current QC.


Acceptance or Rejection

QCs tests are Mandatory (M) or Optional (O).

  • A product is REJECTED if it fails the installation or configuration process.
  • A product is REJECTED if it fails ANY Mandatory QC.
  • A product is VERIFIED if it pass ALL assigned QCs.
  • A product is VERIFIED if it fails ANY Optional QC.
I have the impression that mandatory/optional mixes two aspects of the verification:
a) The verifier's obligation to execute a QC test (mandatory vs. optional), and
b) The verification outcome when a fault is detected (critical vs. non-critical)

Assuming the following definition for CRITICAL/NON-CRITICAL:
A CRITICAL Quality Criterion MUST will cause immediate and unconditional rejection of the tested product if a failure of any type is detected during testing. 
A NON-CRITICAL Quality Criterion may still lead to acceptance of the tested product if a documented(!) workaround to avoid or mitigate the failure exists.

And using the definition of MANDATORY/OPTIONAL:
A MANDATORY Quality Criterion must be assessed or verified for any software release supplied by a Technology Provider. An OPTIONAL Quality Criterion may be
assessed or tested by the verifier.
 
With the definition of ASSESSING/TESTING as:
"A verifier is ASSESSING a Quality Criterion by analysing documentation (release notes, known bugs list, documented changes, test results, etc.) supplied 
by the Technology Provider without actual independent test execution. A Quality Criteron is TESTED if the verifier independently executes the test associated 
to the Quality Criterion using the same sources of documentation, but an test infrastucture independent from the Technology Provider."

Those aspects may provide you with a matrix guide for verifiers to produce reliable and fair outcomes:
'''1) MANDATORY and CRITICAL'''''' QC''': '''MUST''' be assessed by the verifier, and '''MUST''' be tested by the verifier for any software release (whether major, minor or revision release.
'''2) MANDATORY and NON-CRITICAL QC''': '''MUST''' be assessed for minor and revision releases, '''MAY''' be tested for minor releases (depending on other circumstances) a '''MUST''' be tested for major releases.  
'''3) OPTIONAL and NON-CRITICAL QC:''' '''MAY''' be assessed for any release, and '''MAY''' be tested for any release, if time permits.
'''4) OPTIONAL and CRITICAL QC''': This combination does not make sense.

For all cases 1 - 3 the criticality determines the outcome of the verification based on detected failures.


Minor or Major releases

QC verification is different for UMD Minor or Major releases:

  • Major releases (not backwards compatible):
    • Verifier MUST actively assess all assigned QCs in all middleware products. (executing specific tests, read available documentation, etc).
    • Product installation from scratch (or upgrade if It's supported by the product).


  • Minor releases (backwards compatible):
    • Verifiers only checks QCs affected by update changes.
    • Package update installation and verification.
    • Product installation from scratch.


Verifiers List

Verifiers list is updated for each UMD release and it is available here: SA2.3 Verifiers List


SA2 UMD1.0 Testbed

Service Host
topbdii topbdii02.ncg.ingrid.pt
site bdii sbdii02.ncg.ingrid.pt
voms server voms02.ncg.ingrid.pt
arc-ce test03.egi.cesga.es
arc info test04.egi.cesga.es
WMS tst04.ific.uv.es
LB tst05.ific.uv.es
UI test13.egi.cesga.es