Difference between revisions of "UMDQualityCriteria"

From EGIWiki
Jump to: navigation, search
(Identified Capabilities)
 
(16 intermediate revisions by 2 users not shown)
Line 1: Line 1:
 +
== UMD Quality Criteria ==
 +
 
All the software included in the Unified Middleware Distribution (UMD) must meet a set of Quality Criteria defined by EGI. The Quality Criteria can be classified into generic criteria, i.e. criteria which sould hold for any component of the UMD, and specific criteria, i.e. criteria valid for a particular component only.
 
All the software included in the Unified Middleware Distribution (UMD) must meet a set of Quality Criteria defined by EGI. The Quality Criteria can be classified into generic criteria, i.e. criteria which sould hold for any component of the UMD, and specific criteria, i.e. criteria valid for a particular component only.
  
== Validation of Criteria ==
+
Any Technology Provider (TP) willing to have their software verified by the SA2 team should carefully read them.
 
 
In order to be verified, quality criteria is specified as a set of tests. Those tests must ensure the correctness, completeness and security of each service. Software providers must include with each component a complete test plan that covers all the quality criteria. The test plan is composed by:
 
* General description of the test plan.
 
* A Test Suite following the template described in the next section for each of the test cases included in:
 
** Generic criteria.
 
** Specific criteria applicable to the component.
 
** Bugs detected by EGI
 
 
 
=== Template for test validation ===
 
 
 
For each test specified in the criteria, there must be a report. The report should contain several sections:
 
==== Description ====
 
Description of the test objectives. Include any references to specific bugs that are tested.
 
 
 
==== Set up ====
 
 
 
* A description of required steps prior to the execution of the test, and if relevant, differences between this set up and the standard installation procedure for the software. Tests should be written in such a way that they can be integrated into different frameworks. Manual testing should be avoided.
 
 
 
* Tests must be fully configurable either by options to the test script or a configuration file (key/value pairs resp. environment variables for export). There must be no hardcoded paths etc.
 
 
 
* If the test has prerequisites (e.g. a valid proxy) this must be documented and a check at the beginning of the test has to be implemented.
 
 
 
* Detailed documentation on how to run the test, that allows the validation team to repeat the test if necessary.
 
 
 
==== Results ====
 
Tests should have a simple output (text or simple html) that allows adding a post processing step to the results in order to be transformed for use with a particular presentation framework. Description of the test results for these cases:
 
 
 
===== Correct input/Normal workflow =====
 
Testing the software as it should work, with correct input. Describe different inputs combinations used for testing and when this test should pass or fail. Description of the output observed when running the test (most of the time a copy and paste of the commands or screenshot should be enough) that is reproducible by the validation team.
 
 
 
===== Error workflow - erroneous input =====
 
Testing the error codes: providing wrong input to the software should return the proper error messages. Describe different inputs combinations used for testing and when this test should pass or fail. Description of the output observed when running the test (most of the time a copy and paste of the commands or screenshot should be enough) that is reproducible by the validation team.
 
 
 
==== Non tested features ====
 
Describe in this section any features of the software that miss a test.
 
 
 
== Generic acceptance criteria ==
 
 
 
[[EGI-InSPIRE:UMDQualityCriteria:Generic]]
 
 
 
== Specific acceptance criteria ==
 
 
 
'''OLD''': There is a template for the creation of new critera at [[EGI-InSPIRE:UMDQualityCriteria:Template]]
 
 
 
=== Defined Capabilities ===
 
 
 
These capabilities are identified and defined in the current [https://documents.egi.eu/secure/ShowDocument?docid=100 UMD RoadMap]:
 
 
 
* Functional Capabilities
 
** [[EGI-InSPIRE:UMDQualityCriteria:InformationDiscovery | Information Discovery ]]
 
** [[EGI-InSPIRE:UMDQualityCriteria:ComputingServices | Compute]]
 
** [[EGI-InSPIRE:UMDQualityCriteria:ComputeJobScheduling | Compute Job Scheduling ]]
 
** [[EGI-InSPIRE:UMDQualityCriteria:FileAccess | File Access]] '''OLD''' [[EGI-InSPIRE:UMDQualityCriteria:StorageServices]]
 
** [[EGI-InSPIRE:UMDQualityCriteria:FileTransfer | File Transfer]] 
 
** [[EGI-InSPIRE:UMDQualityCriteria:ParallelJob | Parallel Job]] 
 
 
 
* Security Capabilities ('''OLD''' [[EGI-InSPIRE:UMDQualityCriteria:SecurityServices | security services]])
 
** [[EGI-InSPIRE:UMDQualityCriteria:Authentication | Authentication ]]
 
** [[EGI-InSPIRE:UMDQualityCriteria:CredentialManagement | Credential Management ]]
 
** [[EGI-InSPIRE:UMDQualityCriteria:UserManagement | User Management ]]
 
** [[EGI-InSPIRE:UMDQualityCriteria:Authorization | Authorization ]]
 
** [[EGI-InSPIRE:UMDQualityCriteria:CA | CAs]]
 
 
 
* Operational Capabilities
 
** [[EGI-InSPIRE:UMDQualityCriteria:Monitoring | Monitoring (Nagios)]]
 
** [[EGI-InSPIRE:UMDQualityCriteria:Accounting | Accounting]]
 
  
=== Identified Capabilities ===
+
=== QC Documents ===
These capabilities are identified in the [https://documents.egi.eu/secure/ShowDocument?docid=100 UMD RoadMap] but still waiting for EGI Community input for definition:
 
  
==== Functional Capabilities ====
+
The Quality Criteria (QC) is written following the classification of capabilities defined in can the [https://documents.egi.eu/secure/ShowDocument?docid=272 UMD RoadMap], one document for each type of capability. Take into account that a software component may cover QC specified in more than one of those documents.
* File Encryption/Decryption
 
  
: Sensitive data needs to be stored securely. Before being stored in a remote file store the file may need to be encrypted and then on retrieval de-encrypted before use. The capability should also provide solutions relating to the storage of the keys needed to perform these tasks.
+
=== QC Roadmap ===
  
* Metadata Catalogue
+
QC definition is a continuous process driven by the requirements of users and operations communities, however the verification of new software releases will be done against fixed releases of the QC documents. At least every 6 months a new major version of the QC will be released, although more frequent releases may occur if necessary.
: The metadata catalogue is used to store and query information relating to the data (files, databases, etc.) stored within the production infrastructure.  
 
  
* Database Access
+
Each new release of the QC will be announced in this page and the TCB mailing list. Along with the announcement of each release of QC specification, a date for its use in verification will be specified. It is expected that verification will use new releases of the QC within 2 to 4 weeks from the release date.
: Many communities are moving to the use of structured data stored in relational databases. These need to be accessible for controlled use by remote users as any other e-Infrastructure resource.
 
  
* File Transfer Scheduling
+
Each QC release will contain documentation with:
: The bandwidth linking resource sites is a resource that needs to be managed in the same way compute resources at a site are accessed through a job scheduler. By being able to schedule wide area data transfers, requests can be prioritised and managed. This would include the capability to monitor and restart transfers as required.
+
* major changes introduced in the version.
 +
* criteria to be deprecated in next release of the QC, if any.
  
* Remote Instrumentation
+
As soon as a major update of the QC is released, the next version will be made available as draft in the DocDB. This draft will allow the TP to plan their testing efforts.  
: Instruments are data sources frequently encountered within e-Infrastructures. As part of a distributed computing architecture providing remote access to manage and monitor these instruments is becoming increasingly important within some communities.
 
  
* Workflow
+
==== Current QC Specification ====
: The ability to define, initiate, manage and monitor a workflow is a key capability across many user communities. It is also a capability that can be deployed by a user or a user community (i.e. it does not need to be a service provided as part of the core infrastructure) but the various workflow systems may have requirements that need to be supported within the core infrastructure.
 
  
==== Operational Capabilities ====
+
Current QC Specification can be found at the [https://documents.egi.eu/public/ShowDocument?docid=240 QC document] in EGI DocDB.
* Virtual Image Management
+
Check the [[EGI-InSPIRE:UMDQualityCriteria:QC1 | release notes]].
: As virtual machine images become the default approach to providing the environment for both jobs and services, increased effort is needed on building the trust model around the distribution of images. Resource providers will need a mechanism for images to be distributed, cached and trusted for execution on their sites.  
 
  
* Virtual Machine Management
+
==== Future Releases ====
: The core functionality is for authorized users to manage the virtual machine life-cycle and configuration on a remote site (i.e. start, stop, pause, etc.) Machine images would be selected from a trusted repository at the site that would be configured according to site policy. Together this would allow site managers to determine both who could control the virtual machines running on their sites and who generated the images used on their site.
 
  
* Messaging
+
{| class="wikitable sortable"  style="border:1px solid black" cellspacing="0" cellpadding="3" border="1"
: Within distributed systems, a message ‘bus’ provides a reliable mechanism for data items to be sent between producers and (multiple) consumers. Such a capability, once established, can be reused by many different software services.
+
|- style="background:Lightgray"  align="left"
 +
! DocDB link
 +
! Expected Release Date
 +
! Expected Verification Date
 +
! More information
 +
|-
 +
| [https://documents.egi.eu/public/ShowDocument?docid=240 240]
 +
| 10. 02. 2011
 +
| 10. 02. 2011
 +
| [[EGI-InSPIRE:UMDQualityCriteria:QC1 | release notes]]
 +
|-
 +
| -
 +
| 01. 08. 2011
 +
| 15. 08. 2011
 +
| -
 +
|}
  
 +
==== Past QC releases ====
  
[[Category:WP5-SA2]]
+
There are no past QC releases.

Latest revision as of 22:11, 24 December 2014

UMD Quality Criteria

All the software included in the Unified Middleware Distribution (UMD) must meet a set of Quality Criteria defined by EGI. The Quality Criteria can be classified into generic criteria, i.e. criteria which sould hold for any component of the UMD, and specific criteria, i.e. criteria valid for a particular component only.

Any Technology Provider (TP) willing to have their software verified by the SA2 team should carefully read them.

QC Documents

The Quality Criteria (QC) is written following the classification of capabilities defined in can the UMD RoadMap, one document for each type of capability. Take into account that a software component may cover QC specified in more than one of those documents.

QC Roadmap

QC definition is a continuous process driven by the requirements of users and operations communities, however the verification of new software releases will be done against fixed releases of the QC documents. At least every 6 months a new major version of the QC will be released, although more frequent releases may occur if necessary.

Each new release of the QC will be announced in this page and the TCB mailing list. Along with the announcement of each release of QC specification, a date for its use in verification will be specified. It is expected that verification will use new releases of the QC within 2 to 4 weeks from the release date.

Each QC release will contain documentation with:

  • major changes introduced in the version.
  • criteria to be deprecated in next release of the QC, if any.

As soon as a major update of the QC is released, the next version will be made available as draft in the DocDB. This draft will allow the TP to plan their testing efforts.

Current QC Specification

Current QC Specification can be found at the QC document in EGI DocDB. Check the release notes.

Future Releases

DocDB link Expected Release Date Expected Verification Date More information
240 10. 02. 2011 10. 02. 2011 release notes
- 01. 08. 2011 15. 08. 2011 -

Past QC releases

There are no past QC releases.