Alert.png The wiki is deprecated and due to be decommissioned by the end of September 2022.
The content is being migrated to other supports, new updates will be ignored and lost.
If needed you can get in touch with EGI SDIS team using operations @ egi.eu.

Difference between revisions of "UMDQualityCriteria"

From EGIWiki
Jump to navigation Jump to search
Line 1: Line 1:
= Generic acceptance criteria =
= UMD Quality Criteria =


== Documentation ==
All the software included in the Universal Middleware Distribution (UMD) must meet a set of Quality Criteria defined by EGI. The Quality Criteria can be classified into generic criteria, i.e. criteria which sould hold for any component of the UMD, and specific criteria, i.e. criteria valid for a particular component only.
 
In order to be verified, quality criteria is specified as a set of tests. Those tests must ensure the correctness, completeness and security of each service. Software providers must include with each component a complete test plan that covers all the quality criteria. The test plan is composed by:
* Documentation on how to run the tests
* A test suite for each of the generic criteria
* A test suite for each of the specific criteria applicable to the component
* Tests results for all the specified tests
 
In the case of revision releases, the test plan must include specific tests for the bugs fixed in the release.
 
== Generic acceptance criteria ==
 
=== Documentation ===
Services in UMD must include a comprehensive documentation written in a uniform and clear style, which reflects all of the following items:
Services in UMD must include a comprehensive documentation written in a uniform and clear style, which reflects all of the following items:
* Functional description of the software.
* Functional description of the software.
Line 12: Line 24:
'''Verification''': existence of the documentation with all the required items.
'''Verification''': existence of the documentation with all the required items.


== Testability ==
=== Source Code Quality and Availability ===
UMD software must support testing of its features and functionality. The tests must ensure the correctness, completeness, security and scalability of each service. Error situation and input validation have to be included in the test suite. The software provider must define a test plan for each service and provide a test suite and the results of running it against each new release. Interoperability with the rest of the UMD software must be explicitly tested. A global test suite for the UMD distribution will guarantee the correct behavior of the complete set of services in a controlled environment.  
The source code of each component of the UMD middleware should follow a coherent and clear programming style that helps in the readability of the code and eases maintenance, testing, debugging, fixing, modification and portability of the software. Open source components must publicly offer their source code and the license with the binaries.


'''Verification''': existence of the test plan, test suite and the results of the test plan.
'''Verification''': for Open Source components, availability of the code and license. Source code quality metrics are desirable.


== Configuration ==  
=== Management, Monitoring, Traceability ===
Tools for the automatic or semi-automatic configuration of the services must be provided with the software. These tools should allow the unassisted configuration of the services for the most common use casses while being customizable for advanced use cases. Complete manual configuration must be always allowed.


'''Verification''': test suite must include the configuration mechanisms and tools. Yaim is considered as the preferred tool.
All the services must include tools related to:
 
== Management, Monitoring, Traceability==
All the services must include tools (and tests for them) related to:
* Starting, stopping, suspending, listing and querying the status of all the service daemons.
* Starting, stopping, suspending, listing and querying the status of all the service daemons.
* Checking the responsiveness of all the service components or daemons (expected ports open and expected answer to commands received)
* Checking the responsiveness of all the service components or daemons  
* Checking the correctness of the service components behavior (expected actions after a request are taken)
* Checking the correctness of the service components behavior (expected actions after a request are taken)
* Tracing all the user actions in the system (e.g. by generating logs)
* Tracing all the user actions in the system (e.g. by generating logs)
Line 32: Line 40:
Monitorization should also be easily used in existing monitoring systems such as Nagios.  
Monitorization should also be easily used in existing monitoring systems such as Nagios.  


'''Verification''':Test suite must include tests on this functionality.
'''Verification''':  
Test suite with tests to:
* start, stop, suspend, and query status of service
* check responsiveness of service (expected ports open and expected answer to commands received)
* check correctness of service behavior (expexted actions after a request are taken)
* track of user actions in the system (generation of logs and accounting information)


== Interoperability ==
=== Configuration ===  
UMD services should be interoperable between them. All the services should assure their correct functionality within the rest of the UMD middleware for a given major release before entering the distribution. Ideally, backward compatibility between major releases should be kept. Interoperability between other middleware distributions is recommended, therefore compliance to existing and future standards should be priority in all the distributed software.  
Tools for the automatic or semi-automatic configuration of the services must be provided with the software. These tools should allow the unassisted configuration of the services for the most common use cases while being customizable for advanced user. Complete manual configuration must be always allowed.


'''Verification''': interoperability test of the service, assuring the correct behavior within the environment.
'''Verification''': test suite must include the configuration mechanisms and tools. Yaim is considered as the preferred tool.
 
== Source Code Quality and Availability ==
The source code of each component of the UMD middleware should follow a coherent and clear programming style that helps in the readability of the code and eases maintenance, testing, debugging, fixing, modification and portability of the software. Open source components must publicly offer their source code and the license with the binaries.
 
'''Verification''': for Open Source components, availability of the code and license. Source code quality metrics are desirable.


= Specific acceptance criteria =
== Specific acceptance criteria ==
This section will detail the specific acceptance criteria for each of the services that are part of the UMD.  
This section will detail the specific acceptance criteria for each of the services that are part of the UMD.  


Revision as of 16:37, 7 June 2010

UMD Quality Criteria

All the software included in the Universal Middleware Distribution (UMD) must meet a set of Quality Criteria defined by EGI. The Quality Criteria can be classified into generic criteria, i.e. criteria which sould hold for any component of the UMD, and specific criteria, i.e. criteria valid for a particular component only.

In order to be verified, quality criteria is specified as a set of tests. Those tests must ensure the correctness, completeness and security of each service. Software providers must include with each component a complete test plan that covers all the quality criteria. The test plan is composed by:

  • Documentation on how to run the tests
  • A test suite for each of the generic criteria
  • A test suite for each of the specific criteria applicable to the component
  • Tests results for all the specified tests

In the case of revision releases, the test plan must include specific tests for the bugs fixed in the release.

Generic acceptance criteria

Documentation

Services in UMD must include a comprehensive documentation written in a uniform and clear style, which reflects all of the following items:

  • Functional description of the software.
  • User documentation, including complete man pages of the commands and user guides.
  • Complete API documentation (if there is an API)
  • Administrator documentation that includes the installation procedure; detailed configuration of service; starting, stopping and querying service procedures; ports (or port ranges) used and expected connections to those ports; cron jobs needed for the service)
  • List of processes that are expected to run, giving a typical load of the service. List of how state information is managed and debugging information (e.g.: list of log files, any files or databases containing service information).
  • Notes on the testing procedure and expected tests results.

Verification: existence of the documentation with all the required items.

Source Code Quality and Availability

The source code of each component of the UMD middleware should follow a coherent and clear programming style that helps in the readability of the code and eases maintenance, testing, debugging, fixing, modification and portability of the software. Open source components must publicly offer their source code and the license with the binaries.

Verification: for Open Source components, availability of the code and license. Source code quality metrics are desirable.

Management, Monitoring, Traceability

All the services must include tools related to:

  • Starting, stopping, suspending, listing and querying the status of all the service daemons.
  • Checking the responsiveness of all the service components or daemons
  • Checking the correctness of the service components behavior (expected actions after a request are taken)
  • Tracing all the user actions in the system (e.g. by generating logs)

Ideally, these tools should be also available remotely, allowing operators to react timely to problems in the infrastructure. A uniform interface for remote management and monitoring should be followed by all the services. Monitorization should also be easily used in existing monitoring systems such as Nagios.

Verification: Test suite with tests to:

  • start, stop, suspend, and query status of service
  • check responsiveness of service (expected ports open and expected answer to commands received)
  • check correctness of service behavior (expexted actions after a request are taken)
  • track of user actions in the system (generation of logs and accounting information)

Configuration

Tools for the automatic or semi-automatic configuration of the services must be provided with the software. These tools should allow the unassisted configuration of the services for the most common use cases while being customizable for advanced user. Complete manual configuration must be always allowed.

Verification: test suite must include the configuration mechanisms and tools. Yaim is considered as the preferred tool.

Specific acceptance criteria

This section will detail the specific acceptance criteria for each of the services that are part of the UMD.

Retrieved from "https://wiki.egi.eu/w/index.php?title=UMDQualityCriteria&oldid=861"