Alert.png The wiki is deprecated and due to be decommissioned by the end of September 2022.
The content is being migrated to other supports, new updates will be ignored and lost.
If needed you can get in touch with EGI SDIS team using operations @ egi.eu.

Difference between revisions of "UMDQualityCriteria Generic"

From EGIWiki
Jump to navigation Jump to search
m
Line 5: Line 5:


=== Functional description ===
=== Functional description ===
==== Description ====
'''Description'''
 
Document with a functional description of the component
Document with a functional description of the component
==== Verification ====
 
'''Verification'''
 
Existence and quality of the document
 
'''Pass/Fail criteria'''
 
The document must exist and contain basic information about the component
 
=== Release notes ===
'''Description'''
 
Document with the release notes of the component
 
'''Verification'''
 
Existence and quality of the document
Existence and quality of the document
'''Pass/Fail criteria'''
The release notes document must exist and contain most relevant information about the release


=== User documentation ===
=== User documentation ===
==== Description ====
'''Description'''
 
User guide describing the functionality of the software and how to use it
User guide describing the functionality of the software and how to use it
==== Verification ====
 
'''Verification'''
 
Check availability of user guide. Check completeness of user guide.
Check availability of user guide. Check completeness of user guide.
'''Pass/Fail criteria'''
User guide must exist and contain information about the usage.


=== Man pages ===
=== Man pages ===
==== Description ====
'''Description'''
Commands must include documentation on its usage and options. It may not be applicable to services without user interface commands
 
==== Verification ====
Commands should include documentation on its usage and options. ''It may not be applicable to services without user interface commands''
Availability of man pages.  
 
'''Verification'''
 
Availability and quality of man pages.
 
'''Pass/Fail criteria'''
 
If the component provides command line tools, they must have man pages. Man pages should contain information about the usage. If man pages are not available, '''comprehensive''' help options must be included with the command with information about the usage.


=== API Documentation ===
=== API Documentation ===
==== Description ====
'''Description'''
If service has an API, include documentation of all its functions.
If service has an API, include documentation of all its functions.
==== Verification ====
 
Availability of API documentation.
'''Verification'''
 
Availability and quality of API documentation.
 
'''Pass/Fail criteria'''
 
If the component provides an API, documentation must exist and cover all the functions of the API. For each function, there must be a description of its functionality and its arguments.


=== Administrator Documentation ===
=== Administrator Documentation ===
==== Descrpiton ====
'''Descrpiton'''
 
Administrator guide describing installation, configuration and operation of the system
Administrator guide describing installation, configuration and operation of the system
==== Verification ====
 
'''Verification'''
 
Availability and completeness of the guide.
Availability and completeness of the guide.
'''Pass/Fail criteria'''
Installation guide must exist. Configuration and operation documents for services must also exist.


=== Service Reference Card ===
=== Service Reference Card ===
==== Description ====
'''Description'''
 
For each of the services that the component runs, include the following documentation:
For each of the services that the component runs, include the following documentation:


Line 66: Line 114:
| Include here any other relevant information about the service
| Include here any other relevant information about the service
|}
|}
'''Pass/Fail Criteria'''
All the services needed by a component must have the previous table with complete information.


== Source Code Quality ==
== Source Code Quality ==
Line 72: Line 124:


=== License ===
=== License ===
==== Description ====
'''Description'''
 
All UMD components must have a compatible license for using it in the EGI infrastructure
All UMD components must have a compatible license for using it in the EGI infrastructure
==== Verification ====
 
'''Verification'''
 
Check the component license
Check the component license
'''Pass/Fail criteria'''
TBD: Compatible licenses in EGI ?


=== Source Code Availability ===
=== Source Code Availability ===
==== Description ====
'''Description'''
 
Open source code must provide pointers to the source
Open source code must provide pointers to the source
==== Verification ====
 
Existence of source code  
'''Verification'''
 
Check existence of source code  
 
'''Pass/Fail Criteria'''
 
If the component is open source, source must be available for download.


=== Build Procedures ===
=== Build Procedures ===
==== Description ====
'''Description'''
 
Open Source components should provide a documented and reproducible build procedure
Open Source components should provide a documented and reproducible build procedure
==== Verification ====
 
Existence of build documentation. Existence of automatic build procedures.
'''Verification'''
 
Check existence of build documentation and existence of automatic build procedures.
 
'''Pass/Fail Criteria'''
 
Build documentation for open source components must exist. Automatic build procedures are recommendable.
 


== Management, Monitoring, Traceability ==
== Management, Monitoring, Traceability ==
Line 93: Line 167:
TBD: Is there any standard interface for this?
TBD: Is there any standard interface for this?


=== Service control ===
=== Service control and status ===
==== Description ====
'''Description'''
Services run by the component must provide a mechanism for starting and stopping the services.  
 
==== Verification ====
Services run by the component must provide a mechanism for starting, stopping and querying the status of the services following the typical init scripts conventions as documented in http://refspecs.freestandards.org/LSB_3.1.0/LSB-Core-generic/LSB-Core-generic/iniscrptact.html
Test for the mechanism that starts/stops the services. Test all possible combinations:
 
'''Verification'''
 
Check the start/stop mechanism for the service. Test all possible combinations:
* start with already started services
* start with already started services
* start with services stopped
* start with services stopped
* stop with stopped services
* stop with stopped services
* stop with started services
* stop with started services
* check status with started services
* check status with stopped services


=== Service status ===
'''Pass/Fail criteria'''
==== Description ====
 
Services must provide a mechanism to query their status.
Service must provide mechanisms for starting and stopping services. They must work properly in '''all''' the cases described above.
==== Verification ====
Test the mechanism that queries the service status with both running and stopped service.


=== Service responsiveness ===
=== Service responsiveness ===
==== Descripition ====
'''Descripition'''
 
Services must react to requests timely. They should provide a mechanism to check their responsiveness to the requests
Services must react to requests timely. They should provide a mechanism to check their responsiveness to the requests
==== Verification ====
 
''Verification'''
 
Provide a test that checks the service responds to requests (e.g. expected ports are open, expected answer to requests is received).  
Provide a test that checks the service responds to requests (e.g. expected ports are open, expected answer to requests is received).  


=== Service correctness ===
'''Pass/Fail criteria'''
==== Description ====
 
Services should react to requests and perform the expexted actions after that request. A test for checking the correctness of the request should be provided
Test for service responsiveness must be available and executed correctly.
==== Verification ====
 
Provide a test that checks the correctness of typical requests that may be integrated in a monitoring framework. '''To Be Defined''': monitoring framework to use (probably Nagios), include documentation on how to write the tests.
=== Service monitoring ===
'''Description'''
 
All components in the EGI testbed must provide monitoring probes that can be executed automatically by a monitoring framework such as Nagios. The probe should check the service responsiveness and correctness (good replies for typical requests).
Documentation for the current monitoring framework for EGI is available at: '''TBD''' ?
 
'''Verification'''
 
Check the existence of the component probe and check its integration in Nagios
 
'''Pass/Fail criteria'''
 
Probe must exist and must be executed correctly in the monitoring framework.


=== Traceability ===
=== Traceability ===
==== Description ====
'''Description ''
 
All actions taken by the service should be traceable by the service administrator. Log files or similar mechanisms should be provided by the service.
All actions taken by the service should be traceable by the service administrator. Log files or similar mechanisms should be provided by the service.
==== Verification ====
 
'''Verification'''
 
Provide a test that checks that the tracing mechanism of the service is working properly (e.g. log file of the service is generated)
Provide a test that checks that the tracing mechanism of the service is working properly (e.g. log file of the service is generated)
'''Pass/Fail Criteria'''
Test for the tracing mechanism must exist. It must be executed correctly.

Revision as of 16:15, 7 October 2010

Documentation

Services in UMD must include a comprehensive documentation written in a uniform and clear style. All Quality Criteria described below may be met by a single document that contains all the requested sections.

Functional description

Description

Document with a functional description of the component

Verification

Existence and quality of the document

Pass/Fail criteria

The document must exist and contain basic information about the component

Release notes

Description

Document with the release notes of the component

Verification

Existence and quality of the document

Pass/Fail criteria

The release notes document must exist and contain most relevant information about the release

User documentation

Description

User guide describing the functionality of the software and how to use it

Verification

Check availability of user guide. Check completeness of user guide.

Pass/Fail criteria

User guide must exist and contain information about the usage.

Man pages

Description

Commands should include documentation on its usage and options. It may not be applicable to services without user interface commands

Verification

Availability and quality of man pages.

Pass/Fail criteria

If the component provides command line tools, they must have man pages. Man pages should contain information about the usage. If man pages are not available, comprehensive help options must be included with the command with information about the usage.

API Documentation

Description If service has an API, include documentation of all its functions.

Verification

Availability and quality of API documentation.

Pass/Fail criteria

If the component provides an API, documentation must exist and cover all the functions of the API. For each function, there must be a description of its functionality and its arguments.

Administrator Documentation

Descrpiton

Administrator guide describing installation, configuration and operation of the system

Verification

Availability and completeness of the guide.

Pass/Fail criteria

Installation guide must exist. Configuration and operation documents for services must also exist.

Service Reference Card

Description

For each of the services that the component runs, include the following documentation:

Service Reference Card
Service ServiceName
Description Description of the service
Init scripts List of init scripts for the service, expected run levels
Daemons List of daemons needed for the service
Configuration List of configuration files used by the service
Logs List of log files used by the service
Open ports List of ports the service uses
Cron List of crons used by the service
Any other relevant information Include here any other relevant information about the service

Pass/Fail Criteria

All the services needed by a component must have the previous table with complete information.

Source Code Quality

The source code of each component of the UMD middleware should follow a coherent and clear programming style that helps in the readability of the code and eases maintenance, testing, debugging, fixing, modification and portability of the software. Open source components must publicly offer their source code and the license with the binaries.

License

Description

All UMD components must have a compatible license for using it in the EGI infrastructure

Verification

Check the component license

Pass/Fail criteria

TBD: Compatible licenses in EGI ?

Source Code Availability

Description

Open source code must provide pointers to the source

Verification

Check existence of source code

Pass/Fail Criteria

If the component is open source, source must be available for download.

Build Procedures

Description

Open Source components should provide a documented and reproducible build procedure

Verification

Check existence of build documentation and existence of automatic build procedures.

Pass/Fail Criteria

Build documentation for open source components must exist. Automatic build procedures are recommendable.


Management, Monitoring, Traceability

UMD components should have mechanisms for managing them, monitoring their status and tracing actions they perform on the system. Ideally, these should be also available remotely, allowing operators to react timely to problems in the infrastructure. TBD: Is there any standard interface for this?

Service control and status

Description

Services run by the component must provide a mechanism for starting, stopping and querying the status of the services following the typical init scripts conventions as documented in http://refspecs.freestandards.org/LSB_3.1.0/LSB-Core-generic/LSB-Core-generic/iniscrptact.html

Verification

Check the start/stop mechanism for the service. Test all possible combinations:

  • start with already started services
  • start with services stopped
  • stop with stopped services
  • stop with started services
  • check status with started services
  • check status with stopped services

Pass/Fail criteria

Service must provide mechanisms for starting and stopping services. They must work properly in all the cases described above.

Service responsiveness

Descripition

Services must react to requests timely. They should provide a mechanism to check their responsiveness to the requests

Verification'

Provide a test that checks the service responds to requests (e.g. expected ports are open, expected answer to requests is received).

Pass/Fail criteria

Test for service responsiveness must be available and executed correctly.

Service monitoring

Description

All components in the EGI testbed must provide monitoring probes that can be executed automatically by a monitoring framework such as Nagios. The probe should check the service responsiveness and correctness (good replies for typical requests). Documentation for the current monitoring framework for EGI is available at: TBD ?

Verification

Check the existence of the component probe and check its integration in Nagios

Pass/Fail criteria

Probe must exist and must be executed correctly in the monitoring framework.

Traceability

'Description

All actions taken by the service should be traceable by the service administrator. Log files or similar mechanisms should be provided by the service.

Verification

Provide a test that checks that the tracing mechanism of the service is working properly (e.g. log file of the service is generated)

Pass/Fail Criteria

Test for the tracing mechanism must exist. It must be executed correctly.