Alert.png The wiki is deprecated and due to be decommissioned by the end of September 2022.
The content is being migrated to other supports, new updates will be ignored and lost.
If needed you can get in touch with EGI SDIS team using operations @ egi.eu.

Difference between revisions of "UMDQualityCriteria Generic"

From EGIWiki
Jump to navigation Jump to search
(Created page with '= Documentation = Services in UMD must include a comprehensive documentation written in a uniform and clear style. Quality Criteria to be checked for documentation: == Functiona…')
 
 
(21 intermediate revisions by one other user not shown)
Line 1: Line 1:
= Documentation =
== Documentation ==


Services in UMD must include a comprehensive documentation written in a uniform and clear style.
Services in UMD must include a comprehensive documentation written in a uniform and clear style.
Quality Criteria to be checked for documentation:
All Quality Criteria described below may be met by a single document that contains all the requested sections.
== Functional description ==
 
=== Description ===
=== Functional description ===
'''Description'''
 
Document with a functional description of the component
Document with a functional description of the component
=== Verification ===
Existence of the document


== User documentation ==
'''Input from TP'''
=== Description ===
 
User guide describing the functionality of the software and how to use it
Document (or link) with functional description of component
=== Verification ===
 
'''Pass/Fail criteria'''
 
The document must exist and contain basic information about the component
 
=== Release notes ===
'''Description'''
 
Document with the release notes of the component
 
'''Input from TP'''
 
Document (or link) with Release Notes of component
 
'''Pass/Fail criteria'''
 
The release notes document must exist and contain most relevant information about the release.
 
=== User documentation ===
'''Description'''
 
User guide describing the functionality of the software and how to use it. '''Only applicable to services meant to be used by end-users'''
 
'''Input from TP'''
 
Document (or link) with User guide of component
 
'''Verification'''
 
Check availability of user guide. Check completeness of user guide.
Check availability of user guide. Check completeness of user guide.


== Man pages ==
'''Pass/Fail criteria'''
=== Description ===
 
Commands must include documentation on its usage and options. It may not be applicable to services without user interface commands
User guide must exist and contain information about the usage.
=== Verification ===
 
Availability of man pages.  
=== Man pages ===
'''Description'''
 
Commands should include documentation on its usage and options. '''Only applicable to services with user interface commands'''
 
'''Input from TP'''
 
Document (or link) with man pages of component
 
'''Verification'''
 
Availability and quality of man pages.
 
'''Pass/Fail criteria'''
 
If the component provides command line tools, they must have man pages. Man pages should contain information about the usage. If man pages are not available, '''comprehensive''' help options must be included with the command with information about the usage.
 
=== API Documentation ===
'''Description'''
 
If service has an API, include documentation of all its functions. '''Only applicable to services with an API'''


== API Documentation ==
'''Input from TP'''
=== Description ===
 
If service has an API, include documentation of all its functions.
Document (or link) with API documentation
=== Verification ===
 
Availability of API documentation.
'''Verification'''
 
Availability and quality of API documentation.
 
'''Pass/Fail criteria'''
 
If the component provides an API, documentation must exist and cover all the functions of the API. For each function, there must be a description of its functionality and its arguments.
 
=== Administrator Documentation ===
'''Descrpiton'''


== Administrator Documentation ==
=== Descrpiton ===
Administrator guide describing installation, configuration and operation of the system
Administrator guide describing installation, configuration and operation of the system
=== Verification ===
 
'''Input from TP'''
 
Document (or link) with Administrator documentation
 
'''Verification'''
 
Availability and completeness of the guide.
Availability and completeness of the guide.


== Service Reference Card ==
'''Pass/Fail criteria'''
=== Description ===
 
For each of the services that the component runs, include the following documentation:
Installation guide must exist. Configuration and operation documents for services must also exist.
 
=== Service Reference Card ===
'''Description'''
 
For each of the services that the component runs, include a reference card. '''Only applicable to components that need to run services'''
 
'''Input from TP'''
 
Document (or link) with reference cards for each of the services of the component. The reference card must include the following documentation:


{| border="1"
{| border="1"
Line 66: Line 136:
|}
|}


'''Pass/Fail Criteria'''
All the services needed by a component must have the previous table with complete information.


= Source Code Quality =
== Source Code Quality ==


The source code of each component of the UMD middleware should follow a coherent and clear programming style that helps in the readability of the code and eases maintenance, testing, debugging, fixing, modification and portability of the software. Open source components must publicly offer their source code and the license with the binaries.
The source code of each component of the UMD middleware should follow a coherent and clear programming style that helps in the readability of the code and eases maintenance, testing, debugging, fixing, modification and portability of the software. Open source components must publicly offer their source code and the license with the binaries.


== License ==
=== License ===
=== Description ===
'''Description'''
 
All UMD components must have a compatible license for using it in the EGI infrastructure
All UMD components must have a compatible license for using it in the EGI infrastructure
=== Verification ===
Check the component license


== Source Code Availability ==
'''Input from TP'''
=== Description ===
 
Component license (link or document)
 
'''Pass/Fail criteria'''
 
Pass: if the licens if compatible with licenses in EGI ('''TDB: which are these compatible licenses?''')
 
=== Source Code Availability [Non Blocking] ===
'''Description'''
 
Open source code must provide pointers to the source
Open source code must provide pointers to the source
=== Verification ===
Existence of source code


== Build Procedures ==
'''Input from TP'''
=== Description ===
 
Link to source code of component or tarball containing source.
 
'''Pass/Fail Criteria'''
 
If the component is open source, Pass if source is available for download.
 
=== Build Procedures [Non Blocking] ===
'''Description'''
 
Open Source components should provide a documented and reproducible build procedure
Open Source components should provide a documented and reproducible build procedure
=== Verification ===
 
Existence of build documentation. Existence of automatic build procedures.
'''Input from TP'''
 
Build documentation or automatic build system.
 
'''Pass/Fail Criteria'''
 
Check existence of build documentation and existence of automatic build procedures.
Build documentation for open source components must exist. Automatic build procedures are recommendable.
 
== Management, Monitoring, Traceability ==
UMD components should have mechanisms for managing them, monitoring their status and tracing actions they perform on the system. Ideally, these should be also available remotely, allowing operators to react timely to problems in the infrastructure.
TBD: Is there any standard interface for this?
 
=== Service control and status ===
'''Description'''
 
Services run by the component must provide a mechanism for starting, stopping and querying the status of the services following the typical init scripts conventions as documented in http://refspecs.freestandards.org/LSB_3.1.0/LSB-Core-generic/LSB-Core-generic/iniscrptact.html. '''Only applicable to components that need to run services'''
 
'''Input from TP'''
 
Start/stop mechanism for each of the services. Test that checks the mechanism is working correctly for all possible combinations
* start with already started services
* start with services stopped
* stop with stopped services
* stop with started services
* check status with started services
* check status with stopped services
 
'''Verification'''
 
Check the start/stop mechanism for the service. Perform the mechanism tests.
 
'''Pass/Fail criteria'''
 
Service must provide mechanisms for starting and stopping services. They must work properly in '''all''' the cases described above.
 
=== Service responsiveness ===
'''Descripition'''
 
Services must react to requests timely. They should provide a mechanism to check their responsiveness to the requests
 
'''Verification'''
 
Provide a test that checks the service responds to requests (e.g. expected ports are open, expected answer to requests is received).
 
'''Pass/Fail criteria'''
 
Test for service responsiveness must be available and executed correctly.
 
=== Service monitoring ===
'''Description'''
 
All components in the EGI testbed must provide monitoring probes that can be executed automatically by a monitoring framework such as Nagios. The probes should check the service responsiveness and correctness (good replies for typical requests).
 
Documentation for the current monitoring framework for EGI is available at: '''TBD''' ?
 
Particular monitoring probes are defined at the Specific Quality Criteria for each component. Here a list of generic probes that must be provided are listed.
 
==== Certificate lifetime ====
'''Description'''
 
Provide a monitoring probe that test the host certificate lifetime for the service is valid.
This QC is only applicable to services that need a host certificate.
 
'''Test Input'''
 
Host to check.
 
'''Test Output'''
 
OK if certificate is valid. ERROR if certificate is not valid.
 
'''Pass/Fail Criteria'''
 
''Pass'': probe exists and calculate the certificate lifetime correctly. Probe can integrated in the monitoring framework.
 
''Fail'': if the probe does not exist or is not able to get the certificate lifetime or it cannot be integrated in the monitoring framework.
 
=== Traceability ===
'''Description '''
 
All actions taken by the service should be traceable by the service administrator. Log files or similar mechanisms should be provided by the service.
 
'''Verification'''
 
Provide a test that checks that the tracing mechanism of the service is working properly (e.g. log file of the service is generated)
 
'''Pass/Fail Criteria'''
 
Test for the tracing mechanism must exist. It must be executed correctly.
 
== Software Packaging ==
'''Description'''
 
UMD Software must be available in the packaging formats of the supported OS.
 
'''Pass/Fail Criteria'''
 
* For all software:
** Provide a tar.gz/zip or similar distribution
 
* For RPM based OS:
** Provide RPM packages with correct dependencies
 
* For DEB based OS:
** Provide DEB packages with correct dependencies.
 
The supported OSes is determined by the Portability QC of each component.

Latest revision as of 22:10, 24 December 2014

Documentation

Services in UMD must include a comprehensive documentation written in a uniform and clear style. All Quality Criteria described below may be met by a single document that contains all the requested sections.

Functional description

Description

Document with a functional description of the component

Input from TP

Document (or link) with functional description of component

Pass/Fail criteria

The document must exist and contain basic information about the component

Release notes

Description

Document with the release notes of the component

Input from TP

Document (or link) with Release Notes of component

Pass/Fail criteria

The release notes document must exist and contain most relevant information about the release.

User documentation

Description

User guide describing the functionality of the software and how to use it. Only applicable to services meant to be used by end-users

Input from TP

Document (or link) with User guide of component

Verification

Check availability of user guide. Check completeness of user guide.

Pass/Fail criteria

User guide must exist and contain information about the usage.

Man pages

Description

Commands should include documentation on its usage and options. Only applicable to services with user interface commands

Input from TP

Document (or link) with man pages of component

Verification

Availability and quality of man pages.

Pass/Fail criteria

If the component provides command line tools, they must have man pages. Man pages should contain information about the usage. If man pages are not available, comprehensive help options must be included with the command with information about the usage.

API Documentation

Description

If service has an API, include documentation of all its functions. Only applicable to services with an API

Input from TP

Document (or link) with API documentation

Verification

Availability and quality of API documentation.

Pass/Fail criteria

If the component provides an API, documentation must exist and cover all the functions of the API. For each function, there must be a description of its functionality and its arguments.

Administrator Documentation

Descrpiton

Administrator guide describing installation, configuration and operation of the system

Input from TP

Document (or link) with Administrator documentation

Verification

Availability and completeness of the guide.

Pass/Fail criteria

Installation guide must exist. Configuration and operation documents for services must also exist.

Service Reference Card

Description

For each of the services that the component runs, include a reference card. Only applicable to components that need to run services

Input from TP

Document (or link) with reference cards for each of the services of the component. The reference card must include the following documentation:

Service Reference Card
Service ServiceName
Description Description of the service
Init scripts List of init scripts for the service, expected run levels
Daemons List of daemons needed for the service
Configuration List of configuration files used by the service
Logs List of log files used by the service
Open ports List of ports the service uses
Cron List of crons used by the service
Any other relevant information Include here any other relevant information about the service

Pass/Fail Criteria

All the services needed by a component must have the previous table with complete information.

Source Code Quality

The source code of each component of the UMD middleware should follow a coherent and clear programming style that helps in the readability of the code and eases maintenance, testing, debugging, fixing, modification and portability of the software. Open source components must publicly offer their source code and the license with the binaries.

License

Description

All UMD components must have a compatible license for using it in the EGI infrastructure

Input from TP

Component license (link or document)

Pass/Fail criteria

Pass: if the licens if compatible with licenses in EGI (TDB: which are these compatible licenses?)

Source Code Availability [Non Blocking]

Description

Open source code must provide pointers to the source

Input from TP

Link to source code of component or tarball containing source.

Pass/Fail Criteria

If the component is open source, Pass if source is available for download.

Build Procedures [Non Blocking]

Description

Open Source components should provide a documented and reproducible build procedure

Input from TP

Build documentation or automatic build system.

Pass/Fail Criteria

Check existence of build documentation and existence of automatic build procedures. Build documentation for open source components must exist. Automatic build procedures are recommendable.

Management, Monitoring, Traceability

UMD components should have mechanisms for managing them, monitoring their status and tracing actions they perform on the system. Ideally, these should be also available remotely, allowing operators to react timely to problems in the infrastructure. TBD: Is there any standard interface for this?

Service control and status

Description

Services run by the component must provide a mechanism for starting, stopping and querying the status of the services following the typical init scripts conventions as documented in http://refspecs.freestandards.org/LSB_3.1.0/LSB-Core-generic/LSB-Core-generic/iniscrptact.html. Only applicable to components that need to run services

Input from TP

Start/stop mechanism for each of the services. Test that checks the mechanism is working correctly for all possible combinations

  • start with already started services
  • start with services stopped
  • stop with stopped services
  • stop with started services
  • check status with started services
  • check status with stopped services

Verification

Check the start/stop mechanism for the service. Perform the mechanism tests.

Pass/Fail criteria

Service must provide mechanisms for starting and stopping services. They must work properly in all the cases described above.

Service responsiveness

Descripition

Services must react to requests timely. They should provide a mechanism to check their responsiveness to the requests

Verification

Provide a test that checks the service responds to requests (e.g. expected ports are open, expected answer to requests is received).

Pass/Fail criteria

Test for service responsiveness must be available and executed correctly.

Service monitoring

Description

All components in the EGI testbed must provide monitoring probes that can be executed automatically by a monitoring framework such as Nagios. The probes should check the service responsiveness and correctness (good replies for typical requests).

Documentation for the current monitoring framework for EGI is available at: TBD ?

Particular monitoring probes are defined at the Specific Quality Criteria for each component. Here a list of generic probes that must be provided are listed.

Certificate lifetime

Description

Provide a monitoring probe that test the host certificate lifetime for the service is valid. This QC is only applicable to services that need a host certificate.

Test Input

Host to check.

Test Output

OK if certificate is valid. ERROR if certificate is not valid.

Pass/Fail Criteria

Pass: probe exists and calculate the certificate lifetime correctly. Probe can integrated in the monitoring framework.

Fail: if the probe does not exist or is not able to get the certificate lifetime or it cannot be integrated in the monitoring framework.

Traceability

Description

All actions taken by the service should be traceable by the service administrator. Log files or similar mechanisms should be provided by the service.

Verification

Provide a test that checks that the tracing mechanism of the service is working properly (e.g. log file of the service is generated)

Pass/Fail Criteria

Test for the tracing mechanism must exist. It must be executed correctly.

Software Packaging

Description

UMD Software must be available in the packaging formats of the supported OS.

Pass/Fail Criteria

  • For all software:
    • Provide a tar.gz/zip or similar distribution
  • For RPM based OS:
    • Provide RPM packages with correct dependencies
  • For DEB based OS:
    • Provide DEB packages with correct dependencies.

The supported OSes is determined by the Portability QC of each component.