TSA2.3: Verification of conformance criteria

From EGIWiki
Jump to: navigation, search

TSA2.3: Verification of conformance criteria

Necessary services to deliver the functionality of the repository and its surrounding process (FTP server, web server, issue tracker, version control system, etc.) will be provided and maintained

Main Items


The main objective of the TSA2.3 is to verify the quality of the software provided by the TP before entering the SR phase and going to production. By doing so we prevent that software that might work enters into the SR and even goes into production but that doesn´t follow the quality criteria defined in TSA2.2. Some of the reasons for doing the verification before the software enters the stage rollout are:

- Check that the bugs reported in the previous release of the software have been corrected (work in collaboration with DMSU) by the TP.

- Software can work well in the SR but might not have all the functionalities required

- Software might not be safe, well documented, or have the necessary installation rules or licenses

The verification process

When a new release is available, the TP has to follow the NSRW [2]. Once the software is correctly uploaded to the repository, the release enters into the verification phase. The requirements are that the TP has to provide all the necessary information to the verifier (QCV) so that the QCV can assess that the TP has tested in advance the quality of the software. Depending on the type of release, different actions will be taken by the QCV:

- Revision release: the QCV has to check that the TP has done the testing and provides all the results required in the QC of the component. To do so, the QCV provides a form (or template, QCVT) that the TP should fill for the release with the information about where is the documentation and the results of the different criteria. The QCV will check that all the information is correct and the tests have been passed succesfully. If some information is not available or not well documented according to the QC, the QCV will ask for it and the TP will have to response in time to the request. Failure in doing so satisfactory might reject the release.

- Minor release: in a first stage the QCV follows the same procedure than with the minor releases. If the information about the testing is correct, then the QCV has to select randomly one or more criterias or tests and ask to the TP to repeat it with the review of the QCV or the TP will have to provide access to the system where tests these items with all the relevant instructions. The TP has to provide the necessary technologies (desktop sharing, VLC, virtual machines or others) in order to work together with the QCV. The number of criterias that will be re-tested will depend on the reliability of the TP, the criticity of the release and the experience of the QCV.

- Major release: for a major release, the QCV first checks the completeness of the information about the tests provided by the TP. Once this step is passed, the TP and the QCV will have to reproduce all the tests in close collaboration to ensure that all of the criterias have been correctly verified. The procedures to do so can be the same that with the minor release. Taking in account the number of criterias the verification could be done by more than one QCV and in the same way the TP will have to provide more than one contact point for the review. The QCV can ask for more tests (specially integration tests) in order to assess the overall quality of the software.

More information is available in [1], specifically in table 1: SA2 activity by major, minor or revision release

As a result of the process the QCV will produce a report (QCVR) that will be available to the TP with the findings of the process and will accept or reject the release.

Other TSA2.3 dutties

- Work on the definition and implementation of the NSRW

- Prepare and write the templates for the verification

- Get information for the TP release contact and inform about the process and status of the release

- Escalate any issue with the TP to the TCB

- Metrics definition and calculation

- Get feedback from DMSU about the process and how to improve it to ensure the quality of the software.

- Work on technologies to collaborate on the assessment of the criteria and the evaluation of the software with the TP.

- Take care about the releases planned in the next periods to organize the workload.

About the responsibility of TSA2.3 QCV

As the verification process is in the chain of release workflows, the QCV have different clients with different expectations:

- The TP wants to provide the software as fastests as possible

- The SR team wants to software to be tested as soon as possible

- The RC usually wants new software releases because they provide new functionalities and/or important bugfixes

- The RC wants also good quality software

- The SA2 activity coordinator wants the releases to be validated the sooner the better

So the verification has to deal with the different requirements of these communities. It is crucial then than all are aware of the process, agree on it, its implications and the responsabilities of the different parties. If a new release doesn´t pass the QC verification process and has to be fixed by the TP before going to production, this will mean a delay in the release of the software and probably a failure to accomplish the SLA by the TP, so the pressure to the verifier will be high and pressure to bypass the QC verification will also be higher by the TP. However, it is the responsibility of the TSA2.3 to do this job. But it is important also to have these responsabilities:

- SA2 coordinator should enforce the process

- EGI director and EB should also enforce this process and the correctness of the work done by the people involved

Without the correct support by the higher decission bodies, the QCV would fail under the pressure of the TP, RC and others, meaning that the quality of the software would be higher that expected, and much of the responsibility would go to the SA2.3 activity. So it has to be stated that imposing this assessment and verification needs time and resources. If this is not accepted, then this activity will have to be redesigned.

Who is involved in the activity

TSA2.3 is responsibility of IBERRID, through CSIC (mainly CESGA and IFCA) in Spain and LIP in Portugal. A distributed team is in charge of doing the verification and holidays and shiftings are done to assure that allways there is at least one person working on the verification of the releases. In any case, a good planning and advise about the releases to be distributed by the different TP is needed in order to organize the work properly. If the workload is high other partners could take part of the work specially if they provide extra experience in the pieces of software to be tested, in the same way as the stage rollout.

The role of the TP in the verification

The TP not only has to develop good software with new functionalities but also needs to do it following the QC defined in TSA2.2. In order to do that, the TP are not only required to read the QC but to assure that the software that they produce follows these criteria. This quality level would be impossible to reach if the TP doesn´t verify the software (if they don´t do that the verification will fail and the release will not go into production unless there is a high pressure or demand for it and low implication in the process by the higher decission bodies, so in this case low quality software will enter into production). If the TP has done this verification succesfully in advance, they will be able to quickly fill the QCVT and demonstrate that they have followed QC requirements.

For example, to verify that one service has the start, status and stop commands, and their functionality, the TP could provide a video recording the console and showing the outcome of each of these commands. In the QCVT the TP will have to provide a link to this video. The QCV will have to determine if this is enough of more information needs to be provided. Also, in the case of minor and major releases, the QCV will ask for access to a system where the software was tested and do the tests by himself or with the close collaboration of the TP.

To summarize:

- The TP has to be aware of the QC

- The TP has to know and enroll properly to the NSRW

- The TP has to verify in advance the quality of the software following the QC applicable to each component. It would not be realistic to think that the software will pass the quality criteria if the TP doesn´t verify it in house

- As a result of the previous requirement, the TP has to provide all the tests run and outcomes to the verifier completelly filling the QCVT

- The TP has to provide any additional information about this process to QCF

- In the case of minor and major releases, the TP has to provide access to the systems where the software was tested

It could be stated that either: - The TP doesn´t have the resources to test the software

- The systems or platforms where the TP tests the software are not similar to the ones where the software will run

In any of the previou situation, we believe that the quality of the software could be really poor if the TP doesn´t have such a platform. In any case, the EGI Inspire should work in close collaboration with the TP to provide such resources, but this is, in principle, out of the scope of TSA2.3 activity

Human resources and funding

2 FTEs are available in total for TSA2.2 and TSA2.3 as funded effort. 2 FTEs more are available as unfunded effort in IBEGRID.

Some variations and comments:

The verification process could be done in the SR. However we believe there are reasons for not doing it:

- The resources consumed within EGI once the software enters into SR are higher

- The fallback is more dramatic and time consuming

- The focus of the SR is not to verify the quality of the software, but that new software releases work properly in a production environment and do not cause a disruption in the continuity of the services.

However, under some circunstances, some tests could and should be verified in this stage. For example, in case that the verifier thinks that one criteria is not enough documented or not sure about its correctnes, the verifier could advise of this situation and work with the SR team in order to check that this criteria is really fullfilled once the software is in the SR and before entering into production. The TP could also ask for this kind of verfication if they are able to demonstrate that some tests can only be done in this environment to guarantee the validity of the results. In this sense, as the SR is coordinate by IBERGRID (LIP in Portugal) we can profit from synergies in the overall process. In any case we have to highlight that these kind of tests should be the exception and complementary to the verification phase and not a sustitute of it.

In the same way, the QCV team has to work with the QC definition team and the DMSU to provide feedback about the quality of the software and the procedure to guarantee it.

During the verification process, we expect that some releases do not pass repeatly the QC or the TP doesn´t accept the criteria. In that case, the QCV will inform the TSA2.3 coordinator who will also inform the TCB to take the necessary actions.


M.SA2.4 Number of new releases validated against defined criteria: Measures the workload on the validation team

M.SA2.5 Mean time taken to validate a releas: Indicates how responsive the team is to validating releases

M.SA2.6 Number of releases failing validation: Indicates the quality assurance process of the software providers


[1] [[ https://documents.egi.eu/secure/RetrieveFile?docid=68&version=7&filename=EGI-MS503-final.pdf%7CMS503]]

[2] NSRW

[3] [[ https://www.egi.eu/indico/contributionDisplay.py?sessionId=56&contribId=260&confId=48%7CQuality Criteria Verification. Software Provisioning in EGI. Presentation EGI TF 2010 ]]

Related Links

SA2 Centric

SA2 Meetings


EGI-InSPIRE-SA2 Action Items

Feedback ON NSRW

Feedback from M. David

Feedback from M. Drescher






Current Quality Criteria: EGI-InSPIRE:UMDQualityCriteria

The current template for Quality Criteria verification can be found here.




UMD RoadMap




-- carlosf 10:06, 21 November 2010 (UTC)--