Difference between revisions of "Infrastructure Manager Availability and Continuity Plan"
(11 intermediate revisions by the same user not shown) | |||
Line 24: | Line 24: | ||
previous plans are collected here: | previous plans are collected here: https://documents.egi.eu/document/3858 | ||
= Hardware HA Configuration = | = Hardware HA Configuration = | ||
Line 30: | Line 30: | ||
= Availability requirements and performances = | = Availability requirements and performances = | ||
In the OLA it was agreed the following performances targets, on a monthly basis: | In the OLA it was agreed the following performances targets, on a monthly basis: | ||
*Availability: | *Availability: 95% | ||
*Reliability: | *Reliability: 95% | ||
Other availability requirements: | |||
* the | * The service can be accessed using a CLI and a web UI. | ||
* The | * To access the web UI EGI-Chekin credential is required. | ||
* | * The client only needs to set the EGI-Chekin credentials to access EGI Cloud compute sites. | ||
* In case of using an on premise cloud provider, any user can use the CLI. | |||
The service availability is regularly tested by nagios probes eu.egi.grycap.IM-Check: https://argo-mon.egi.eu/nagios/cgi-bin/status.cgi?servicegroup=SERVICE_es.upv.grycap.im&style=overview | |||
The performances reports in terms of Availability and Reliability are produced by [ | The performances reports in terms of Availability and Reliability are produced by [https://argo.egi.eu/egi/OPS-MONITOR-Critical ARGO] on an almost real time basis and they are also periodically collected into the [https://documents.egi.eu/public/ShowDocument?docid=2324 Documentation Database]. | ||
= Risks assessment and management = | = Risks assessment and management = | ||
For more details, please look at the (google spreadsheet link). We will report here a summary of the assessment. | For more details, please look at the (google spreadsheet link). We will report here a summary of the assessment. | ||
Line 69: | Line 65: | ||
Furthermore there are daily backups of the whole VMs. In case of error it can be migrated to another cloud provider. Also we have RADL/ansible recipes to re-deploy the full K8s and IM services | Furthermore there are daily backups of the whole VMs. In case of error it can be migrated to another cloud provider. Also we have RADL/ansible recipes to re-deploy the full K8s and IM services | ||
| style="background: green"| Low | | style="background: green"| Low | ||
| | | up to 4 hours | ||
| the measures already in place are considered satisfactory and risk level is acceptable | | the measures already in place are considered satisfactory and risk level is acceptable | ||
|- | |- | ||
Line 78: | Line 74: | ||
Furthermore there are daily backups of the whole VMs. In case of error it can be migrated to another cloud provider. Also we have RADL/ansible recipes to re-deploy the full K8s and IM services. The software, the SO and availability of them is regularly checked using nagios | Furthermore there are daily backups of the whole VMs. In case of error it can be migrated to another cloud provider. Also we have RADL/ansible recipes to re-deploy the full K8s and IM services. The software, the SO and availability of them is regularly checked using nagios | ||
| style="background: green"| Low | | style="background: green"| Low | ||
| | | up to 4 hours | ||
| the measures already in place are considered satisfactory and risk level is acceptable | | the measures already in place are considered satisfactory and risk level is acceptable | ||
|- | |- | ||
Line 86: | Line 82: | ||
| We have a CI/CD process in our jenkins intance. All the changes made at master branch are deployed into the IM devel instance. When a new IM version is released the pord instance is updated | | We have a CI/CD process in our jenkins intance. All the changes made at master branch are deployed into the IM devel instance. When a new IM version is released the pord instance is updated | ||
| style="background: green"| Low | | style="background: green"| Low | ||
| | | up to 4 hours | ||
| the measures already in place are considered satisfactory and risk level is acceptable | | the measures already in place are considered satisfactory and risk level is acceptable | ||
|- | |- | ||
Line 93: | Line 89: | ||
| All | | All | ||
| Network managed centrally at UPV with established measures to mitigate issues | | Network managed centrally at UPV with established measures to mitigate issues | ||
| style="background: | | style="background: yellow"| Medium | ||
| | | up to 4 hours | ||
| the measures already in place are considered satisfactory and risk level is acceptable | | the measures already in place are considered satisfactory and risk level is acceptable | ||
|- | |- | ||
Line 102: | Line 98: | ||
| The technical staff is available on a best effort basis, during holidays, and it is possible to access using VPN. | | The technical staff is available on a best effort basis, during holidays, and it is possible to access using VPN. | ||
Some other technical staff is available to address urgent issues (such as power interruption or similar) | Some other technical staff is available to address urgent issues (such as power interruption or similar) | ||
| style="background: | | style="background: yellow"| Medium | ||
| | | 1 or more working days | ||
| the measures already in place are considered satisfactory and risk level is acceptable | | the measures already in place are considered satisfactory and risk level is acceptable | ||
|- | |- | ||
Line 112: | Line 108: | ||
We have hardware shared with other projects that can be used temporary | We have hardware shared with other projects that can be used temporary | ||
In case of major distruption, the service can be migrated to another cloud provider | In case of major distruption, the service can be migrated to another cloud provider | ||
| style="background: | | style="background: yellow"| Medium | ||
| | | 1 or more working days | ||
| the measures already in place are considered satisfactory and risk level is acceptable | | the measures already in place are considered satisfactory and risk level is acceptable | ||
|- | |- | ||
Line 122: | Line 118: | ||
Full daily backup of the VM. | Full daily backup of the VM. | ||
| style="background: green"| Low | | style="background: green"| Low | ||
| | | up to 4 hours | ||
| the measures already in place are considered satisfactory and risk level is acceptable | | the measures already in place are considered satisfactory and risk level is acceptable | ||
|- | |- | ||
Line 131: | Line 127: | ||
The firewall helps to prevent this kind of attacks. | The firewall helps to prevent this kind of attacks. | ||
| style="background: green"| Low | | style="background: green"| Low | ||
| | | up to 8 hours | ||
| the measures already in place are considered satisfactory and risk level is acceptable | | the measures already in place are considered satisfactory and risk level is acceptable | ||
|} | |} | ||
== Outcome == | == Outcome == | ||
The level of all the identified risks is acceptable and the countermeasures already adopted are considered satisfactory | The level of all the identified risks is acceptable and the countermeasures already adopted are considered satisfactory | ||
== Additional information == | == Additional information == | ||
* The provider has got procedures for the several countermeasures to invoke in case of risk occurrence | |||
* The Availability targets don't change in case the plan is invoked. | |||
The support unit | * Recovery requirements (in general they are the outcomes of the BIA). | ||
** Maximum tolerable period of disruption (MTPoD) (the maximum amount of time that a service can be unavailable or undelivered after an event that causes disruption to operations, before its stakeholders perceive unacceptable consequences): <p style="color: blue>to be defined</p> | |||
** Recovery time objective (RTO) (the acceptable amount of time to restore the service in order to avoid unacceptable consequences associated with a break in continuity (this has to be less than MTPoD)): <p style="color: blue>to be defined</p> | |||
** Recovery point objective (RPO) (the acceptable latency of data that will not be recovered): <p style="color: blue>to be defined</p> | |||
* Approach for the return to normal working conditions as reported in the risk assessment. | |||
* The support unit '''Infrastructure Manager''' shall be used to report any incident or service request | |||
* The providers can contact EGI Operations via ticket or email in case the continuity plan is invoked, or to discuss any change to it. | |||
= Availability and Continuity test = | = Availability and Continuity test = | ||
The proposed A/C test will focus on a recovery scenario: the service is supposed to have been disrupted and needs to be reinstalled from scratch. Typically this covers the risks 1,2, and 7. | The proposed A/C test will focus on a recovery scenario: the service is supposed to have been disrupted and needs to be reinstalled from scratch. Typically this covers the risks 1,2, and 7. | ||
The last backup of the data will be used for restoring the service, verifying how much information will be lost, and the time spent will be measured. | The last backup of the data will be used for restoring the service, verifying how much information will be lost, and the time spent will be measured. | ||
Performing this test will be useful to spot any issue in the recovery procedures of the service. | Performing this test will be useful to spot any issue in the recovery procedures of the service. | ||
Line 167: | Line 157: | ||
== Test outcome == | == Test outcome == | ||
The whole system can be recovered from the VMs backup in less than 4 hours. | |||
= Revision History = | = Revision History = | ||
Line 178: | Line 169: | ||
|- | |- | ||
| | | | ||
| | | Alessandro Paolini | ||
| | | 2022-01-14 | ||
| | | filled several information in; recovery requirements to be defined. | ||
|- | |- | ||
| | | |
Latest revision as of 11:54, 14 January 2022
Main | EGI.eu operations services | Support | Documentation | Tools | Activities | Performance | Technology | Catch-all Services | Resource Allocation | Security |
Documentation menu: | Home • | Manuals • | Procedures • | Training • | Other • | Contact ► | For: | VO managers • | Administrators |
Back to main page: Services Availability Continuity Plans
Introduction
This page reports on the Availability and Continuity Plan for the Infrastructure Manager (IM) service and it is the result of the risks assessment conducted for this service: a series of risks and treats has been identified and analysed, along with the correspondent countermeasures currently in place. Whenever a countermeasure is not considered satisfactory for either avoiding or reducing the likelihood of the occurrence of a risk, or its impact, it is agreed with the service provider a new treatment for improving the availability and continuity of the service. The process is concluded with an availability and continuity test.
Last | Next | |
---|---|---|
Risks assessment | 2021-10-15 | 2022 Q3 |
Av/Co plan and test |
previous plans are collected here: https://documents.egi.eu/document/3858
Hardware HA Configuration
All the components of the service (IM service, IM Web and IM Dashboard) are deployed on top of a Kubernetes cluster. In particular the IM service is served behind an HAproxy used as a load balancer using at least four IM server instances spread in at least two different working nodes, for increased redundancy.
Availability requirements and performances
In the OLA it was agreed the following performances targets, on a monthly basis:
- Availability: 95%
- Reliability: 95%
Other availability requirements:
- The service can be accessed using a CLI and a web UI.
- To access the web UI EGI-Chekin credential is required.
- The client only needs to set the EGI-Chekin credentials to access EGI Cloud compute sites.
- In case of using an on premise cloud provider, any user can use the CLI.
The service availability is regularly tested by nagios probes eu.egi.grycap.IM-Check: https://argo-mon.egi.eu/nagios/cgi-bin/status.cgi?servicegroup=SERVICE_es.upv.grycap.im&style=overview
The performances reports in terms of Availability and Reliability are produced by ARGO on an almost real time basis and they are also periodically collected into the Documentation Database.
Risks assessment and management
For more details, please look at the (google spreadsheet link). We will report here a summary of the assessment.
Risks analysis
Risk id | Risk description | Affected components | Established measures | Risk level | Expected duration of downtime / time for recovery | Comment |
---|---|---|---|---|---|---|
1 | Service unavailable / loss of data due to hardware failure | All | The IM services are deployed in a K8s cluster with 1 FE and 2 WNs. WNs are hosted in different nodes so if one of the nodes fails the cluster continues working.
Furthermore there are daily backups of the whole VMs. In case of error it can be migrated to another cloud provider. Also we have RADL/ansible recipes to re-deploy the full K8s and IM services |
Low | up to 4 hours | the measures already in place are considered satisfactory and risk level is acceptable |
2 | Service unavailable / loss of data due to software failure | All | The IM services are deployed in a K8s cluster with 1 FE and 2 WNs. WNs are hosted in different nodes so if one of the nodes fails the cluster continues working.
Furthermore there are daily backups of the whole VMs. In case of error it can be migrated to another cloud provider. Also we have RADL/ansible recipes to re-deploy the full K8s and IM services. The software, the SO and availability of them is regularly checked using nagios |
Low | up to 4 hours | the measures already in place are considered satisfactory and risk level is acceptable |
3 | service unavailable / loss of data due to human error | All | We have a CI/CD process in our jenkins intance. All the changes made at master branch are deployed into the IM devel instance. When a new IM version is released the pord instance is updated | Low | up to 4 hours | the measures already in place are considered satisfactory and risk level is acceptable |
4 | service unavailable for network failure (Network outage with causes external of the site) | All | Network managed centrally at UPV with established measures to mitigate issues | Medium | up to 4 hours | the measures already in place are considered satisfactory and risk level is acceptable |
5 | Unavailability of key technical and support staff (holidays period, sickness, ...) | All | The technical staff is available on a best effort basis, during holidays, and it is possible to access using VPN.
Some other technical staff is available to address urgent issues (such as power interruption or similar) |
Medium | 1 or more working days | the measures already in place are considered satisfactory and risk level is acceptable |
6 | Major disruption in the data centre. Fire, flood or electric failure for example | All | We have spare parts to replace hardware damages.
We have hardware shared with other projects that can be used temporary In case of major distruption, the service can be migrated to another cloud provider |
Medium | 1 or more working days | the measures already in place are considered satisfactory and risk level is acceptable |
7 | Major security incident. The system is compromised by external attackers and needs to be reinstalled and restored. | All | Security updates are applied regularly.
Full daily backup of the VM. |
Low | up to 4 hours | the measures already in place are considered satisfactory and risk level is acceptable |
8 | (D)DOS attack. The service is unavailable because of a coordinated DDOS. | All | Network is monitored,
The firewall helps to prevent this kind of attacks. |
Low | up to 8 hours | the measures already in place are considered satisfactory and risk level is acceptable |
Outcome
The level of all the identified risks is acceptable and the countermeasures already adopted are considered satisfactory
Additional information
- The provider has got procedures for the several countermeasures to invoke in case of risk occurrence
- The Availability targets don't change in case the plan is invoked.
- Recovery requirements (in general they are the outcomes of the BIA).
- Maximum tolerable period of disruption (MTPoD) (the maximum amount of time that a service can be unavailable or undelivered after an event that causes disruption to operations, before its stakeholders perceive unacceptable consequences):
to be defined
- Recovery time objective (RTO) (the acceptable amount of time to restore the service in order to avoid unacceptable consequences associated with a break in continuity (this has to be less than MTPoD)):
to be defined
- Recovery point objective (RPO) (the acceptable latency of data that will not be recovered):
to be defined
- Maximum tolerable period of disruption (MTPoD) (the maximum amount of time that a service can be unavailable or undelivered after an event that causes disruption to operations, before its stakeholders perceive unacceptable consequences):
- Approach for the return to normal working conditions as reported in the risk assessment.
- The support unit Infrastructure Manager shall be used to report any incident or service request
- The providers can contact EGI Operations via ticket or email in case the continuity plan is invoked, or to discuss any change to it.
Availability and Continuity test
The proposed A/C test will focus on a recovery scenario: the service is supposed to have been disrupted and needs to be reinstalled from scratch. Typically this covers the risks 1,2, and 7. The last backup of the data will be used for restoring the service, verifying how much information will be lost, and the time spent will be measured.
Performing this test will be useful to spot any issue in the recovery procedures of the service.
Test details
Test outcome
The whole system can be recovered from the VMs backup in less than 4 hours.
Revision History
Version | Authors | Date | Comments |
---|---|---|---|
Alessandro Paolini | 2022-01-14 | filled several information in; recovery requirements to be defined. | |