Alert.png The wiki is deprecated and due to be decommissioned by the end of September 2022.
The content is being migrated to other supports, new updates will be ignored and lost.
If needed you can get in touch with EGI SDIS team using operations @ egi.eu.

Difference between revisions of "WI03 RC and RP OLA violation report followup"

From EGIWiki
Jump to navigation Jump to search
(16 intermediate revisions by 3 users not shown)
Line 39: Line 39:
#'''SLM''': Produce '''Availability/Reliability report'''  
#'''SLM''': Produce '''Availability/Reliability report'''  
#'''SLM:''' Create DOC DB entry and add link to  https://wiki.egi.eu/wiki/Availability_and_reliability_monthly_statistics  
#'''SLM:''' Create DOC DB entry and add link to  https://wiki.egi.eu/wiki/Availability_and_reliability_monthly_statistics  
#'''SLM''': Send mail to operations @ egi.eu  
#'''SLM''': Send mail to operations @ egi.eu
#'''CA''': Create summary for
 
#*'''NGIs: underperforming ava/rel'''
<br>
#*'''NGIs: high Unknown'''
#*'''RCs: underpeforming ava/rel for 3 months'''
#'''CA''':&nbsp;Send summary to Opertions@ egi.eu


|-
|-
Line 60: Line 57:
#Create report for '''ROD performance index, NGI&nbsp;SAM&nbsp; and QoS '''<br>  
#Create report for '''ROD performance index, NGI&nbsp;SAM&nbsp; and QoS '''<br>  
#Add reports to DOC&nbsp;DB entry for [https://wiki.egi.eu/wiki/ROD_performance_index#Performance_reports ROD performance index], NGI&nbsp;SAM (same as for BDII)&nbsp; and QoS <br>  
#Add reports to DOC&nbsp;DB entry for [https://wiki.egi.eu/wiki/ROD_performance_index#Performance_reports ROD performance index], NGI&nbsp;SAM (same as for BDII)&nbsp; and QoS <br>  
#Create summary and send to Opertions@ egi.eu
#Create summary and send to Opertions@ egi.eu
#Create summary for <br>#*'''NGIs: underperforming ava/rel''' <br>#*'''NGIs: high Unknown''' <br>#*'''RCs: underpeforming ava/rel for 3 months''' <br>#'''CA''':&amp;nbsp;Send summary to Opertions@ egi.eu


|-
|-
Line 66: Line 64:
| <br>  
| <br>  
| 10<br>  
| 10<br>  
| MK or CA<br>  
| VS<br>  
|  
|  
#Put summary of reports together<br>  
#Put summary of reports together<br>  
#Send email to NOC&nbsp;managers about all reports (link to DOC&nbsp;DB)  
#Send email to NOC&nbsp;managers about all reports (link to DOC&nbsp;DB)  
#Create master and child GGUS tickets against NGIs  
#Create master and child GGUS tickets against NGIs
#Add ticket URL to [[Underperforming sites and suspensions|Underperforming sites and suspensions]]
#Follow-up GGUS tickets against NGIs
#Follow-up GGUS tickets against NGIs  
#&nbsp;Update [[Underperforming sites and suspensions|Underperforming sites and suspensions]]


|}
|}


<br>


 
=== NGI managers email (example)<br> ===
=== NGI managers email (example)<br> ===
 
<pre>Subject:&nbsp;EGI RC OLA and RP OLA Reports for October 2014
<pre>Subject:&nbsp;EGI RC OLA and RP OLA Reports for October 2014


Line 98: Line 93:


Best Regards
Best Regards
</pre>
</pre>  
 
=== DOC&nbsp;DB content (example) ===
=== DOC&nbsp;DB content (example) ===
 
 
<pre>Title:&nbsp;EGI RC OLA and RP OLA Reports for October 2014
<pre>Title:&nbsp;EGI RC OLA and RP OLA Reports for October 2014


Line 117: Line 109:
https://documents.egi.eu/document/463
https://documents.egi.eu/document/463
</pre>  
</pre>  
=== GGUS&nbsp;Ticket content (template) ===
=== GGUS&nbsp;Ticket content (template) ===
<pre>Subject: $NGI_SU - $date - RP/RC OLA violation
<pre>$NGI - $MONTH $YEAR - RP/RC OLA performance


Dear $NGI_SU,
Dear NGI/ROC,


According to the Service targets reports for Resource infrastructure Provider [1] and Resource Center[2] OLA
the EGI RC OLA and RP OLA Report for November 2020 has been produced and is available at the following links:
for $date following problems accures:
- NGIs reports: http://egi.ui.argo.grnet.gr/egi/report-ar/Critical/NGI?month=2020-11 (Clicking on the NGI name, it will be displayed the resource centres A/R figures)
- RCs reports: http://egi.ui.argo.grnet.gr/egi/report-ar/Critical/SITES?month=2020-11


Report: https://documents.egi.eu/public/ShowDocument?docid=2352
According to the Service targets reports for Resource infrastructure Provider [1] and Resource Centre[2] OLAs, the following problems occurred:


============= RC Availability Reliability [2]==========
============= RC Availability Reliability [2]==========


According to recent availability/reliability report following sites have achieved insufficient performance below Availability target
According to recent availability/reliability report following sites have achieved insufficient performance below Availability target threshold in 3 consecutive months (September, October, and November):
threshold in 3 consecutive months:
 
* site name - ava:  rel:
$SITE
*
$SITE
*
$SITE


* The sites will be suspended 10 working days after receiving this ticket unless NGI intervene. *
* During the 10 working days after receiving this ticket the NGI can suspend the site or ask to not suspend the site by providing adequate explanation. If no answer is provided to this ticket, the NGI will be contacted by email; if no reply will be provided to the email, the site will be suspended[6].  


If NGI intervene and performance is still below targets 3 days after the intervention, the site will also be suspended.
If NGI intervene and performance is still below targets 3 days after the intervention, the site will also be suspended.


If you think that the site should not be suspended please provide justification in this ticket within 10  
If you think that the site should not be suspended please provide justification in this ticket within 10 working days. In case the site performance rises above targets within 3 days from providing explanation, the site will not be suspended. Otherwise EGI Operations may decide on suspension of the site.
working days. In case the site performance rises above targets within 3 days from providing explanation,  
the site will not be suspended. Otherwise EGI Operations may decide on suspension of the site.


============= RP Availability Reliability [1]==========
According to recent availability/reliability report the sites operated by your Operations Center have achieved insufficient
average performance:
- Availability:
- Reliability:
We would like to kindly ask you to take an action to improve your Resource Infrastructure performance.
============= RP Unknown [1]==========
Unknown metric in your NGI has been spotted with higher than 10% value.
Note:
Having a high percentage of UNKNOWN status of sites implies that there is no data available regarding the
site availability for a substantial amount of time. For more information about UNKNOWN status please visit [6]
We would like to kindly ask you to take an action to decrease Unknown metric.
============= Top BDII Availability Reliability [1]==========
According to availability/reliability Top Level BDII report your Top Level BDII achieved insufficient
performance:
- Availability:
- Reliability:
We would like to kindly ask you to take an action to improve your Top-BDII performance.
Note:
If the top-BDII performance was due to problems with middleware or you believe there are errors in the computation,
please send a GGUS ticket to the Service Level Management support unit, following the procedure [4]
If you need information on how to set up a highly available Top-BDII, have a look at [5]
============= SAM Availability Reliability [1]==========
According to availability/reliability SAM report your NGI SAM instance achieved insufficient
performance:
- Availability:
- Reliability:
We would like to kindly ask you to take an action to improve your NGI SAM performance.
============= ROD performance index [1]==========
According to recent report more than 10 items were not handled by your ROD team according to
the escalation procedure described in [3]
No. Tickets expired occurrence:
No. Alarms older than 72h occurrence:
We would like to kindly ask you to take an action to improve your ROD team performance.


============= Quality of Support [1]==========
============= Quality of Support [1]==========
Line 209: Line 144:
top priority (expected 1 working day):   
top priority (expected 1 working day):   


In order to see the details (https://wiki.egi.eu/wiki/Service_Level_Target_-_Quality_of_Support ):
1) go on https://ggus.eu/?mode=report_view
2) choose "response times" or "tickets submitted" and the proper timeframe
3) select your NGI
4) group by Responsible unit and priority
5) click on the lines displayed for getting the tickets details


**********************


Please investigate the reasons for mentioned above situation and provide an explanation for each case in this ticket within 10 days  
Please investigate the reasons for mentioned above situation and provide an explanation for each case in this ticket within 10 days from receiving it.  
from receiving it.  


If no explanation is provided, the EGI Operation will be informed about the situation and may take further steps.
If no explanation is provided, the EGI Operation will be informed about the situation and may take further steps.
Line 222: Line 162:
Links:
Links:


[1] https://documents.egi.eu/public/ShowDocument?docid=463
[1] https://documents.egi.eu/public/ShowDocument?docid=463 "Resource infrastructure Provider Operational Level Agreement"


[2] https://documents.egi.eu/public/ShowDocument?docid=31
[2] https://documents.egi.eu/public/ShowDocument?docid=31 "Resource Centre Operational Level Agreement"


[3] https://wiki.egi.eu/wiki/PROC01
[3] https://wiki.egi.eu/wiki/PROC01 "EGI Infrastructure Oversight escalation"


[4] https://wiki.egi.eu/wiki/PROC10
[4] https://wiki.egi.eu/wiki/PROC10 "Recomputation of SAM results or availability reliability statistics"


[5] https://wiki.egi.eu/wiki/MAN05
[5] https://wiki.egi.eu/wiki/MAN05 "top-BDII and site-BDII High Availability"


[6] https://wiki.egi.eu/wiki/Unknown_issue
[6] https://wiki.egi.eu/wiki/PROC04 "Quality verification of monthly availability and reliability statistics"


Best Regards,
Best Regards,
EGI Operations
EGI Operations
</pre>  
</pre>  
More info about [[Ticket generator Availability Reliability|Ticket generator for A/R]]  
More info about [[Ticket generator Availability Reliability|Ticket generator for A/R]]  

Revision as of 10:28, 7 December 2020

Main EGI.eu operations services Support Documentation Tools Activities Performance Technology Catch-all Services Resource Allocation Security


EGI Infrastructure Operations Oversight menu: Home EGI.eu Operations Team Regional Operators (ROD) 




RC and RP OLA violation work instruction for EGI Operations

This page describes steps which should be taken to follow RC and RP OLA violation issues.

General info

  • Receiver: NGI
  • Subject: RC and RP OLA violation
  • Threshold
  • Goal: We expect to see improvement
  • Deadline for answers: 10 days
  • No response

Steps

Starts with 1 day of the month

Step [#]

Max. Duration [work days]

(time before moving to next step)

Responsible Step
1
1 10
SLM, CA
  1. SLM: Produce Availability/Reliability report
  2. SLM: Create DOC DB entry and add link to  https://wiki.egi.eu/wiki/Availability_and_reliability_monthly_statistics
  3. SLM: Send mail to operations @ egi.eu


2
PS
  1. Create report for NGI Top BDII
  2. Add report to DOC DB entry created by SLM
  3. Create summary and send to opertions@ egi.eu
3 MK
  1. Create report for ROD performance index, NGI SAM  and QoS
  2. Add reports to DOC DB entry for ROD performance index, NGI SAM (same as for BDII)  and QoS
  3. Create summary and send to Opertions@ egi.eu
  4. Create summary for
    #*NGIs: underperforming ava/rel
    #*NGIs: high Unknown
    #*RCs: underpeforming ava/rel for 3 months
    #CA:&nbsp;Send summary to Opertions@ egi.eu
2

10
VS
  1. Put summary of reports together
  2. Send email to NOC managers about all reports (link to DOC DB)
  3. Create master and child GGUS tickets against NGIs
  4. Follow-up GGUS tickets against NGIs


NGI managers email (example)

Subject: EGI RC OLA and RP OLA Reports for October 2014

Content: Dear NGI managers

Please find EGI RC OLA and RP OLA Reports for October 2014 under:

https://documents.egi.eu/public/ShowDocument?docid=2352

Entry includes reports:
* EGI Cloud RC A/R
* EGI RP/RC A/R/U
* EGI RP Quality of Support
* EGI RP ROD performance index
* EGI RP SAM A/R
* EGI RP Top-BDII A/R

Best Regards

DOC DB content (example)

Title: EGI RC OLA and RP OLA Reports for October 2014

Abstract: Container for reports supporting 

Resource Centre Operational Level Agreement

https://documents.egi.eu/document/31

and

Resource infrastructure Provider Operational Level Agreement

https://documents.egi.eu/document/463

GGUS Ticket content (template)

$NGI - $MONTH $YEAR - RP/RC OLA performance

Dear NGI/ROC,	

the EGI RC OLA and RP OLA Report for November 2020 has been produced and is available at the following links:
- NGIs reports: http://egi.ui.argo.grnet.gr/egi/report-ar/Critical/NGI?month=2020-11 (Clicking on the NGI name, it will be displayed the resource centres A/R figures)
- RCs reports: http://egi.ui.argo.grnet.gr/egi/report-ar/Critical/SITES?month=2020-11

According to the Service targets reports for Resource infrastructure Provider [1] and Resource Centre[2] OLAs, the following problems occurred:

============= RC Availability Reliability [2]==========

According to recent availability/reliability report following sites have achieved insufficient performance below Availability target threshold in 3 consecutive months (September, October, and November):

$SITE
$SITE
$SITE 

* During the 10 working days after receiving this ticket the NGI can suspend the site or ask to not suspend the site by providing adequate explanation. If no answer is provided to this ticket, the NGI will be contacted by email; if no reply will be provided to the email, the site will be suspended[6]. 

If NGI intervene and performance is still below targets 3 days after the intervention, the site will also be suspended.

If you think that the site should not be suspended please provide justification in this ticket within 10 working days. In case the site performance rises above targets within 3 days from providing explanation, the site will not be suspended. Otherwise EGI Operations may decide on suspension of the site.


============= Quality of Support [1]==========

According to recent report your NGI achieved insufficient Quality of Support performance: 

less urgent (expected 5 working days): 
urgent (expected 5 working days):
very urgent (expected 1 working day):
top priority (expected 1 working day):  

In order to see the details (https://wiki.egi.eu/wiki/Service_Level_Target_-_Quality_of_Support ):

1) go on https://ggus.eu/?mode=report_view
2) choose "response times" or "tickets submitted" and the proper timeframe
3) select your NGI
4) group by Responsible unit and priority
5) click on the lines displayed for getting the tickets details


Please investigate the reasons for mentioned above situation and provide an explanation for each case in this ticket within 10 days from receiving it. 

If no explanation is provided, the EGI Operation will be informed about the situation and may take further steps.

**********************


Links:

[1] https://documents.egi.eu/public/ShowDocument?docid=463 "Resource infrastructure Provider Operational Level Agreement"

[2] https://documents.egi.eu/public/ShowDocument?docid=31 "Resource Centre Operational Level Agreement"

[3] https://wiki.egi.eu/wiki/PROC01 "EGI Infrastructure Oversight escalation"

[4] https://wiki.egi.eu/wiki/PROC10 "Recomputation of SAM results or availability reliability statistics"

[5] https://wiki.egi.eu/wiki/MAN05 "top-BDII and site-BDII High Availability"

[6] https://wiki.egi.eu/wiki/PROC04 "Quality verification of monthly availability and reliability statistics"

Best Regards,
EGI Operations

More info about Ticket generator for A/R