Alert.png The wiki is deprecated and due to be decommissioned by the end of September 2022.
The content is being migrated to other supports, new updates will be ignored and lost.
If needed you can get in touch with EGI SDIS team using operations @ egi.eu.

Difference between revisions of "Agenda-2018-03-12"

From EGIWiki
Jump to navigation Jump to search
 
(39 intermediate revisions by 2 users not shown)
Line 1: Line 1:
{{Template:Op menubar}} {{Template:Doc_menubar}} {{TOC_right}}  
{{Template:Op menubar}} {{Template:Doc_menubar}} {{TOC_right}}  
[[Category:Grid Operations Meetings]]
[[Category:Grid Operations Meetings]]
'''AGENDA CONTENT UNDER CONSTRUCTION'''


= General information =
= General information =
Line 10: Line 8:
== UMD/CMD ==
== UMD/CMD ==
* UMD4 regular release (4.7.0) planned for April 2018; dedicated updates are always possible
* UMD4 regular release (4.7.0) planned for April 2018; dedicated updates are always possible
* '''UMD3 shutdown plan'''
* '''UMD3 shutdown plan'''  
** WMS dismission ongoing
** UMD team tested upgrading the umd-release package from UMD3/SL6 to UMD4/SL6, tests are OK, instructions to be broadcast to NGIs/sites
** UMD team tested upgrading the umd-release package from UMD3/SL6 to UMD4/SL6, tests are OK, instructions to be broadcast to NGIs/sites
** please remember that during the switch you can jump to new major versions, so '''PLEASE CHECK RELEASE NOTES CAREFULLY BEFORE UPGRADING AND SET PROPER DOWNTIMES WHERE NEEDED'''
** please remember that during the switch you can jump to new major versions, so '''PLEASE CHECK RELEASE NOTES CAREFULLY BEFORE UPGRADING AND SET PROPER DOWNTIMES WHERE NEEDED'''
Line 19: Line 16:
*** [https://wiki.egi.eu/wiki/UMD_products reference to products in UMD]
*** [https://wiki.egi.eu/wiki/UMD_products reference to products in UMD]
*** [http://repository.egi.eu/category/umd_releases/distribution/umd-4/ official EGI repository page for UMD4]
*** [http://repository.egi.eu/category/umd_releases/distribution/umd-4/ official EGI repository page for UMD4]
*** [https://forge.puppet.com/egiqc/umd UMD puppet modules] and [https://galaxy.ansible.com/egi-qc/umd/ UMD ansible modules] are puppet/ansible modules adapted/validated by the UMD team to fit the UMD verification process


For any kind of operational issues, please contact operations at egi.eu or open a ticket to the Operations SU in GGUS
For any kind of operational issues, please contact operations at egi.eu or open a ticket to the Operations SU in GGUS
For issues with the distribution, please open a ticket to the Software Provisioning SU in GGUS
For issues with the distribution, please open a ticket to the Software Provisioning SU in GGUS
* UMD 4.6.1 fix release in progress, to be released today/tomorrow
** APEL-SSM 2.2.0, DPM 1.9.2, XRootD 4.7.1, CVMFS server 2.4.4, CVMFS 2.4.4, APEL 1.6.1. All updates both for CentOS7 and SL6


*CMD-OS update still in preparation: found SR for cloudkeeper, but unable to get packages for Ubuntu; no way to include yet user id isolatin patch for Mitaka/Ubuntu
*CMD-OS update still in preparation: found SR for cloudkeeper, but unable to get packages for Ubuntu; no way to include yet user id isolatin patch for Mitaka/Ubuntu
* aside the official configuration procedures provided by the single Technology Providers, [https://forge.puppet.com/egiqc/umd UMD puppet modules] and [https://galaxy.ansible.com/egi-qc/umd/ UMD ansible modules] are puppet/ansible modules adapted/validated by the UMD team to fit the UMD verification process (see Pablo's presentation in the Indico agenda)


== Preview repository  ==
== Preview repository  ==
*Release on 2018-03-07:
** '''[[Preview 1.17.0]]''' [https://appdb.egi.eu/store/software/preview.repository/releases/1.0/1.17.0/ AppDB info] (sl6): APEL Client/Server 1.6.1, frontier-squid 3.5.27-3.1, FTS 3.7.8, srm-ifce 1.24.3, STORM 1.11.13, xrootd 4.8.1
** '''[[Preview 2.17.0]]''' [https://appdb.egi.eu/store/software/preview.repository/releases/2.0/2.17.0/ AppDB info] (CentOS 7): APEL Client/Server 1.6.1, frontier-squid 3.5.27-3.1, FTS 3.7.8, srm-ifce 1.24.3, xrootd 4.8.1


= Operations  =
= Operations  =
Line 59: Line 63:
**AsiaPacific:
**AsiaPacific:
***TW-NCHC https://ggus.eu/index.php?mode=ticket_info&ticket_id=133282 (DPM problems solved, statistics are improving)  
***TW-NCHC https://ggus.eu/index.php?mode=ticket_info&ticket_id=133282 (DPM problems solved, statistics are improving)  
**NGI_AEGIS: https://ggus.eu/index.php?mode=ticket_info&ticket_id=132809 (AEGIS01-IPB-SCL)  
**NGI_AEGIS: https://ggus.eu/index.php?mode=ticket_info&ticket_id=132809 (AEGIS01-IPB-SCL) SOLVED
**NGI_IL: https://ggus.eu/index.php?mode=ticket_info&ticket_id=133284 (IL-TAU-HEP, TECHNION-HEP) intermittent failures, asked help to ARGO; misconfiguration of the lcg-voms2.cern.ch vomses and lsc files.
**NGI_IL: https://ggus.eu/index.php?mode=ticket_info&ticket_id=133284 (IL-TAU-HEP, TECHNION-HEP) (SOLVED) intermittent failures, asked help to ARGO; misconfiguration of the lcg-voms2.cern.ch vomses and lsc files.
**NGI_IT https://ggus.eu/index.php?mode=ticket_info&ticket_id=133285 (IGI-BOLOGNA INFN-BOLOGNA-T3 INFN-CNAF-LHCB INFN-T1) the sites are slowly coming back online
**NGI_IT https://ggus.eu/index.php?mode=ticket_info&ticket_id=133285 (IGI-BOLOGNA INFN-BOLOGNA-T3 INFN-CNAF-LHCB INFN-T1) the sites are slowly coming back online
**NGI_PL: https://ggus.eu/index.php?mode=ticket_info&ticket_id=132815 PSNC (intermittent failures with QCG and SRM)
**NGI_PL: https://ggus.eu/index.php?mode=ticket_info&ticket_id=132815 PSNC (intermittent failures with QCG and SRM)


*Underperformed sites after 3 consecutive months, underperformed NGIs, QoS violations:  
*Underperformed sites after 3 consecutive months, underperformed NGIs, QoS violations:  
 
**NGI_BG BG05-SUGrid https://ggus.eu/index.php?mode=ticket_info&ticket_id=133884
**NGI_CH: https://ggus.eu/index.php?mode=ticket_info&ticket_id=133885
***T3_CH_PSI
***UNIBE-ID
***UNIGE-DPNC
**NGI_PL: CYFRONET-LCG2 https://ggus.eu/index.php?mode=ticket_info&ticket_id=133886 
   
   


Line 72: Line 81:


== NGI ARGUS servers not properly configured  ==
== NGI ARGUS servers not properly configured  ==
'''IMPORTANT: The ARGUS version that fixes the PAP permissions has been released in UMD 4.6: http://repository.egi.eu/2017/12/18/release-umd-4-6-0/'''
'''Please update to this version.'''


As reported by EGI-CSIRT during the [https://indico.egi.eu/indico/event/3849/ February OMB], several NGI ARGUS servers are not properly configured and they are not passing the nagios tests (eu.egi.Argus-DNs probe):
As reported by EGI-CSIRT during the [https://indico.egi.eu/indico/event/3849/ February OMB], several NGI ARGUS servers are not properly configured and they are not passing the nagios tests (eu.egi.Argus-DNs probe):
Line 77: Line 89:
https://argo-mon.egi.eu/nagios/cgi-bin/status.cgi?servicegroup=SERVICE_ngi.ARGUS&style=detail
https://argo-mon.egi.eu/nagios/cgi-bin/status.cgi?servicegroup=SERVICE_ngi.ARGUS&style=detail


Information about the nagios error and possible solutions: https://wiki.egi.eu/wiki/EGI_CSIRT:Central_emergency_suspension#NGI_Argus_Monitoring
How the probe works:
*Central Argus server suspend 1 fake DN each day
*Argo looks for this DN in NGI Arguses
Requirements for passing the test:
*NGI Argus must be configured to fetch DNs
*Argo must be able to connect to NGI Argus
*Argo DN must be authorized to query DNs
 
Information about the nagios errors and possible solutions: https://wiki.egi.eu/wiki/EGI_CSIRT:Central_emergency_suspension#NGI_Argus_Monitoring


Information on global banning set-up: https://wiki.nikhef.nl/grid/Argus_Global_Banning_Setup_Overview#NGI_Argus
Information on global banning set-up: https://wiki.nikhef.nl/grid/Argus_Global_Banning_Setup_Overview#NGI_Argus
'''Authorization error''':
*AsiaPacific https://ggus.eu/index.php?mode=ticket_info&ticket_id=133862 (SOLVED)
*NGI_ARMGRID https://ggus.eu/index.php?mode=ticket_info&ticket_id=133859
*NGI_CHINA https://ggus.eu/index.php?mode=ticket_info&ticket_id=133860 (SOLVED)
*NGI_GE https://ggus.eu/index.php?mode=ticket_info&ticket_id=133861 (SOLVED)
*NGI_GRNET https://ggus.eu/index.php?mode=ticket_info&ticket_id=133858 (SOLVED)
*NGI_HU https://ggus.eu/index.php?mode=ticket_info&ticket_id=133863 (SOLVED)
*NGI_IBERGRID https://ggus.eu/index.php?mode=ticket_info&ticket_id=133864 (SOLVED)
*NGI_IT https://ggus.eu/index.php?mode=ticket_info&ticket_id=133857 (SOLVED)
*NGI_PL https://ggus.eu/index.php?mode=ticket_info&ticket_id=133856 (SOLVED)
*ROC_CANADA https://ggus.eu/index.php?mode=ticket_info&ticket_id=133865 (SOLVED)
'''Connection error''':
*NGI_AEGIS https://ggus.eu/index.php?mode=ticket_info&ticket_id=133869 (SOLVED)
*NGI_CH https://ggus.eu/index.php?mode=ticket_info&ticket_id=133870 (SOLVED)
*NGI_FI https://ggus.eu/index.php?mode=ticket_info&ticket_id=133872
*NGI_FRANCE https://ggus.eu/index.php?mode=ticket_info&ticket_id=133873
*NGI_HR https://ggus.eu/index.php?mode=ticket_info&ticket_id=133867
*NGI_NL https://ggus.eu/index.php?mode=ticket_info&ticket_id=133868 (SOLVED)
*NGI_TR https://ggus.eu/index.php?mode=ticket_info&ticket_id=133871 (SOLVED)
*ROC_LA https://ggus.eu/index.php?mode=ticket_info&ticket_id=133866


== Decommissioning EMI WMS  ==
== Decommissioning EMI WMS  ==
Line 89: Line 131:
**there might be several VOs enabled on your WMS servers: in case only few of them need to use the service for few weeks more, you might disable the other VOs  
**there might be several VOs enabled on your WMS servers: in case only few of them need to use the service for few weeks more, you might disable the other VOs  
*On Dec 14th EGI Operations sent a new broadcast to the VOs reminding the users the forthcoming WMS decommission  
*On Dec 14th EGI Operations sent a new broadcast to the VOs reminding the users the forthcoming WMS decommission  
*After the end of February, EGI Operations will open a ticket to the sites that haven't started the decommission process yet
*From March, the nagios probe [https://wiki.egi.eu/wiki/MW_SAM_tests#General_tests eu.egi.sec.WMS] will return a CRITICAL status for the servers still in production, and the ROD teams will open a ticket to the sites that haven't finished the decommission process yet


'''[https://goc.egi.eu/portal/index.php?Page_Type=Downtimes_Calendar&view=month&certStatus=Certified&service_type=WMS WMS servers in downtime on GOC-DB]'''  
'''[https://goc.egi.eu/portal/index.php?Page_Type=Downtimes_Calendar&view=month&certStatus=Certified&service_type=WMS WMS servers in downtime on GOC-DB]'''  


VOs have to find alternatives or migrate to DIRAC:  
[https://goc.egi.eu/portal/index.php?Page_Type=Services&serviceType=WMS&selectItemserviceType=WMS&ngi=&searchTerm=&production=TRUE&monitored=TRUE&certStatus=Certified&scopeMatch=all&servKeyNames=&servKeyValue= 32 WMS still registered on GOC-DB as production and monitored]
 
*'''the [[HOWTO22]] explains how a VO can request the access to DIRAC4EGI and how interact with it by CLI'''


=== WMS decommissioning status ===
'''[https://argo-mon.egi.eu/nagios/cgi-bin/status.cgi?servicegroup=SERVICE_WMS&style=detail Status on nagios]'''
 
[https://goc.egi.eu/portal/index.php?Page_Type=Services&serviceType=WMS&selectItemserviceType=WMS&ngi=&searchTerm=&production=TRUE&monitored=TRUE&certStatus=Certified&scopeMatch=all&servKeyNames=&servKeyValue= 42 WMS still production and monitored]
 
{| width="830" cellspacing="1" cellpadding="1" border="1"
|+ WMS decommissioning status
|-
| '''Last update'''
| '''NGI/ROC'''
| '''STATUS'''
| '''Comments'''
|-
| 2018-02-12
| NGI_FRANCE
| DONE
| All WMS decommissioned
|-
|
|
|
|
|}
 
<br>


== IPv6 readiness plans  ==
== IPv6 readiness plans  ==
Line 134: Line 151:
*webdav endpoints registered in GOC-DB: https://goc.egi.eu/gocdbpi/public/?method=get_service&amp;&amp;service_type=webdav  
*webdav endpoints registered in GOC-DB: https://goc.egi.eu/gocdbpi/public/?method=get_service&amp;&amp;service_type=webdav  
*link to nagios results: https://argo-mon.egi.eu/nagios/cgi-bin/status.cgi?servicegroup=SERVICE_webdav&amp;style=detail
*link to nagios results: https://argo-mon.egi.eu/nagios/cgi-bin/status.cgi?servicegroup=SERVICE_webdav&amp;style=detail
** '''2 CRITICAL, 2 WARNING out of 23 (OK status: 82.6%)'''


List of sites that not have completed the configuration yet:  
List of sites that not have completed the configuration yet:  


*NGI_AEGIS: '''AEGIS01-IPB-SCL''' https://ggus.eu/index.php?mode=ticket_info&amp;ticket_id=131033 (hardware problems...)  
*NGI_AEGIS: '''AEGIS01-IPB-SCL''' https://ggus.eu/index.php?mode=ticket_info&amp;ticket_id=131033 (hardware problems...)  
*NGI_HR: '''egee.srce.hr''' https://ggus.eu/index.php?mode=ticket_info&amp;ticket_id=131041 (in progress...)
*NGI_HR: '''egee.srce.hr''' https://ggus.eu/index.php?mode=ticket_info&amp;ticket_id=131041 (webdav URL missing in GOC-DB...)
 
List of sites that disabled webdav: UNIGE-DPNC, [https://ggus.eu/index.php?mode=ticket_info&ticket_id=131037 GR-01-AUTH], [https://ggus.eu/index.php?mode=ticket_info&ticket_id=131038 HG-03-AUTH], CETA-GRID, [https://ggus.eu/index.php?mode=ticket_info&ticket_id=131052 WUT]
 
For registering on GOC-DB the webdav service endpoint, follow the [[HOWTO21]] in order to filling in the proper information. '''In particular''':


*register a new service endpoint, separated from the SRM one;
List of sites that disabled webdav: UNIGE-DPNC, [https://ggus.eu/index.php?mode=ticket_info&ticket_id=131037 GR-01-AUTH], [https://ggus.eu/index.php?mode=ticket_info&ticket_id=131038 HG-03-AUTH], CETA-GRID, [https://ggus.eu/index.php?mode=ticket_info&ticket_id=131052 WUT]
*on GOC-DB fill in the webdav URL containing also the VO ops folder, for example: https://darkstorm.cnaf.infn.it:8443/webdav/ops or https://hepgrid11.ph.liv.ac.uk/dpm/ph.liv.ac.uk/home/ops/
**it corresponds to the value of GLUE2 attribute GLUE2EndpointURL (containing the used port and without the VO folder);
*verify that the webdav url (for example: https://darkstorm.cnaf.infn.it:8443/webdav ) is properly accessible.


== Storage accounting deployment  ==
== Storage accounting deployment  ==
Line 161: Line 172:
After setting up a daily cron job and running the accounting software, look for your data in the Accounting Portal: http://goc-accounting.grid-support.ac.uk/storagetest/storagesitesystems.html. If it does not appear within 24 hours, or there are other errors, please open a GGUS ticket to APEL who will help debug the process.  
After setting up a daily cron job and running the accounting software, look for your data in the Accounting Portal: http://goc-accounting.grid-support.ac.uk/storagetest/storagesitesystems.html. If it does not appear within 24 hours, or there are other errors, please open a GGUS ticket to APEL who will help debug the process.  


'''PROBLEM''': several (DPM) sites are using an old version of the star-accounting.py script. This leads to records having an EndTime 30 days in the future. The star-accounting.py script version to use is v1.0.4 (http://svnweb.cern.ch/world/wsvn/lcgdm/lcg-dm/trunk/scripts/StAR-accounting/star-accounting.py).
'''IMPORTANT''': Do not encrypt the storage records with your host certificate, please comment out the “server_cert” variable in sender.cfg
 
The APEL team opened tickets for this issue:
 
*'''AEGIS02-RCUB''': https://ggus.eu/index.php?mode=ticket_info&amp;ticket_id=131892 (SOLVED)
*'''AEGIS03-ELEF-LEDA''': https://ggus.eu/index.php?mode=ticket_info&amp;ticket_id=131893 (SOLVED)
*'''AUVERGRID''': https://ggus.eu/index.php?mode=ticket_info&amp;ticket_id=131894 (SOLVED)
*'''CAMK''': https://ggus.eu/index.php?mode=ticket_info&amp;ticket_id=131895 (SOLVED)
*'''CETA-GRID''': https://ggus.eu/index.php?mode=ticket_info&amp;ticket_id=131896 (SOLVED)
*'''GARR-01-DIR''': https://ggus.eu/index.php?mode=ticket_info&amp;ticket_id=131897 (SOLVED)
*'''IN2P3-LPC''': https://ggus.eu/index.php?mode=ticket_info&amp;ticket_id=131917 (SOLVED)
*'''RO-02-NIPNE''': https://ggus.eu/index.php?mode=ticket_info&amp;ticket_id=131918 (SOLVED)
*'''RO-07-NIPNE''': https://ggus.eu/index.php?mode=ticket_info&amp;ticket_id=131920 (SOLVED)
*'''TOKYO-LCG2''': https://ggus.eu/index.php?mode=ticket_info&amp;ticket_id=131921 (SOLVED)
*'''TW-NTU-HEP''': https://ggus.eu/index.php?mode=ticket_info&amp;ticket_id=131923 (SOLVED)
*'''UA-ISMA''': https://ggus.eu/index.php?mode=ticket_info&amp;ticket_id=131925
*'''UKI-NORTHGRID-SHEF-HEP''': https://ggus.eu/index.php?mode=ticket_info&amp;ticket_id=131926 (SOLVED)
*'''TASK''': https://ggus.eu/index.php?mode=ticket_info&amp;ticket_id=131928 (SOLVED)
 
'''PROBLEM number 2''': the APEL repository is receiving an increasing number of storage records that have been encrypted with something that isn’t the APEL certificate, so the records can’t be read them (and so the sender is unknown). If your site isn’t successfully publishing, please comment out the “server_cert” variable in sender.cfg


List of sites already publishing and of tickets opened is '''[[Storage accounting deployment|reported here]]'''. Several sites are not publishing the storage accounting data yet. '''NGIs please follow-up with the sites the configuration of the script in order to speed-up the process.'''
'''List of sites already publishing and of tickets opened is [[Storage accounting deployment|reported here]]'''. Several sites are not publishing the storage accounting data yet. '''NGIs please follow-up with the sites the configuration of the script in order to speed-up the process.'''


= AOB  =
= AOB  =
Line 188: Line 180:
*NGI_FRANCE reorganizing the '''national operations following a''' '''distributed model''' (not anymore operated by <span class="im">CC-IN2P3</span>); are other NGIs reorganising as well? how? do you have suggestions?  
*NGI_FRANCE reorganizing the '''national operations following a''' '''distributed model''' (not anymore operated by <span class="im">CC-IN2P3</span>); are other NGIs reorganising as well? how? do you have suggestions?  
*do you have suggestions to '''improve the EGI Operations''' meeting itself or to '''improve the distribution of the items discussed/reported between EGI Operations and OMB'''?
*do you have suggestions to '''improve the EGI Operations''' meeting itself or to '''improve the distribution of the items discussed/reported between EGI Operations and OMB'''?
* Operations Portal VO cards issue: VO ID card change request submitted by WLCG to introduce multi-core parameters as a set of explicit fields instead of a free-text block. https://ggus.eu/index.php?mode=ticket_info&ticket_id=133915


== Next meeting  ==
== Next meeting  ==


*'''Mar 12th, 2018''' https://indico.egi.eu/indico/event/3639/
*'''Apr 9th, 2018''' https://indico.egi.eu/indico/event/3911/

Latest revision as of 09:16, 20 March 2018

Main EGI.eu operations services Support Documentation Tools Activities Performance Technology Catch-all Services Resource Allocation Security


Documentation menu: Home Manuals Procedures Training Other Contact For: VO managers Administrators


General information

Middleware

UMD/CMD

  • UMD4 regular release (4.7.0) planned for April 2018; dedicated updates are always possible
  • UMD3 shutdown plan

For any kind of operational issues, please contact operations at egi.eu or open a ticket to the Operations SU in GGUS For issues with the distribution, please open a ticket to the Software Provisioning SU in GGUS

  • UMD 4.6.1 fix release in progress, to be released today/tomorrow
    • APEL-SSM 2.2.0, DPM 1.9.2, XRootD 4.7.1, CVMFS server 2.4.4, CVMFS 2.4.4, APEL 1.6.1. All updates both for CentOS7 and SL6
  • CMD-OS update still in preparation: found SR for cloudkeeper, but unable to get packages for Ubuntu; no way to include yet user id isolatin patch for Mitaka/Ubuntu
  • aside the official configuration procedures provided by the single Technology Providers, UMD puppet modules and UMD ansible modules are puppet/ansible modules adapted/validated by the UMD team to fit the UMD verification process (see Pablo's presentation in the Indico agenda)

Preview repository

  • Release on 2018-03-07:
    • Preview 1.17.0 AppDB info (sl6): APEL Client/Server 1.6.1, frontier-squid 3.5.27-3.1, FTS 3.7.8, srm-ifce 1.24.3, STORM 1.11.13, xrootd 4.8.1
    • Preview 2.17.0 AppDB info (CentOS 7): APEL Client/Server 1.6.1, frontier-squid 3.5.27-3.1, FTS 3.7.8, srm-ifce 1.24.3, xrootd 4.8.1

Operations

GGUS Support Unit review

  • EGCF 2015 -> contacted email
  • EMI WN 2015 -> to be changed together with UI
  • EMIR 2013 -> contacted email
  • OpenNebula 2015 -> to be reviewed it in the context of redefinition of the fedcloud related SU
  • rOCCI 2015 -> to be reviewed it in the context of redefinition of the fedcloud related SU
  • UNICORE-Client 2013 -> can be closed

ARGO/SAM


FedCloud

  • hardening FedCloud Appliances in App-DB (in progress)
  • cloudkeeper for OpenStack (the vmcatcher replacement) cannot yet be distributed for Mitaka/Ubuntu through CMD (missing packages), proposed to FedCloud TF an installation campaign to bypass (this time) the UMD process

Feedback from Helpdesk

Monthly Availability/Reliability


suspended sites:

  • PK-CIIT and TW-FTT for security reasons

NGI ARGUS servers not properly configured

IMPORTANT: The ARGUS version that fixes the PAP permissions has been released in UMD 4.6: http://repository.egi.eu/2017/12/18/release-umd-4-6-0/ Please update to this version.

As reported by EGI-CSIRT during the February OMB, several NGI ARGUS servers are not properly configured and they are not passing the nagios tests (eu.egi.Argus-DNs probe):

https://argo-mon.egi.eu/nagios/cgi-bin/status.cgi?servicegroup=SERVICE_ngi.ARGUS&style=detail

How the probe works:

  • Central Argus server suspend 1 fake DN each day
  • Argo looks for this DN in NGI Arguses

Requirements for passing the test:

  • NGI Argus must be configured to fetch DNs
  • Argo must be able to connect to NGI Argus
  • Argo DN must be authorized to query DNs

Information about the nagios errors and possible solutions: https://wiki.egi.eu/wiki/EGI_CSIRT:Central_emergency_suspension#NGI_Argus_Monitoring

Information on global banning set-up: https://wiki.nikhef.nl/grid/Argus_Global_Banning_Setup_Overview#NGI_Argus

Authorization error:

Connection error:

Decommissioning EMI WMS

WMS servers can be decommissioned. Please follow the procedure PROC12. The plan is:

  • Starting from January 2018, put the WMS servers in draining: this will block the submission of new jobs and will allow the jobs previously submitted to finish
    • inform in advance your users that you are going to put in draining and then dismiss the WMS servers (as per PROC12)
    • there might be several VOs enabled on your WMS servers: in case only few of them need to use the service for few weeks more, you might disable the other VOs
  • On Dec 14th EGI Operations sent a new broadcast to the VOs reminding the users the forthcoming WMS decommission
  • From March, the nagios probe eu.egi.sec.WMS will return a CRITICAL status for the servers still in production, and the ROD teams will open a ticket to the sites that haven't finished the decommission process yet

WMS servers in downtime on GOC-DB

32 WMS still registered on GOC-DB as production and monitored

Status on nagios

IPv6 readiness plans

webdav probes in OPERATORS profile

The webdav probes was included in the ARGO_MON_OPERATORS profile after the approval in the January OMB: in this way the failures will generate an alarm on the dashboard, and the ROD teams can open a ticket to the failing sites. If no particular issue occurs, and if at least 75% of webdav endpoint are passing the tests, the probes will be added in the ARGO_MON_CRITICAL profile, so the results of these probes will be taken into account for the A/R figures.

List of sites that not have completed the configuration yet:

List of sites that disabled webdav: UNIGE-DPNC, GR-01-AUTH, HG-03-AUTH, CETA-GRID, WUT

Storage accounting deployment

During the September meeting, OMB has approved the full-scale deployment of storage accounting. The APEL team has tested it with a group of early adopters sites, and the results prove that storage accounting is now production-ready.

Storage accounting is currently supported only for the DPM and dCache storage elements therefore only the resource centres deploying these kind of storage elements are requested to publish storage accounting data.

In order to properly install and configure the storage accounting scripts, please follow the instructions reported in the wiki: https://wiki.egi.eu/wiki/APEL/Storage

IMPORTANT: be sure to have installed the star-accounting.py script v1.0.4 (http://svnweb.cern.ch/world/wsvn/lcgdm/lcg-dm/trunk/scripts/StAR-accounting/star-accounting.py)

After setting up a daily cron job and running the accounting software, look for your data in the Accounting Portal: http://goc-accounting.grid-support.ac.uk/storagetest/storagesitesystems.html. If it does not appear within 24 hours, or there are other errors, please open a GGUS ticket to APEL who will help debug the process.

IMPORTANT: Do not encrypt the storage records with your host certificate, please comment out the “server_cert” variable in sender.cfg

List of sites already publishing and of tickets opened is reported here. Several sites are not publishing the storage accounting data yet. NGIs please follow-up with the sites the configuration of the script in order to speed-up the process.

AOB

  • NGI_FRANCE reorganizing the national operations following a distributed model (not anymore operated by CC-IN2P3); are other NGIs reorganising as well? how? do you have suggestions?
  • do you have suggestions to improve the EGI Operations meeting itself or to improve the distribution of the items discussed/reported between EGI Operations and OMB?

Next meeting