Alert.png The wiki is deprecated and due to be decommissioned by the end of September 2022.
The content is being migrated to other supports, new updates will be ignored and lost.
If needed you can get in touch with EGI SDIS team using operations @ egi.eu.

Difference between revisions of "Agenda-2020-07-13"

From EGIWiki
Jump to navigation Jump to search
 
(36 intermediate revisions by 2 users not shown)
Line 19: Line 19:


== ARGO/SAM  ==
== ARGO/SAM  ==
*new metrics added to ARGO_MON_OPERATORS profile on [https://ggus.eu/index.php?mode=ticket_info&ticket_id=147169 May 27th]: eu.egi.CREAMCE-JobSubmit, eu.egi.CREAMCE.WN-Csh, eu.egi.CREAMCE.WN-Softver
* [https://argo-mon-fedcloud.cro-ngi.hr/nagios/cgi-bin/status.cgi?servicegroup=SERVICE_org.opensciencegrid.htcondorce&style=detail HTCondor-CE probes] included in the ARGO_MON_OPERATORS profile on May 13th: https://ggus.eu/index.php?mode=ticket_info&ticket_id=146949
** 57 endpoints, 17 CRITICAL, success rate is about 70.2%
** '''on Sept 1st they will be included in the [https://poem.egi.eu/ui/public_metricprofiles/ARGO_MON_CRITICAL ARGO_MON_CRITICAL] profile (A/R computation)'''
*** '''please fix the failures by that date'''
** working on the probe for the host certificate validity check: [https://ggus.eu/index.php?mode=ticket_info&ticket_id=147386 GGUS 147386]
* CREAM-CE metrics in the ARGO_MON_OPERATORS profile on [https://ggus.eu/index.php?mode=ticket_info&ticket_id=147169 May 27th]: eu.egi.CREAMCE-JobSubmit, eu.egi.CREAMCE.WN-Csh, eu.egi.CREAMCE.WN-Softver
** [https://argo-mon.egi.eu/nagios/cgi-bin/status.cgi?servicegroup=SERVICE_CREAM-CE&style=detail results]:  177 endpoints, 15 WARNING (Timeout occurred (900 sec) ), 53 CRITICAL. Success rate 70% (61.6% including the WARNING)
** [https://argo-mon.egi.eu/nagios/cgi-bin/status.cgi?servicegroup=SERVICE_CREAM-CE&style=detail results]:  177 endpoints, 15 WARNING (Timeout occurred (900 sec) ), 53 CRITICAL. Success rate 70% (61.6% including the WARNING)
*When eu.egi.CREAMCE.WN-Softver is successful:
*When eu.egi.CREAMCE.WN-Softver is successful:
Line 37: Line 42:
  **** std.out ****
  **** std.out ****
  ERROR: unable to find glite, EMI, LCG or UMD WN version on n1037-amd
  ERROR: unable to find glite, EMI, LCG or UMD WN version on n1037-amd
* [https://argo-mon-fedcloud.cro-ngi.hr/nagios/cgi-bin/status.cgi?servicegroup=SERVICE_org.opensciencegrid.htcondorce&style=detail HTCondor-CE probes] included in the ARGO_MON_OPERATORS profile on May 13th: https://ggus.eu/index.php?mode=ticket_info&ticket_id=146949
** 56 endpoints, 15 CRITICAL, success rate is about 73.2%


== FedCloud  ==
== FedCloud  ==
Line 66: Line 68:


[https://wiki.egi.eu/wiki/Verify_Configuration_Records#2020-05 List of tickets].
[https://wiki.egi.eu/wiki/Verify_Configuration_Records#2020-05 List of tickets].
* 30 tickets
* Not yet solved after 1 month: 16


== Monthly Availability/Reliability ==
== Monthly Availability/Reliability ==
Line 71: Line 75:
*Under-performed sites in the past A/R reports with issues not yet fixed:
*Under-performed sites in the past A/R reports with issues not yet fixed:
**AfricaArabia: https://ggus.eu/index.php?mode=ticket_info&ticket_id=146877
**AfricaArabia: https://ggus.eu/index.php?mode=ticket_info&ticket_id=146877
***ZA-WITS-CORE: SE hardware problem, machine sent to the vendor; CREAM-CE failures due to a [https://cream-guide.readthedocs.io/en/latest/Releases.html#release-1-16-6 known issue] with the classads library ([https://ggus.eu/index.php?mode=ticket_info&ticket_id=146979 GGUS 146979])
***'''ZA-WITS-CORE''': SE hardware problem, machine sent to the vendor; CREAM-CE failures due to a [https://cream-guide.readthedocs.io/en/latest/Releases.html#release-1-16-6 known issue] with the classads library ([https://ggus.eu/index.php?mode=ticket_info&ticket_id=146979 GGUS 146979])
**AsiaPacific: https://ggus.eu/index.php?mode=ticket_info&ticket_id=142591
**AsiaPacific: https://ggus.eu/index.php?mode=ticket_info&ticket_id=142591
***INDIACMS-TIFR: SRM service not published in the BDII ([https://ggus.eu/index.php?mode=ticket_info&ticket_id=142245 142245]), DPM 1.9.0 version ([https://ggus.eu/index.php?mode=ticket_info&ticket_id=145676 145676]); installing a new DPM headnode
***'''INDIACMS-TIFR''': SRM service not published in the BDII ([https://ggus.eu/index.php?mode=ticket_info&ticket_id=142245 142245]), DPM 1.9.0 version ([https://ggus.eu/index.php?mode=ticket_info&ticket_id=145676 145676]); installing a new DPM headnode
**NGI_CH: https://ggus.eu/index.php?mode=ticket_info&ticket_id=146870
***CSCS-LCG2: intermittent igtf failures
***UNIBE-LHEP: issues with DPM upgrade and some network problems; statistics are improving
**NGI_CH: https://ggus.eu/index.php?mode=ticket_info&ticket_id=145818
***UNIGE-DPNC: new ARC-CE put in production at the end of April, new failures due to an expired certificate
**NGI_DE: https://ggus.eu/index.php?mode=ticket_info&ticket_id=146871
**NGI_DE: https://ggus.eu/index.php?mode=ticket_info&ticket_id=146871
***GoeGRID: CREAM-CE intermittent failures not affecting ATLAS; failures with ARC-CE
***'''GoeGRID''': CREAM-CE intermittent failures not affecting ATLAS; failures with ARC-CE
**NGI_IT: https://ggus.eu/index.php?mode=ticket_info&ticket_id=146868
**NGI_DE: https://ggus.eu/index.php?mode=ticket_info&ticket_id=147313
***CIRMMP: SRM unreachable due to a firewall, statistics are now improving
***'''mainz''': some problems in March and April, that could not be fixed easily; in May, the HPC infrastructure was attacked and the whole computer center was shut down; in downtime.
***INFN-LECCE: CREAMCE and SRM failures, recovering...
***'''wuppertalprod''': SRM failures to to a BDII issue, fixed
**NGI_PL: https://ggus.eu/index.php?mode=ticket_info&ticket_id=140557
***TASK: QCG problems was fixed; some CREAM-CE failures; the SRM tests are still failing and the DPM upgrade isn't completed yet
**NGI_UK:
**NGI_UK:
***UKI-NORTHGRID-SHEF-HEP: https://ggus.eu/index.php?mode=ticket_info&ticket_id=146455 needing to re-install the ARC-CE  
***'''UKI-NORTHGRID-SHEF-HEP''': https://ggus.eu/index.php?mode=ticket_info&ticket_id=146455 ARC-CE re-installed, some condor problems to fix
***UKI-SOUTHGRID-SUSX: https://ggus.eu/index.php?mode=ticket_info&ticket_id=144720 Migration from CREAM to ARC, WN migration to CentOS7; SRM to be decommissioned; ARC-CE was failing the IGTF test
***'''UKI-SOUTHGRID-SUSX''': https://ggus.eu/index.php?mode=ticket_info&ticket_id=144720 Migration from CREAM to ARC, WN migration to CentOS7; SRM to be decommissioned; ARC-CE was failing the IGTF test, then solved; site-bdii failures.
**NGI_UK: https://ggus.eu/index.php?mode=ticket_info&ticket_id=146875
***UKI-LT2-QMUL: intermittent webdav failures, plans to upgrade STORM and LUSTRE in the coming months; suggested to mark the webdav endpoint as "not production"
**ROC_CANADA: https://ggus.eu/index.php?mode=ticket_info&ticket_id=146452
**ROC_CANADA: https://ggus.eu/index.php?mode=ticket_info&ticket_id=146452
***CA-SFU-T2: SLURM problems caused failures to site-BDII freshness check due to some old jobs not properly cancelled; recovered
***'''CA-SFU-T2''': SLURM problems caused failures to site-BDII freshness check due to some old jobs not properly cancelled; recovered
***CA-WATERLOO-T2: SRM failures not involving production VOs, fixed; some unscheduled downtime affected the the A/R figures; some services still to restore.
***'''CA-WATERLOO-T2''': SRM failures not involving production VOs, fixed; some unscheduled downtime affected the the A/R figures; ARC-CE and Site-BDII back in production with a minimum set of resources; A/R figures are improving.
**NGI_UA: https://ggus.eu/index.php?mode=ticket_info&ticket_id=146454
*Under-performed sites after 3 consecutive months, under-performed NGIs, QoS violations: ('''June 2020'''):
***UA-KNU: GlueSEUniqueID wasn't published due to missing DNS back-resolving record for IPv6 address of the SE. Deployed a new ARC-CE version 6, igtf probe returns UNKNOWN.
** NGI_BG: https://ggus.eu/index.php?mode=ticket_info&ticket_id=147747
*Under-performed sites after 3 consecutive months, under-performed NGIs, QoS violations: ('''May 2020'''):
*** '''BG01-IPP'''
**NGI_DE: https://ggus.eu/index.php?mode=ticket_info&ticket_id=147313
** AsiaPacific: https://ggus.eu/index.php?mode=ticket_info&ticket_id=147748
***mainz: some problems in March and April, that could not be fixed easily; in May, the HPC infrastructure was attacked and the whole computer center was shut down.
*** '''HK-HKU-CC-01'''
***wuppertalprod: SRM failures to to a BDII issue, fixed
*** '''TW-NCUHEP'''
**NGI_NDGF: https://ggus.eu/index.php?mode=ticket_info&ticket_id=147310
** NGI_NL: https://ggus.eu/index.php?mode=ticket_info&ticket_id=147749
***FI_HIP_T2: configuration problem that prevents some of the OPS-jobs from running
*** '''SARA-MATRIX''': SRM not published in the BDII: planned a network change to solve the problem.
**NGI_PL: https://ggus.eu/index.php?mode=ticket_info&ticket_id=147311
** NGI_UA: https://ggus.eu/index.php?mode=ticket_info&ticket_id=147750
***NCBJ-CIS
*** '''UA-ISMA''': migration to ARC6 and other planned software updates
***PSNC
***WCSS64
**NGI_UA: https://ggus.eu/index.php?mode=ticket_info&ticket_id=147312
***UA_ICYB_ARC: cluster storage failure, improving...
*sites suspended:
*sites suspended:
**GRID-UNAM (under-performing, DPM upgrade not completed)


== IPv6 readiness plans  ==
== IPv6 readiness plans  ==
Line 128: Line 118:
|-
|-
| 2020-06-08 || 75 || 42 || Some ARC endpoints publish a timestamp instead of a version like 5.X.Y; we can fairly assume they are ARC6 nightly builds, but we're going to close the corresponding tickets after explicit confirmation from the site admin.
| 2020-06-08 || 75 || 42 || Some ARC endpoints publish a timestamp instead of a version like 5.X.Y; we can fairly assume they are ARC6 nightly builds, but we're going to close the corresponding tickets after explicit confirmation from the site admin.
|-
| 2020-07-13 || 53 || 29 || -
|}
|}


== LCGDM end of support and migration to / enabling of DOME ==
== Storage accounting ==


* '''The DPM team has agreed to extend support for security updates for the DPM legacy functionality until 30 September 2019'''. However, affected service providers should still plan to disable legacy mode well before this date.
Many of sites stopped the publication of storage accounting records. Opened [https://ggus.eu/index.php?mode=ticket_search&show_columns_check%5B0%5D=TICKET_TYPE&show_columns_check%5B1%5D=AFFECTED_VO&show_columns_check%5B2%5D=AFFECTED_SITE&show_columns_check%5B3%5D=PRIORITY&show_columns_check%5B4%5D=RESPONSIBLE_UNIT&show_columns_check%5B5%5D=STATUS&show_columns_check%5B6%5D=DATE_OF_CHANGE&show_columns_check%5B7%5D=SHORT_DESCRIPTION&show_columns_check%5B8%5D=SCOPE&su_hierarchy=0&keyword=publishing+storage+accounting+records&specattrib=none&status=all&typeofproblem=all&ticket_category=all&date_type=creation+date&tf_radio=1&timeframe=any&from_date=10+Jul+2020&to_date=11+Jul+2020&orderticketsby=REQUEST_ID&orderhow=desc&search_submit=GO%21&ticket_per_page=60  57 tickets] to fix that.
**more details: https://wiki.egi.eu/wiki/DPM_End_of_legacy-mode_support
* page for checking when the records were published: http://goc-accounting.grid-support.ac.uk/storagetest/storagesitesystems.html
* since [http://lcgdm.web.cern.ch/dpm-1103-released-epel DPM 1.10.3 release], it is possible enabling the non-legacy mode DOME (Disk operations Management Engine, [https://gitlab.cern.ch/lcgdm/dmlite/blob/master/doc/dome/dome.pdf see documentation])
* [http://accounting-devel.egi.eu/storage.php Accounting Portal Prototype view]
**latest [http://lcgdm.web.cern.ch/dmlitedpm-1132-released-epel DPM/dmlite release 1.13.2] has been also released in [http://repository.egi.eu/2020/03/26/release-umd-4-10-0/ UMD 4.10.0]
* '''EGI Operations sent a broadcast to site-admins([https://operations-portal.egi.eu/broadcast/archive/2426 first bunch] and [https://operations-portal.egi.eu/broadcast/archive/2438 second bunch]) and [https://operations-portal.egi.eu/broadcast/archive/2420 VO managers] with a survey to fill in''':
**site-admins: to know the upgrade plans: https://www.surveymonkey.com/r/2PBZSNB
**VO Managers: to know any barriers in moving away from using SRM protocol https://www.surveymonkey.com/r/2ZMZJVG
 
* '''Deployment statistics (Jun 5th):'''
$ ldapsearch -x -LLL -H ldap://egee-bdii.cnaf.infn.it:2170 -b "GLUE2GroupID=grid,o=glue" '(&(objectClass=GLUE2Manager)(GLUE2ManagerProductName=DPM))' GLUE2ManagerProductVersion GLUE2ManagerID | grep GLUE2ManagerProductVersion | sort | uniq -c
      1 GLUE2ManagerProductVersion: 1.10.0
    65 GLUE2ManagerProductVersion: 1.13.0
      2 GLUE2ManagerProductVersion: 1.13.1
    11 GLUE2ManagerProductVersion: 1.13.2
      3 GLUE2ManagerProductVersion: 1.8.10
      1 GLUE2ManagerProductVersion: 1.8.9
      4 GLUE2ManagerProductVersion: 1.9.0
 
 
Liasing with WLCG to follow-up the upgrade. Opened GGUS tickets asking the following:
* all the sites with older DPM versions than 1.12 are suggested to upgrade to the latest DPM version , following the guide [https://twiki.cern.ch/twiki/bin/view/DPM/DpmSetup1100 DPM upgrade] (chapter 1 Upgrade to DPM 1.10.0 "Legacy Flavour" and chapter 2 Upgrade to DPM 1.10.0 "Dome Flavour")
**DOME and the old LCGDM (srm protocol) will coexist
*'''Monitoring''': sites should enable the monitoring of the HTTP/WebDav and/or GridFTP endpoints
**register the storage service endpoint as webdav and/or globus-GRIDFTP service type, with production flag disabled, providing respectively the URL field and the Extension Properties information as explained in the [https://wiki.egi.eu/wiki/HOWTO21#webdav HOWTO21]
**check if the tests are ok
**switch the production flag to "yes"
 
=== List of tickets  ===
 
*[https://ggus.eu/index.php?mode=ticket_search&show_columns_check%5B%5D=TICKET_TYPE&show_columns_check%5B%5D=AFFECTED_VO&show_columns_check%5B%5D=AFFECTED_SITE&show_columns_check%5B%5D=PRIORITY&show_columns_check%5B%5D=RESPONSIBLE_UNIT&show_columns_check%5B%5D=STATUS&show_columns_check%5B%5D=DATE_OF_CHANGE&show_columns_check%5B%5D=SHORT_DESCRIPTION&show_columns_check%5B%5D=SCOPE&ticket_id=&supportunit=&su_hierarchy=0&former_su=&vo=&user=&keyword=DOME+enabling&involvedsupporter=&assignedto=&affectedsite=&specattrib=none&status=&priority=&typeofproblem=all&ticket_category=all&mouarea=&date_type=creation+date&tf_radio=1&timeframe=any&from_date=10+Oct+2019&to_date=11+Oct+2019&untouched_date=&scope=&orderticketsby=REQUEST_ID&orderhow=desc&search_submit=GO%21 DPM upgrade, DOME enabling, and monitoring] (33)
*[https://ggus.eu/index.php?mode=ticket_search&show_columns_check%5B%5D=TICKET_TYPE&show_columns_check%5B%5D=AFFECTED_VO&show_columns_check%5B%5D=AFFECTED_SITE&show_columns_check%5B%5D=PRIORITY&show_columns_check%5B%5D=RESPONSIBLE_UNIT&show_columns_check%5B%5D=STATUS&show_columns_check%5B%5D=DATE_OF_CHANGE&show_columns_check%5B%5D=SHORT_DESCRIPTION&show_columns_check%5B%5D=SCOPE&ticket_id=&supportunit=&su_hierarchy=0&former_su=&vo=&user=&keyword=enabling+DOME&involvedsupporter=&assignedto=&affectedsite=&specattrib=none&status=&priority=&typeofproblem=all&ticket_category=all&mouarea=&date_type=creation+date&tf_radio=1&timeframe=any&from_date=10+Oct+2019&to_date=11+Oct+2019&untouched_date=&scope=&orderticketsby=REQUEST_ID&orderhow=desc&search_submit=GO%21 enabling DOME and proper monitoring] (6)
*tickets opened by WLCG: [https://ggus.eu/index.php?mode=ticket_search&show_columns_check%5B%5D=TICKET_TYPE&show_columns_check%5B%5D=AFFECTED_VO&show_columns_check%5B%5D=AFFECTED_SITE&show_columns_check%5B%5D=PRIORITY&show_columns_check%5B%5D=RESPONSIBLE_UNIT&show_columns_check%5B%5D=STATUS&show_columns_check%5B%5D=DATE_OF_CHANGE&show_columns_check%5B%5D=SHORT_DESCRIPTION&show_columns_check%5B%5D=SCOPE&ticket_id=&supportunit=&su_hierarchy=0&former_su=&vo=&user=&keyword=has+to+be+configured+with+DOME&involvedsupporter=&assignedto=&affectedsite=&specattrib=none&status=&priority=&typeofproblem=all&ticket_category=all&mouarea=&date_type=creation+date&tf_radio=1&timeframe=any&from_date=10+Oct+2019&to_date=11+Oct+2019&untouched_date=&scope=&orderticketsby=REQUEST_ID&orderhow=desc&search_submit=GO%21 13] and [https://ggus.eu/index.php?mode=ticket_search&show_columns_check%5B%5D=TICKET_TYPE&show_columns_check%5B%5D=AFFECTED_VO&show_columns_check%5B%5D=AFFECTED_SITE&show_columns_check%5B%5D=PRIORITY&show_columns_check%5B%5D=RESPONSIBLE_UNIT&show_columns_check%5B%5D=STATUS&show_columns_check%5B%5D=DATE_OF_CHANGE&show_columns_check%5B%5D=SHORT_DESCRIPTION&show_columns_check%5B%5D=SCOPE&ticket_id=&supportunit=&su_hierarchy=0&former_su=&vo=&user=&keyword=Upgrade+and+reconfigure+DPM+at&involvedsupporter=&assignedto=&affectedsite=&specattrib=none&status=&priority=&typeofproblem=all&ticket_category=all&mouarea=&date_type=creation+date&tf_radio=1&timeframe=any&from_date=10+Oct+2019&to_date=11+Oct+2019&untouched_date=&scope=&orderticketsby=REQUEST_ID&orderhow=desc&search_submit=GO%21 7]
 
<br>
 
{| width="1200" border="1"
|-
! scope="col" | Site
! scope="col" | Ticket
! scope="col" | Notes
|-
| AEGIS03-ELEF-LEDA
| https://ggus.eu/index.php?mode=ticket_info&amp;ticket_id=143152
| SE marked as not production due to some issues that need to be fixed
|-
| INDIACMS-TIFR
| https://ggus.eu/index.php?mode=ticket_info&amp;ticket_id=142245
| new dpm headnode installed with legacy mode, [https://goc.egi.eu/portal/index.php?Page_Type=Downtime&id=28824 downtime for migration]
|-
| OBSPM
| https://ggus.eu/index.php?mode=ticket_info&amp;ticket_id=143169
| they asked for quattor documentation...
|-
| TASK
| https://ggus.eu/index.php?mode=ticket_info&amp;ticket_id=143174
| problem with starting xrootd...
|-
| UA_ICYB_ARC
| https://ggus.eu/index.php?mode=ticket_info&amp;ticket_id=143178
|
|-
| WCSS64
| https://ggus.eu/index.php?mode=ticket_info&amp;ticket_id=143182
| they need some time to gain enough knowledge for doing the upgrade....
|-
| GR-07-UOI-HEPLAB
| https://ggus.eu/index.php?mode=ticket_info&amp;ticket_id=143467
| still on slc6, some problems with the upgrade; before the end of the year all the site will be migrated to CentOS 7
|-
| Hephy-Vienna
| https://ggus.eu/index.php?mode=ticket_info&amp;ticket_id=143277
| DPM will be replaced with EOS during Q1 2020
|-
| HK-LCG2
| https://ggus.eu/index.php?mode=ticket_info&amp;ticket_id=143471
| April 2020
|-
| ICM
| https://ggus.eu/index.php?mode=ticket_info&amp;ticket_id=143091
| dpm in the newest version. Now we are setting quota tokens...
|-
| IN2P3-IPNL
| https://ggus.eu/index.php?mode=ticket_info&amp;ticket_id=143082
| migration to EOS, dpm should be dismissed by mid 2020
|-
| IN2P3-IRES
| https://ggus.eu/index.php?mode=ticket_info&amp;ticket_id=143070
| testing the upgrade on the test infrastructure
|-
| NIKHEF-ELPROD
| https://ggus.eu/index.php?mode=ticket_info&amp;ticket_id=143286
| We won't upgrade. we plan to migrate to dCache before the end of 2019.
|-
| PSNC
| https://ggus.eu/index.php?mode=ticket_info&amp;ticket_id=143474
|
|-
| ru-PNPI
| https://ggus.eu/index.php?mode=ticket_info&ticket_id=143281
|
|-
| UKI-SCOTGRID-DURHAM
| https://ggus.eu/index.php?mode=ticket_info&ticket_id=143465
| migration to CentOS7, first....
|-
| UKI-SCOTGRID-ECDF
| https://ggus.eu/index.php?mode=ticket_info&ticket_id=143077
| upgrading to CentOS7 first without DOME
|-
| UKI-SCOTGRID-GLASGOW
| https://ggus.eu/index.php?mode=ticket_info&ticket_id=143076
| plan in progress
|-
| UKI-SOUTHGRID-BRIS-HEP
| https://ggus.eu/index.php?mode=ticket_info&ticket_id=143083
|
|}
 
<br>


== SECMON failures ==
== SECMON failures ==
Line 259: Line 136:


= AOB  =
= AOB  =
* Storage accounting: http://goc-accounting.grid-support.ac.uk/storagetest/storagesitesystems.html
 
** several sites stopped publishing storage accounting records: it needs to investigate on and fix it


== Next meeting  ==
== Next meeting  ==
July 13th, 2020 https://indico.egi.eu/event/4901/
Sept 14th, 2020 https://indico.egi.eu/event/5098/

Latest revision as of 13:24, 14 September 2020

Main EGI.eu operations services Support Documentation Tools Activities Performance Technology Catch-all Services Resource Allocation Security


Documentation menu: Home Manuals Procedures Training Other Contact For: VO managers Administrators


Back to https://wiki.egi.eu/wiki/Operations_Meeting

General information

Middleware

UMD

Preview repository

  • released on 2020-05-08
    • Preview 1.27.0 AppDB info (sl6): ARC 6.5.0 and 6.6.0, CVMFS 2.7.2, dCache 5.2.20, frontier-squid 4.11.2, gfal2 2.17.2, xrootd 4.11.3
    • Preview 2.27.0 AppDB info (CentOS 7): ARC 6.5.0 and 6.6.0, CVMFS 2.7.2, dCache 5.2.20, frontier-squid 4.11.2, gfal2 2.17.2, xrootd 4.11.3

Operations

ARGO/SAM

  • HTCondor-CE probes included in the ARGO_MON_OPERATORS profile on May 13th: https://ggus.eu/index.php?mode=ticket_info&ticket_id=146949
    • 57 endpoints, 17 CRITICAL, success rate is about 70.2%
    • on Sept 1st they will be included in the ARGO_MON_CRITICAL profile (A/R computation)
      • please fix the failures by that date
    • working on the probe for the host certificate validity check: GGUS 147386
  • CREAM-CE metrics in the ARGO_MON_OPERATORS profile on May 27th: eu.egi.CREAMCE-JobSubmit, eu.egi.CREAMCE.WN-Csh, eu.egi.CREAMCE.WN-Softver
    • results: 177 endpoints, 15 WARNING (Timeout occurred (900 sec) ), 53 CRITICAL. Success rate 70% (61.6% including the WARNING)
  • When eu.egi.CREAMCE.WN-Softver is successful:
CREAM JobOutput OK: retrieved outputSandbox: ['std.err', 'std.out']

**** std.err ****


**** std.out ****
egee01 has UMD 3.14.4

When it fails:

CREAM JobOutput ERROR [DONE-OK, exitCode=1 ]: retrieved outputSandbox: ['std.err', 'std.out']

**** std.err ****
 
**** std.out ****
ERROR: unable to find glite, EMI, LCG or UMD WN version on n1037-amd

FedCloud

Feedback from DMSU

Verify configuration records

On a yearly basis, the information registered into GOC-DB need to be verified. NGIs and RCs have been asked to check them. In particular:

  1. NGI managers should review the people registered and the roles assigned to them, and in particular check the following information:
    • E-Mail
    • ROD E-Mail
    • Security E-Mail
NGI Managers should also review the status of the "not certified" RCs, in according to the RC Status Workflow;
  1. RCs administrators should review the people registered and the roles assigned to them, and in particular check the following information:
    • E-Mail
    • telephone numbers
    • CSIRT E-Mail
RC administrators should also review the information related to the registered service endpoints.

The process should be completed by June 22nd.

List of tickets.

  • 30 tickets
  • Not yet solved after 1 month: 16

Monthly Availability/Reliability

IPv6 readiness plans

ARC Middleware 5 end of support, migration to ARC 6

  • Status
Date Number of endpoints in BDII Number of GGUS tickets Issues
2020-06-08 75 42 Some ARC endpoints publish a timestamp instead of a version like 5.X.Y; we can fairly assume they are ARC6 nightly builds, but we're going to close the corresponding tickets after explicit confirmation from the site admin.
2020-07-13 53 29 -

Storage accounting

Many of sites stopped the publication of storage accounting records. Opened 57 tickets to fix that.

SECMON failures

Several CEs are failing the job submission tests, preventing pakiti to check the vulnerabilities fixes on the WNs.

AOB

Next meeting

Sept 14th, 2020 https://indico.egi.eu/event/5098/