Latest revision as of 14:25, 8 June 2020

Main

EGI.eu operations services

Support

Documentation

Tools

Activities

Performance

Technology

Catch-all Services

Resource Allocation

Security

Documentation menu:

Home •

Manuals •

Procedures •

Training •

Other •

Contact ►

For:

VO managers •

Administrators

Back to https://wiki.egi.eu/wiki/Operations_Meeting

General information

Middleware

UMD

plans on CentOS8 STARTED
- https://wiki.egi.eu/wiki/Next_middleware_release

Preview repository

released on 2020-05-08
- Preview 1.27.0 AppDB info (sl6): ARC 6.5.0 and 6.6.0, CVMFS 2.7.2, dCache 5.2.20, frontier-squid 4.11.2, gfal2 2.17.2, xrootd 4.11.3
- Preview 2.27.0 AppDB info (CentOS 7): ARC 6.5.0 and 6.6.0, CVMFS 2.7.2, dCache 5.2.20, frontier-squid 4.11.2, gfal2 2.17.2, xrootd 4.11.3

Operations

ARGO/SAM

new metrics added to ARGO_MON_OPERATORS profile on May 27th: eu.egi.CREAMCE-JobSubmit, eu.egi.CREAMCE.WN-Csh, eu.egi.CREAMCE.WN-Softver
- results: 177 endpoints, 15 WARNING (Timeout occurred (900 sec) ), 53 CRITICAL. Success rate 70% (61.6% including the WARNING)
When eu.egi.CREAMCE.WN-Softver is successful:

CREAM JobOutput OK: retrieved outputSandbox: ['std.err', 'std.out']

**** std.err ****


**** std.out ****
egee01 has UMD 3.14.4

When it fails:

CREAM JobOutput ERROR [DONE-OK, exitCode=1 ]: retrieved outputSandbox: ['std.err', 'std.out']

**** std.err ****
 
**** std.out ****
ERROR: unable to find glite, EMI, LCG or UMD WN version on n1037-amd

HTCondor-CE probes included in the ARGO_MON_OPERATORS profile on May 13th: https://ggus.eu/index.php?mode=ticket_info&ticket_id=146949
- 53 endpoints, 8 CRITICAL, success rate is about 84.9%

FedCloud

Feedback from DMSU

Verify configuration records

On a yearly basis, the information registered into GOC-DB need to be verified. NGIs and RCs have been asked to check them. In particular:

NGI managers should review the people registered and the roles assigned to them, and in particular check the following information:
- E-Mail
- ROD E-Mail
- Security E-Mail

NGI Managers should also review the status of the "not certified" RCs, in according to the RC Status Workflow;

RCs administrators should review the people registered and the roles assigned to them, and in particular check the following information:
- E-Mail
- telephone numbers
- CSIRT E-Mail

RC administrators should also review the information related to the registered service endpoints.

The process should be completed by June 22nd.

List of tickets.

Monthly Availability/Reliability

Under-performed sites in the past A/R reports with issues not yet fixed:
- AfricaArabia: https://ggus.eu/index.php?mode=ticket_info&ticket_id=146877
  - ZA-WITS-CORE: SE hardware problem, machine sent to the vendor; CREAM-CE failures due to a known issue with the classads library (GGUS 146979)
- AsiaPacific: https://ggus.eu/index.php?mode=ticket_info&ticket_id=142591
  - INDIACMS-TIFR: SRM service not published in the BDII (142245), DPM 1.9.0 version (145676); installing a new DPM headnode
- NGI_AEGIS: https://ggus.eu/index.php?mode=ticket_info&ticket_id=146867
  - AEGIS03-ELEF-LEDA: SRM problems, endpoint put out of production until the issues are fixed; CE certificate expired in May.
- NGI_CH: https://ggus.eu/index.php?mode=ticket_info&ticket_id=146870
  - CSCS-LCG2: intermittent igtf failures
  - UNIBE-LHEP: issues with DPM upgrade and some network problems; statistics are improving
- NGI_CH: https://ggus.eu/index.php?mode=ticket_info&ticket_id=145818
  - UNIGE-DPNC: new ARC-CE put in production at the end of April, new failures due to an expired certificate
- NGI_DE: https://ggus.eu/index.php?mode=ticket_info&ticket_id=146871
  - GoeGRID: CREAM-CE intermittent failures not affecting ATLAS; failures with ARC-CE
- NGI_IT: https://ggus.eu/index.php?mode=ticket_info&ticket_id=146868
  - CIRMMP: SRM unreachable due to a firewall, statistics are now improving
  - INFN-LECCE: CREAMCE and SRM failures, recovering...
- NGI_PL: https://ggus.eu/index.php?mode=ticket_info&ticket_id=140557
  - TASK: QCG problems was fixed; some CREAM-CE failures; the SRM tests are still failing and the DPM upgrade isn't completed yet
- NGI_UK:
  - UKI-NORTHGRID-SHEF-HEP: https://ggus.eu/index.php?mode=ticket_info&ticket_id=146455 needing to re-install the ARC-CE
  - UKI-SOUTHGRID-SUSX: https://ggus.eu/index.php?mode=ticket_info&ticket_id=144720 Migration from CREAM to ARC, WN migration to CentOS7; SRM to be decommissioned; ARC-CE was failing the IGTF test
- NGI_UK: https://ggus.eu/index.php?mode=ticket_info&ticket_id=146875
  - UKI-LT2-QMUL: intermittent webdav failures, plans to upgrade STORM and LUSTRE in the coming months; suggested to mark the webdav endpoint as "not production"
- ROC_CANADA: https://ggus.eu/index.php?mode=ticket_info&ticket_id=146452
  - CA-SFU-T2: SLURM problems caused failures to site-BDII freshness check due to some old jobs not properly cancelled; recovered
  - CA-WATERLOO-T2: SRM failures not involving production VOs, fixed; some unscheduled downtime affected the the A/R figures; some services still to restore.
- NGI_UA: https://ggus.eu/index.php?mode=ticket_info&ticket_id=146454
  - UA-KNU: GlueSEUniqueID wasn't published due to missing DNS back-resolving record for IPv6 address of the SE. Deployed a new ARC-CE version 6, igtf probe returns UNKNOWN.
Under-performed sites after 3 consecutive months, under-performed NGIs, QoS violations: (May 2020):
- NGI_DE: https://ggus.eu/index.php?mode=ticket_info&ticket_id=147313
  - mainz: some problems in March and April, that could not be fixed easily; in May, the HPC infrastructure was attacked and the whole computer center was shut down.
  - wuppertalprod: SRM failures to to a BDII issue, fixed
- NGI_NDGF: https://ggus.eu/index.php?mode=ticket_info&ticket_id=147310
  - FI_HIP_T2: configuration problem that prevents some of the OPS-jobs from running
- NGI_PL: https://ggus.eu/index.php?mode=ticket_info&ticket_id=147311
  - NCBJ-CIS
  - PSNC
  - WCSS64
- NGI_UA: https://ggus.eu/index.php?mode=ticket_info&ticket_id=147312
  - UA_ICYB_ARC: cluster storage failure, improving...
sites suspended:
- GRID-UNAM (under-performing, DPM upgrade not completed)

IPv6 readiness plans

please provide updates to the IPv6 assessment (ongoing) https://wiki.egi.eu/w/index.php?title=IPV6_Assessment
if any relevant, information will be summarised at OMB

ARC Middleware 5 end of support, migration to ARC 6

EGI Operations Broadcast
PROC16 Decommission of unsupported software
deadline: end of July
Catalin is in contact with ARC team to get a webinar on ARC administration, scheduled (to be confirmed) for July 6th please contact operations@ for information

Status

Date	Number of endpoints in BDII	Number of GGUS tickets	Issues
2020-06-08	75	42	Some ARC endpoints publish a timestamp instead of a version like 5.X.Y; we can fairly assume they are ARC6 nightly builds, but we're going to close the corresponding tickets after explicit confirmation from the site admin.

LCGDM end of support and migration to / enabling of DOME

The DPM team has agreed to extend support for security updates for the DPM legacy functionality until 30 September 2019. However, affected service providers should still plan to disable legacy mode well before this date.
- more details: https://wiki.egi.eu/wiki/DPM_End_of_legacy-mode_support
since DPM 1.10.3 release, it is possible enabling the non-legacy mode DOME (Disk operations Management Engine, see documentation)
- latest DPM/dmlite release 1.13.2 has been also released in UMD 4.10.0
EGI Operations sent a broadcast to site-admins(first bunch and second bunch) and VO managers with a survey to fill in:
- site-admins: to know the upgrade plans: https://www.surveymonkey.com/r/2PBZSNB
- VO Managers: to know any barriers in moving away from using SRM protocol https://www.surveymonkey.com/r/2ZMZJVG

Deployment statistics (Jun 5th):

$ ldapsearch -x -LLL -H ldap://egee-bdii.cnaf.infn.it:2170 -b "GLUE2GroupID=grid,o=glue" '(&(objectClass=GLUE2Manager)(GLUE2ManagerProductName=DPM))' GLUE2ManagerProductVersion GLUE2ManagerID | grep GLUE2ManagerProductVersion | sort | uniq -c
     1 GLUE2ManagerProductVersion: 1.10.0
    65 GLUE2ManagerProductVersion: 1.13.0
     2 GLUE2ManagerProductVersion: 1.13.1
    11 GLUE2ManagerProductVersion: 1.13.2
     3 GLUE2ManagerProductVersion: 1.8.10
     1 GLUE2ManagerProductVersion: 1.8.9
     4 GLUE2ManagerProductVersion: 1.9.0

Liasing with WLCG to follow-up the upgrade. Opened GGUS tickets asking the following:

all the sites with older DPM versions than 1.12 are suggested to upgrade to the latest DPM version , following the guide DPM upgrade (chapter 1 Upgrade to DPM 1.10.0 "Legacy Flavour" and chapter 2 Upgrade to DPM 1.10.0 "Dome Flavour")
- DOME and the old LCGDM (srm protocol) will coexist
Monitoring: sites should enable the monitoring of the HTTP/WebDav and/or GridFTP endpoints
- register the storage service endpoint as webdav and/or globus-GRIDFTP service type, with production flag disabled, providing respectively the URL field and the Extension Properties information as explained in the HOWTO21
- check if the tests are ok
- switch the production flag to "yes"

List of tickets

DPM upgrade, DOME enabling, and monitoring (33)
enabling DOME and proper monitoring (6)
tickets opened by WLCG: 13 and 7

Site	Ticket	Notes
AEGIS03-ELEF-LEDA	https://ggus.eu/index.php?mode=ticket_info&ticket_id=143152	SE marked as not production due to some issues that need to be fixed
INDIACMS-TIFR	https://ggus.eu/index.php?mode=ticket_info&ticket_id=142245	new dpm headnode installed with legacy mode, downtime for migration
OBSPM	https://ggus.eu/index.php?mode=ticket_info&ticket_id=143169	they asked for quattor documentation...
TASK	https://ggus.eu/index.php?mode=ticket_info&ticket_id=143174	problem with starting xrootd...
UA_ICYB_ARC	https://ggus.eu/index.php?mode=ticket_info&ticket_id=143178
WCSS64	https://ggus.eu/index.php?mode=ticket_info&ticket_id=143182	they need some time to gain enough knowledge for doing the upgrade....
GR-07-UOI-HEPLAB	https://ggus.eu/index.php?mode=ticket_info&ticket_id=143467	still on slc6, some problems with the upgrade; before the end of the year all the site will be migrated to CentOS 7
Hephy-Vienna	https://ggus.eu/index.php?mode=ticket_info&ticket_id=143277	DPM will be replaced with EOS during Q1 2020
HK-LCG2	https://ggus.eu/index.php?mode=ticket_info&ticket_id=143471	April 2020
ICM	https://ggus.eu/index.php?mode=ticket_info&ticket_id=143091	dpm in the newest version. Now we are setting quota tokens...
IN2P3-IPNL	https://ggus.eu/index.php?mode=ticket_info&ticket_id=143082	migration to EOS, dpm should be dismissed by mid 2020
IN2P3-IRES	https://ggus.eu/index.php?mode=ticket_info&ticket_id=143070	testing the upgrade on the test infrastructure
NIKHEF-ELPROD	https://ggus.eu/index.php?mode=ticket_info&ticket_id=143286	We won't upgrade. we plan to migrate to dCache before the end of 2019.
PSNC	https://ggus.eu/index.php?mode=ticket_info&ticket_id=143474
ru-PNPI	https://ggus.eu/index.php?mode=ticket_info&ticket_id=143281
UKI-SCOTGRID-DURHAM	https://ggus.eu/index.php?mode=ticket_info&ticket_id=143465	migration to CentOS7, first....
UKI-SCOTGRID-ECDF	https://ggus.eu/index.php?mode=ticket_info&ticket_id=143077	upgrading to CentOS7 first without DOME
UKI-SCOTGRID-GLASGOW	https://ggus.eu/index.php?mode=ticket_info&ticket_id=143076	plan in progress
UKI-SOUTHGRID-BRIS-HEP	https://ggus.eu/index.php?mode=ticket_info&ticket_id=143083

SECMON failures

Several CEs are failing the job submission tests, preventing pakiti to check the vulnerabilities fixes on the WNs.

AOB

Storage accounting: http://goc-accounting.grid-support.ac.uk/storagetest/storagesitesystems.html
- several sites stopped publishing storage accounting records: it needs to investigate on and fix it

Next meeting

July 13th, 2020 https://indico.egi.eu/event/4901/

@@ Line 3: / Line 3: @@
 Back to https://wiki.egi.eu/wiki/Operations_Meeting
-= General information TO UPDATE =
+= General information =
 = Middleware =
@@ Line 10: / Line 10: @@
 * plans on CentOS8 STARTED
 ** https://wiki.egi.eu/wiki/Next_middleware_release
-== CMD ==
-* CMD-OS released http://repository.egi.eu/2020/02/18/release-cmd-os-1-3-0/
 == Preview repository  ==
@@ Line 23: / Line 19: @@
 == ARGO/SAM  ==
-*new metrics added to ARGO_MON_OPERATORS profile: eu.egi.CREAMCE-JobSubmit, eu.egi.CREAMCE.WN-Csh, eu.egi.CREAMCE.WN-Softver
+*new metrics added to ARGO_MON_OPERATORS profile on [https://ggus.eu/index.php?mode=ticket_info&ticket_id=147169 May 27th]: eu.egi.CREAMCE-JobSubmit, eu.egi.CREAMCE.WN-Csh, eu.egi.CREAMCE.WN-Softver
-** [https://argo-mon.egi.eu/nagios/cgi-bin/status.cgi?servicegroup=SERVICE_CREAM-CE&style=detail results]:  endpoints, -21- WARNING (Timeout occurred (900 sec) ), 53 CRITICAL. Success rate -66.67%- (-50.8 %- considering the WARNING)
+** [https://argo-mon.egi.eu/nagios/cgi-bin/status.cgi?servicegroup=SERVICE_CREAM-CE&style=detail results]:  177 endpoints, 15 WARNING (Timeout occurred (900 sec) ), 53 CRITICAL. Success rate 70% (61.6% including the WARNING)
 *When eu.egi.CREAMCE.WN-Softver is successful:
   CREAM JobOutput OK: retrieved outputSandbox: ['std.err', 'std.out']
   **** std.err ****
- + versionFilter='ή\._PIPE_ί\._PIPE_έ\._PIPE_ΰ\.'
- + type=unknow
- + mwver=error
- + '[' -f /etc/umd-release ']'
- + type=UMD
- ++ cat /etc/umd-release
- ++ awk '{print $3}'
- + mwver=4.1.3
- + set +x
   **** std.out ****
-  atlaswn184 has UMD 4.1.3
+  egee01 has UMD 3.14.4
 When it fails:
   CREAM JobOutput ERROR [DONE-OK, exitCode=1 ]: retrieved outputSandbox: ['std.err', 'std.out']
   **** std.err ****
- + versionFilter='ή\._PIPE_ί\._PIPE_έ\._PIPE_ΰ\.'
- + type=unknow
- + mwver=error
- + '[' -f /etc/umd-release ']'
- + '[' -f glite-version ']'
- + '[' -f /etc/emi-version ']'
- + '[' -f lcg-version ']'
- ++ hostname -s
- + echo 'ERROR: [glite_PIPE_lcg_PIPE_emi]-version was not found in n172'
- + exit 1
   **** std.out ****
-  ERROR: [glite_PIPE_lcg_PIPE_emi]-version was not found in n172
+  ERROR: unable to find glite, EMI, LCG or UMD WN version on n1037-amd
-* HTCondor-CE probe [https://argo-mon-fedcloud.cro-ngi.hr/nagios/cgi-bin/status.cgi?servicegroup=SERVICE_org.opensciencegrid.htcondorce&style=detail deployed on the test instance]: https://ggus.eu/index.php?mode=ticket_info&ticket_id=141177
-** 96 endpoints, 3 WARNING, 14 CRITICAL, success rate is about 82.3%
+* [https://argo-mon-fedcloud.cro-ngi.hr/nagios/cgi-bin/status.cgi?servicegroup=SERVICE_org.opensciencegrid.htcondorce&style=detail HTCondor-CE probes] included in the ARGO_MON_OPERATORS profile on May 13th: https://ggus.eu/index.php?mode=ticket_info&ticket_id=146949
-** to include in the ARGO_MON_OPERATORS profile
+** 53 endpoints, 8 CRITICAL, success rate is about 84.9%
 == FedCloud  ==
@@ Line 71: / Line 47: @@
 == Feedback from DMSU  ==
+== Verify configuration records ==
+On a yearly basis, the information registered into GOC-DB need to be verified.
+NGIs and RCs have been asked to check them. In particular:
+# '''NGI managers should review the people registered and the roles assigned to them, and in particular check the following information:'''
+#* E-Mail
+#* ROD E-Mail
+#* Security E-Mail
+:NGI Managers should also review the status of the "not certified" RCs, in according to the [https://wiki.egi.eu/wiki/PROC09#Resource_Center_status_Workflow RC Status Workflow];
+# '''RCs administrators should review the people registered and the roles assigned to them, and in particular check the following information:'''
+#* E-Mail
+#* telephone numbers
+#* CSIRT E-Mail
+: RC administrators should also review the information related to the registered service endpoints.
+'''The process should be completed by June 22nd.'''
+[https://wiki.egi.eu/wiki/Verify_Configuration_Records#2020-05 List of tickets].
 == Monthly Availability/Reliability ==
@@ Line 108: / Line 103: @@
 ***wuppertalprod: SRM failures to to a BDII issue, fixed
 **NGI_NDGF: https://ggus.eu/index.php?mode=ticket_info&ticket_id=147310
-***FI_HIP_T2
+***FI_HIP_T2: configuration problem that prevents some of the OPS-jobs from running
 **NGI_PL: https://ggus.eu/index.php?mode=ticket_info&ticket_id=147311
 ***NCBJ-CIS
@@ Line 124: / Line 119: @@
 == ARC Middleware 5 end of support, migration to ARC 6 ==
-* No new feature development is planned or going on for ARC5 and no bug-fixing development will happen on ARC5 code base in the future except for security issues.
+* [https://operations-portal.egi.eu/broadcast/archive/2668 EGI Operations Broadcast]
-* Security fixes for ARC5 will be provided till end of June 2020.
+* [https://wiki.egi.eu/wiki/PROC16_Decommissioning_of_unsupported_software PROC16 Decommission of unsupported software]
-* Production Sites already running ARC 5 will be able to get deployment and configuration troubleshooting help via GGUS till end June 2021. This we call "operational site support".
+* deadline: '''end of July'''
-* ARC5 is available in EPEL7 and will stay there. EPEL8 will only contain ARC 6.
+* Catalin is in contact with ARC team to get a webinar on ARC administration, scheduled (to be confirmed) for July 6th please contact operations@ for information
+* Status
-* [https://wiki.egi.eu/wiki/PROC16_Decommissioning_of_unsupported_software PROC16 Decommission of unsupported software]
+{| class="wikitable sortable"
-* Number of ARC-CE endpoints [https://goc.egi.eu/portal/index.php?Page_Type=Services&serviceType=ARC-CE&selectItemserviceType=ARC-CE&ngi=&searchTerm=&production=TRUE&monitored=TRUE&certStatus=Certified&scopeMatch=all&servKeyNames=&servKeyValue= registered on GOCDB] (monitored and production): 138
+|-
-* From the BDII:
+! Date !! Number of endpoints in BDII !! Number of GGUS tickets !! Issues
- $ ldapsearch -x -LLL -H ldap://egee-bdii.cnaf.infn.it:2170 -b "GLUE2GroupID=grid,o=glue" '(&(objectClass=GLUE2Endpoint)(&(GLUE2EndpointImplementationName=nordugrid-arc)(GLUE2EndpointTechnology=gridftp)))' GLUE2EndpointImplementationVersion GLUE2EndpointID | grep GLUE2EndpointImplementationVersion | sort | uniq -c
+|-
-GLUE2EndpointImplementationVersion: 20190403020701
+| 2020-06-08 || 75 || 42 || Some ARC endpoints publish a timestamp instead of a version like 5.X.Y; we can fairly assume they are ARC6 nightly builds, but we're going to close the corresponding tickets after explicit confirmation from the site admin.
-GLUE2EndpointImplementationVersion: 20200217020714
+|}
-GLUE2EndpointImplementationVersion: 20200226020744
-GLUE2EndpointImplementationVersion: 20200228020731
-GLUE2EndpointImplementationVersion: 20200424020718
-GLUE2EndpointImplementationVersion: 20200505020715
-GLUE2EndpointImplementationVersion: 4.1.0
-GLUE2EndpointImplementationVersion: 5.0.2
-GLUE2EndpointImplementationVersion: 5.1.3
-GLUE2EndpointImplementationVersion: 5.3.0
-GLUE2EndpointImplementationVersion: 5.3.1
-GLUE2EndpointImplementationVersion: 5.4.1
-GLUE2EndpointImplementationVersion: 5.4.2
-GLUE2EndpointImplementationVersion: 5.4.3
-GLUE2EndpointImplementationVersion: 5.4.4
-GLUE2EndpointImplementationVersion: 6.2.0
-GLUE2EndpointImplementationVersion: 6.4.1
-GLUE2EndpointImplementationVersion: 6.5.0
-GLUE2EndpointImplementationVersion: 6.6.0
 == LCGDM end of support and migration to / enabling of DOME ==
@@ Line 164: / Line 142: @@
 **VO Managers: to know any barriers in moving away from using SRM protocol https://www.surveymonkey.com/r/2ZMZJVG
-* '''Deployment statistics (May 8th):'''
+* '''Deployment statistics (Jun 5th):'''
   $ ldapsearch -x -LLL -H ldap://egee-bdii.cnaf.infn.it:2170 -b "GLUE2GroupID=grid,o=glue" '(&(objectClass=GLUE2Manager)(GLUE2ManagerProductName=DPM))' GLUE2ManagerProductVersion GLUE2ManagerID | grep GLUE2ManagerProductVersion | sort | uniq -c
 GLUE2ManagerProductVersion: 1.10.0
 GLUE2ManagerProductVersion: 1.13.0
 GLUE2ManagerProductVersion: 1.13.1
 GLUE2ManagerProductVersion: 1.13.2
 GLUE2ManagerProductVersion: 1.8.10
-GLUE2ManagerProductVersion: 1.8.8
 GLUE2ManagerProductVersion: 1.8.9
 GLUE2ManagerProductVersion: 1.9.0
@@ Line 288: / Line 265: @@
 == Next meeting  ==
-June 8th, 2020 https://indico.egi.eu/event/4900/
+July 13th, 2020 https://indico.egi.eu/event/4901/

Difference between revisions of "Agenda-2020-06-08"

Latest revision as of 14:25, 8 June 2020

Contents

General information

Middleware

UMD

Preview repository

Operations

ARGO/SAM

FedCloud

Feedback from DMSU

Verify configuration records

Monthly Availability/Reliability

IPv6 readiness plans

ARC Middleware 5 end of support, migration to ARC 6

LCGDM end of support and migration to / enabling of DOME

List of tickets

SECMON failures

AOB

Next meeting

Navigation menu

Difference between revisions of "Agenda-2020-06-08"

Latest revision as of 14:25, 8 June 2020

General information

Middleware

UMD

Preview repository

Operations

ARGO/SAM

FedCloud

Feedback from DMSU

Verify configuration records

Monthly Availability/Reliability

IPv6 readiness plans

ARC Middleware 5 end of support, migration to ARC 6

LCGDM end of support and migration to / enabling of DOME

List of tickets

SECMON failures

AOB

Next meeting

Navigation menu

Search