Alert.png The wiki is deprecated and due to be decommissioned by the end of September 2022.
The content is being migrated to other supports, new updates will be ignored and lost.
If needed you can get in touch with EGI SDIS team using operations @ egi.eu.

Agenda-2020-06-08

From EGIWiki
Jump to navigation Jump to search
Main EGI.eu operations services Support Documentation Tools Activities Performance Technology Catch-all Services Resource Allocation Security


Documentation menu: Home Manuals Procedures Training Other Contact For: VO managers Administrators


Back to https://wiki.egi.eu/wiki/Operations_Meeting

General information TO UPDATE

Middleware

UMD


CMD

Preview repository

  • released on 2020-05-08
    • Preview 1.27.0 AppDB info (sl6): ARC 6.5.0 and 6.6.0, CVMFS 2.7.2, dCache 5.2.20, frontier-squid 4.11.2, gfal2 2.17.2, xrootd 4.11.3
    • Preview 2.27.0 AppDB info (CentOS 7): ARC 6.5.0 and 6.6.0, CVMFS 2.7.2, dCache 5.2.20, frontier-squid 4.11.2, gfal2 2.17.2, xrootd 4.11.3

Operations

ARGO/SAM

  • new metrics added to ARGO_MON_OPERATORS profile: eu.egi.CREAMCE-JobSubmit, eu.egi.CREAMCE.WN-Csh, eu.egi.CREAMCE.WN-Softver
    • results: endpoints, -21- WARNING (Timeout occurred (900 sec) ), 53 CRITICAL. Success rate -66.67%- (-50.8 %- considering the WARNING)
  • When eu.egi.CREAMCE.WN-Softver is successful:
CREAM JobOutput OK: retrieved outputSandbox: ['std.err', 'std.out']

**** std.err ****
+ versionFilter='ή\._PIPE_ί\._PIPE_έ\._PIPE_ΰ\.'
+ type=unknow
+ mwver=error
+ '[' -f /etc/umd-release ']'
+ type=UMD
++ cat /etc/umd-release
++ awk '{print $3}'
+ mwver=4.1.3
+ set +x


**** std.out ****
atlaswn184 has UMD 4.1.3

When it fails:

CREAM JobOutput ERROR [DONE-OK, exitCode=1 ]: retrieved outputSandbox: ['std.err', 'std.out']

**** std.err ****
+ versionFilter='ή\._PIPE_ί\._PIPE_έ\._PIPE_ΰ\.'
+ type=unknow
+ mwver=error
+ '[' -f /etc/umd-release ']'
+ '[' -f glite-version ']'
+ '[' -f /etc/emi-version ']'
+ '[' -f lcg-version ']'
++ hostname -s
+ echo 'ERROR: [glite_PIPE_lcg_PIPE_emi]-version was not found in n172'
+ exit 1


**** std.out ****
ERROR: [glite_PIPE_lcg_PIPE_emi]-version was not found in n172

FedCloud

Feedback from DMSU

Monthly Availability/Reliability

IPv6 readiness plans

ARC Middleware 5 end of support, migration to ARC 6

  • No new feature development is planned or going on for ARC5 and no bug-fixing development will happen on ARC5 code base in the future except for security issues.
  • Security fixes for ARC5 will be provided till end of June 2020.
  • Production Sites already running ARC 5 will be able to get deployment and configuration troubleshooting help via GGUS till end June 2021. This we call "operational site support".
  • ARC5 is available in EPEL7 and will stay there. EPEL8 will only contain ARC 6.


$ ldapsearch -x -LLL -H ldap://egee-bdii.cnaf.infn.it:2170 -b "GLUE2GroupID=grid,o=glue" '(&(objectClass=GLUE2Endpoint)(&(GLUE2EndpointImplementationName=nordugrid-arc)(GLUE2EndpointTechnology=gridftp)))' GLUE2EndpointImplementationVersion GLUE2EndpointID | grep GLUE2EndpointImplementationVersion | sort | uniq -c
      1 GLUE2EndpointImplementationVersion: 20190403020701
      1 GLUE2EndpointImplementationVersion: 20200217020714
      1 GLUE2EndpointImplementationVersion: 20200226020744
      1 GLUE2EndpointImplementationVersion: 20200228020731
      1 GLUE2EndpointImplementationVersion: 20200424020718
      1 GLUE2EndpointImplementationVersion: 20200505020715
      1 GLUE2EndpointImplementationVersion: 4.1.0
      1 GLUE2EndpointImplementationVersion: 5.0.2
      1 GLUE2EndpointImplementationVersion: 5.1.3
      1 GLUE2EndpointImplementationVersion: 5.3.0
      5 GLUE2EndpointImplementationVersion: 5.3.1
      6 GLUE2EndpointImplementationVersion: 5.4.1
     11 GLUE2EndpointImplementationVersion: 5.4.2
      7 GLUE2EndpointImplementationVersion: 5.4.3
     47 GLUE2EndpointImplementationVersion: 5.4.4
      2 GLUE2EndpointImplementationVersion: 6.2.0
      5 GLUE2EndpointImplementationVersion: 6.4.1
     14 GLUE2EndpointImplementationVersion: 6.5.0
      5 GLUE2EndpointImplementationVersion: 6.6.0

LCGDM end of support and migration to / enabling of DOME

  • Deployment statistics (May 8th):
$ ldapsearch -x -LLL -H ldap://egee-bdii.cnaf.infn.it:2170 -b "GLUE2GroupID=grid,o=glue" '(&(objectClass=GLUE2Manager)(GLUE2ManagerProductName=DPM))' GLUE2ManagerProductVersion GLUE2ManagerID | grep GLUE2ManagerProductVersion | sort | uniq -c
     1 GLUE2ManagerProductVersion: 1.10.0
    66 GLUE2ManagerProductVersion: 1.13.0
     2 GLUE2ManagerProductVersion: 1.13.1
    12 GLUE2ManagerProductVersion: 1.13.2
     3 GLUE2ManagerProductVersion: 1.8.10
     1 GLUE2ManagerProductVersion: 1.8.8
     1 GLUE2ManagerProductVersion: 1.8.9
     4 GLUE2ManagerProductVersion: 1.9.0


Liasing with WLCG to follow-up the upgrade. Opened GGUS tickets asking the following:

  • all the sites with older DPM versions than 1.12 are suggested to upgrade to the latest DPM version , following the guide DPM upgrade (chapter 1 Upgrade to DPM 1.10.0 "Legacy Flavour" and chapter 2 Upgrade to DPM 1.10.0 "Dome Flavour")
    • DOME and the old LCGDM (srm protocol) will coexist
  • Monitoring: sites should enable the monitoring of the HTTP/WebDav and/or GridFTP endpoints
    • register the storage service endpoint as webdav and/or globus-GRIDFTP service type, with production flag disabled, providing respectively the URL field and the Extension Properties information as explained in the HOWTO21
    • check if the tests are ok
    • switch the production flag to "yes"

List of tickets


Site Ticket Notes
AEGIS03-ELEF-LEDA https://ggus.eu/index.php?mode=ticket_info&ticket_id=143152 SE marked as not production due to some issues that need to be fixed
INDIACMS-TIFR https://ggus.eu/index.php?mode=ticket_info&ticket_id=142245 new dpm headnode installed with legacy mode, downtime for migration
OBSPM https://ggus.eu/index.php?mode=ticket_info&ticket_id=143169 they asked for quattor documentation...
TASK https://ggus.eu/index.php?mode=ticket_info&ticket_id=143174 problem with starting xrootd...
UA_ICYB_ARC https://ggus.eu/index.php?mode=ticket_info&ticket_id=143178
WCSS64 https://ggus.eu/index.php?mode=ticket_info&ticket_id=143182 they need some time to gain enough knowledge for doing the upgrade....
GR-07-UOI-HEPLAB https://ggus.eu/index.php?mode=ticket_info&ticket_id=143467 still on slc6, some problems with the upgrade; before the end of the year all the site will be migrated to CentOS 7
Hephy-Vienna https://ggus.eu/index.php?mode=ticket_info&ticket_id=143277 DPM will be replaced with EOS during Q1 2020
HK-LCG2 https://ggus.eu/index.php?mode=ticket_info&ticket_id=143471 April 2020
ICM https://ggus.eu/index.php?mode=ticket_info&ticket_id=143091 dpm in the newest version. Now we are setting quota tokens...
IN2P3-IPNL https://ggus.eu/index.php?mode=ticket_info&ticket_id=143082 migration to EOS, dpm should be dismissed by mid 2020
IN2P3-IRES https://ggus.eu/index.php?mode=ticket_info&ticket_id=143070 testing the upgrade on the test infrastructure
NIKHEF-ELPROD https://ggus.eu/index.php?mode=ticket_info&ticket_id=143286 We won't upgrade. we plan to migrate to dCache before the end of 2019.
PSNC https://ggus.eu/index.php?mode=ticket_info&ticket_id=143474
ru-PNPI https://ggus.eu/index.php?mode=ticket_info&ticket_id=143281
UKI-SCOTGRID-DURHAM https://ggus.eu/index.php?mode=ticket_info&ticket_id=143465 migration to CentOS7, first....
UKI-SCOTGRID-ECDF https://ggus.eu/index.php?mode=ticket_info&ticket_id=143077 upgrading to CentOS7 first without DOME
UKI-SCOTGRID-GLASGOW https://ggus.eu/index.php?mode=ticket_info&ticket_id=143076 plan in progress
UKI-SOUTHGRID-BRIS-HEP https://ggus.eu/index.php?mode=ticket_info&ticket_id=143083


SECMON failures

Several CEs are failing the job submission tests, preventing pakiti to check the vulnerabilities fixes on the WNs.

AOB

Next meeting

June 8th, 2020 https://indico.egi.eu/event/4900/