Alert.png The wiki is deprecated and due to be decommissioned by the end of September 2022.
The content is being migrated to other supports, new updates will be ignored and lost.
If needed you can get in touch with EGI SDIS team using operations @ egi.eu.

Difference between revisions of "Agenda-2020-02-10"

From EGIWiki
Jump to navigation Jump to search
 
(27 intermediate revisions by 3 users not shown)
Line 10: Line 10:
Next scheduled release is February's UMD 4.10.0: https://wiki.egi.eu/wiki/UMD_Release_Schedule
Next scheduled release is February's UMD 4.10.0: https://wiki.egi.eu/wiki/UMD_Release_Schedule


* integration of HTCondor-CE still in progress
* plans on CentOS8 are starting!
* HTCondor-CE released in UMD 4.9.1 http://repository.egi.eu/2019/12/04/release-umd-4-9-1/ pointing to the original documentation (the documentation is OK)
* integration of HTCondor-CE still finalizing
* still needed to contributed to the WN package
** HTCondor-CE released in UMD 4.9.1 http://repository.egi.eu/2019/12/04/release-umd-4-9-1/ pointing to the original documentation (the documentation is OK)
* there is no particular automatic way provided to EGI sites to configure HTCondor-CE
** still needed to contribute to the WN package
* UMD team offers automatic deployments for several well-known products and is willing to provide such kind of recipes; usually these recipes are a result of a * joint effort of the provider together with the UMD team
* '''Are NGIs willing to investigate among their site admins about condor/HTCondor-CE skills and contribute with available recipes or help?''' -> no feedback at last EGI Ops meeting
* '''Are NGIs willing to investigate among their site admins about condor/HTCondor-CE skills and contribute with available recipes or help?'''


== CMD ==
== CMD ==
Still needs to be restarted
* caso 1.8.0 to be included
 
* product updates delayed
* planning the "next maor release" based on new technologies
** need: Dockerfiles, define tests.
** automating the SR (possible?)
** manual final verification is always needed
** kubernetes?


== Preview repository  ==
== Preview repository  ==
*last update released in November: https://wiki.egi.eu/wiki/Preview_Repository
*released on 2020-01-21:
*collecting information for the next update
** '''[[Preview 1.26.0]]''' [https://appdb.egi.eu/store/software/preview.repository/releases/1.0/1.26.0/ AppDB info] (sl6):  ARC 6.4.1, CVMFS 2.7.0, dCache 5.2.13, gfal2 2.17.1, STORM 1.11.17, xrootd 4.11.1
** '''[[Preview 2.26.0]]''' [https://appdb.egi.eu/store/software/preview.repository/releases/2.0/2.26.0/ AppDB info] (CentOS 7):  ARC 6.4.1, CVMFS 2.7.0, dCache 5.2.13, gfal2 2.17.1, STORM 1.11.17, xrootd 4.11.1


= Operations  =
= Operations  =
Line 39: Line 32:
== Feedback from DMSU  ==
== Feedback from DMSU  ==


== Monthly Availability/Reliability (TO UPDATE) ==
== Monthly Availability/Reliability ==


*Under-performed sites in the past A/R reports with issues not yet fixed:
*Under-performed sites in the past A/R reports with issues not yet fixed:
**AfricaArabia: https://ggus.eu/index.php?mode=ticket_info&ticket_id=144717
***MA-04-CNRST-ATLAS: the CREAM-CE problems have been fixed; the DPM server has been upgraded to the latest version, the protocol SRM has been disabled; the webdav and gridftp tests are failing and when the problems are solved, the corresponding service endpoints on GOCDB can be marked as "production"
**AsiaPacific: https://ggus.eu/index.php?mode=ticket_info&ticket_id=142591
**AsiaPacific: https://ggus.eu/index.php?mode=ticket_info&ticket_id=142591
***INDIACMS-TIFR: SRM service not published in the BDII, DPM 1.9.0 version; the new SE will be put in production in February
***INDIACMS-TIFR: SRM service not published in the BDII, DPM 1.9.0 version; the new SE will be put in production in February
***TW-NTU-HEP: some failures due to DPM upgrade, then it recovered
***TW-NTU-HEP: some failures due to DPM upgrade, then it recovered
**NGI_CHINA: https://ggus.eu/index.php?mode=ticket_info&ticket_id=144343
**NGI_CHINA: https://ggus.eu/index.php?mode=ticket_info&ticket_id=144343
***HK-LCG2 failures due to DPM upgrade (not yet completed)
***HK-LCG2 failures due to DPM upgrade which was then completed; initials failures with SRM and webdav have been solved
**NGI_IBERGRID: https://ggus.eu/index.php?mode=ticket_info&ticket_id=144344
**NGI_IBERGRID: https://ggus.eu/index.php?mode=ticket_info&ticket_id=144344
***NCG-INGRID-PT: SRM failures solved in November; Openstack failures occurring again
***NCG-INGRID-PT: SRM failures solved in November; Openstack failures occurred again due to migration to new cloud infrastructure; statistics are improving
**NGI_IT: https://ggus.eu/index.php?mode=ticket_info&ticket_id=144715
***INFN-CATANIA-STACK: failures affecting only the ops VO tenant, the other VOs are not affected
**NGI_IT: https://ggus.eu/index.php?mode=ticket_info&ticket_id=142592
**NGI_IT: https://ggus.eu/index.php?mode=ticket_info&ticket_id=142592
***INFN-LECCE (recovered)
***INFN-LECCE (recovered)
***INFN-MILANO-ATLASC: frequent SRM failures; overload due to atlas usage; improving in January
***INFN-MILANO-ATLASC: frequent SRM failures; overload due to atlas usage; improving in January
**NGI_PL: https://ggus.eu/index.php?mode=ticket_info&ticket_id=143945
**NGI_PL: https://ggus.eu/index.php?mode=ticket_info&ticket_id=143945
***ICM: qcg and SRM failures...
***ICM: qcg and SRM failures; statistics are improving...
**NGI_PL: https://ggus.eu/index.php?mode=ticket_info&ticket_id=140557
**NGI_PL: https://ggus.eu/index.php?mode=ticket_info&ticket_id=140557
***TASK: QCG and SRM issues
***TASK: QCG problems was fixed; some CREAM-CE failures; the SRM tests are still failing and the DPM upgrade wasn't completed yet
**NGI_PL: https://ggus.eu/index.php?mode=ticket_info&ticket_id=143506
**NGI_PL: https://ggus.eu/index.php?mode=ticket_info&ticket_id=143506
***WCSS64: CREAM-CE failures and new qcg failures
***WCSS64: CREAM-CE failures and new qcg failures; improving...
**NGI_RO: https://ggus.eu/index.php?mode=ticket_info&ticket_id=142160
**NGI_RO: https://ggus.eu/index.php?mode=ticket_info&ticket_id=142160
***RO-02-NIPNE: Cooling system problem, migration to CentOS7, long downtime...
***RO-02-NIPNE: Cooling system problem, migration to CentOS7, long downtime; to return in production soon, SRM decommissioned...
 
*Under-performed sites after 3 consecutive months, under-performed NGIs, QoS violations: ('''Dec 2019'''):
**AfricaArabia: https://ggus.eu/index.php?mode=ticket_info&ticket_id=144717
***MA-04-CNRST-ATLAS
**NGI_CH: https://ggus.eu/index.php?mode=ticket_info&ticket_id=144721
***UNIBE-LHEP: re-organization of the storage, re-deployment of ARC-CE
**NGI_FRANCE: https://ggus.eu/index.php?mode=ticket_info&ticket_id=144713
***AUVERGRID: CAs distribution not updated in time
***IN2P3-CC-T2: some trouble with the srmt2 publication due to a change of permissions with a publication file
**NGI_IT: https://ggus.eu/index.php?mode=ticket_info&ticket_id=144715
***INFN-CATANIA-STACK: failures affecting only the ops VO tenant, the other VOs are not affected
**NGI_UA: https://ggus.eu/index.php?mode=ticket_info&ticket_id=144718
**NGI_UA: https://ggus.eu/index.php?mode=ticket_info&ticket_id=144718
***UA_ISMA
***UA_ISMA: SRM failures solved with the DPM upgrade, statistics are improving...
**NGI_UK:
**NGI_UK:
***UKI-NORTHGRID-SHEF-HEP: https://ggus.eu/index.php?mode=ticket_info&ticket_id=144719 SRM is going to be dismissed
***UKI-NORTHGRID-SHEF-HEP: https://ggus.eu/index.php?mode=ticket_info&ticket_id=144719 SRM is going to be dismissed
***UKI-SOUTHGRID-SUSX: https://ggus.eu/index.php?mode=ticket_info&ticket_id=144720 Migration from CREAM to ARC, WN migration to CentOS7
***UKI-SOUTHGRID-SUSX: https://ggus.eu/index.php?mode=ticket_info&ticket_id=144720 Migration from CREAM to ARC, WN migration to CentOS7; SRM started to fail...
*Under-performed sites after 3 consecutive months, under-performed NGIs, QoS violations: ('''Jan 2020'''):
**AfricaArabia: https://ggus.eu/index.php?mode=ticket_info&ticket_id=145457
***MA-01-CNRST
**NGI_IT: https://ggus.eu/index.php?mode=ticket_info&ticket_id=145458
***GARR-01-DIR


*sites suspended:
*sites suspended:
**GR-12-TEIKAV


== IPv6 readiness plans  ==
== IPv6 readiness plans  ==
Line 85: Line 75:
* if any relevant, information will be summarised at  OMB
* if any relevant, information will be summarised at  OMB


== LCGDM end of support and migration to / enabling of DOME (TO UPDATE)==
== LCGDM end of support and migration to / enabling of DOME ==


* '''The DPM team has agreed to extend support for security updates for the DPM legacy functionality until 30 September 2019'''. However, affected service providers should still plan to disable legacy mode well before this date.
* '''The DPM team has agreed to extend support for security updates for the DPM legacy functionality until 30 September 2019'''. However, affected service providers should still plan to disable legacy mode well before this date.
Line 95: Line 85:
**VO Managers: to know any barriers in moving away from using SRM protocol https://www.surveymonkey.com/r/2ZMZJVG
**VO Managers: to know any barriers in moving away from using SRM protocol https://www.surveymonkey.com/r/2ZMZJVG


* '''Deployment statistics (Jan 10th):'''
* '''Deployment statistics (Feb 7th):'''
  $ ldapsearch -x -LLL -H ldap://egee-bdii.cnaf.infn.it:2170 -b "GLUE2GroupID=grid,o=glue" '(&(objectClass=GLUE2Manager)(GLUE2ManagerProductName=DPM))' GLUE2ManagerProductVersion GLUE2ManagerID | grep GLUE2ManagerProductVersion | sort | uniq -c
  $ ldapsearch -x -LLL -H ldap://egee-bdii.cnaf.infn.it:2170 -b "GLUE2GroupID=grid,o=glue" '(&(objectClass=GLUE2Manager)(GLUE2ManagerProductName=DPM))' GLUE2ManagerProductVersion GLUE2ManagerID | grep GLUE2ManagerProductVersion | sort | uniq -c
       4 GLUE2ManagerProductVersion: 1.10.0
       2 GLUE2ManagerProductVersion: 1.10.0
       1 GLUE2ManagerProductVersion: 1.12.0
       1 GLUE2ManagerProductVersion: 1.12.0
     65 GLUE2ManagerProductVersion: 1.13.0
     67 GLUE2ManagerProductVersion: 1.13.0
       2 GLUE2ManagerProductVersion: 1.13.1
       2 GLUE2ManagerProductVersion: 1.13.1
       7 GLUE2ManagerProductVersion: 1.13.2
       7 GLUE2ManagerProductVersion: 1.13.2
       6 GLUE2ManagerProductVersion: 1.8.10
       7 GLUE2ManagerProductVersion: 1.8.10
       1 GLUE2ManagerProductVersion: 1.8.8
       1 GLUE2ManagerProductVersion: 1.8.8
       1 GLUE2ManagerProductVersion: 1.8.9
       1 GLUE2ManagerProductVersion: 1.8.9
Line 132: Line 122:
| AEGIS03-ELEF-LEDA  
| AEGIS03-ELEF-LEDA  
| https://ggus.eu/index.php?mode=ticket_info&ticket_id=143152  
| https://ggus.eu/index.php?mode=ticket_info&ticket_id=143152  
| some problems, trying to complete by first week of november...
| webdav failures...
|-
|-
| ATLAND  
| ATLAND  
| https://ggus.eu/index.php?mode=ticket_info&ticket_id=143154  
| https://ggus.eu/index.php?mode=ticket_info&ticket_id=143154  
| upgrading first the DPM at AstrogridPUC....
| upgrading first the DPM at AstrogridPUC....
|-
| CESGA
| https://ggus.eu/index.php?mode=ticket_info&ticket_id=143157
| grid services to be decommissioned in the next months....
|-
|-
| GARR-01-DIR  
| GARR-01-DIR  
Line 148: Line 134:
| GRID-UNAM  
| GRID-UNAM  
| https://ggus.eu/index.php?mode=ticket_info&ticket_id=143161  
| https://ggus.eu/index.php?mode=ticket_info&ticket_id=143161  
|  
| scheduled a downtime in Feb for upgrade
|-
|-
| INDIACMS-TIFR  
| INDIACMS-TIFR  
Line 160: Line 146:
| MA-04-CNRST-ATLAS  
| MA-04-CNRST-ATLAS  
| https://ggus.eu/index.php?mode=ticket_info&ticket_id=143167  
| https://ggus.eu/index.php?mode=ticket_info&ticket_id=143167  
|  
| SRM disabled; waiting to fix webdav and gridftp failures
|-
|-
| OBSPM  
| OBSPM  
Line 176: Line 162:
| UA_ICYB_ARC  
| UA_ICYB_ARC  
| https://ggus.eu/index.php?mode=ticket_info&ticket_id=143178  
| https://ggus.eu/index.php?mode=ticket_info&ticket_id=143178  
|
|-
| UA-ISMA
| https://ggus.eu/index.php?mode=ticket_info&ticket_id=143176
|  
|  
|-
|-
Line 213: Line 195:
| https://ggus.eu/index.php?mode=ticket_info&ticket_id=143276  
| https://ggus.eu/index.php?mode=ticket_info&ticket_id=143276  
|  
|  
|-
| IR-IPM-HEP
| https://ggus.eu/index.php?mode=ticket_info&ticket_id=143088
| waiting for the monitoring...
|-
|-
| NIKHEF-ELPROD  
| NIKHEF-ELPROD  
Line 232: Line 210:
| TR-10-ULAKBIM
| TR-10-ULAKBIM
| https://ggus.eu/index.php?mode=ticket_info&ticket_id=143278
| https://ggus.eu/index.php?mode=ticket_info&ticket_id=143278
| by the end of January
| waiting for the SRR link
|-
| UKI-NORTHGRID-SHEF-HEP
| https://ggus.eu/index.php?mode=ticket_info&ticket_id=143079
| they plan to decommission the storage and to use XCache
|-
|-
| UKI-SCOTGRID-DURHAM
| UKI-SCOTGRID-DURHAM
Line 261: Line 235:
<br>
<br>


== HTCondorCE integration (TO UPDATE)==
== HTCondorCE integration ==


Link to procedure: https://wiki.egi.eu/wiki/PROC19
Link to procedure: https://wiki.egi.eu/wiki/PROC19
Line 278: Line 252:
***there is a bug on CentOS6 due to python version; the probe will be tested on the CentOS7 nagios instance
***there is a bug on CentOS6 due to python version; the probe will be tested on the CentOS7 nagios instance


== SECMON failures (TO UPDATE)==
== SECMON failures ==
Several CEs are failing the job submission tests, preventing pakiti to check the vulnerabilities fixes on the WNs.
Several CEs are failing the job submission tests, preventing pakiti to check the vulnerabilities fixes on the WNs.
*original ticket: https://ggus.eu/index.php?mode=ticket_info&ticket_id=143837
*original ticket: https://ggus.eu/index.php?mode=ticket_info&ticket_id=143837
Line 288: Line 262:
== Next meeting  ==
== Next meeting  ==


February 10th https://indico.egi.eu/event/4896/
March 9th https://indico.egi.eu/event/4897/

Latest revision as of 15:21, 10 February 2020

Main EGI.eu operations services Support Documentation Tools Activities Performance Technology Catch-all Services Resource Allocation Security


Documentation menu: Home Manuals Procedures Training Other Contact For: VO managers Administrators


Back to https://wiki.egi.eu/wiki/Operations_Meeting

General information (TO UPDATE)

Middleware (TO UPDATE)

UMD

Next scheduled release is February's UMD 4.10.0: https://wiki.egi.eu/wiki/UMD_Release_Schedule

  • plans on CentOS8 are starting!
  • integration of HTCondor-CE still finalizing
  • Are NGIs willing to investigate among their site admins about condor/HTCondor-CE skills and contribute with available recipes or help? -> no feedback at last EGI Ops meeting

CMD

  • caso 1.8.0 to be included

Preview repository

  • released on 2020-01-21:

Operations

ARGO/SAM

FedCloud

Feedback from DMSU

Monthly Availability/Reliability

  • sites suspended:

IPv6 readiness plans

LCGDM end of support and migration to / enabling of DOME

  • Deployment statistics (Feb 7th):
$ ldapsearch -x -LLL -H ldap://egee-bdii.cnaf.infn.it:2170 -b "GLUE2GroupID=grid,o=glue" '(&(objectClass=GLUE2Manager)(GLUE2ManagerProductName=DPM))' GLUE2ManagerProductVersion GLUE2ManagerID | grep GLUE2ManagerProductVersion | sort | uniq -c
     2 GLUE2ManagerProductVersion: 1.10.0
     1 GLUE2ManagerProductVersion: 1.12.0
    67 GLUE2ManagerProductVersion: 1.13.0
     2 GLUE2ManagerProductVersion: 1.13.1
     7 GLUE2ManagerProductVersion: 1.13.2
     7 GLUE2ManagerProductVersion: 1.8.10
     1 GLUE2ManagerProductVersion: 1.8.8
     1 GLUE2ManagerProductVersion: 1.8.9
     5 GLUE2ManagerProductVersion: 1.9.0


Liasing with WLCG to follow-up the upgrade. Opened GGUS tickets asking the following:

  • all the sites with older DPM versions than 1.12 are suggested to upgrade to the latest DPM version , following the guide DPM upgrade (chapter 1 Upgrade to DPM 1.10.0 "Legacy Flavour" and chapter 2 Upgrade to DPM 1.10.0 "Dome Flavour")
    • DOME and the old LCGDM (srm protocol) will coexist
  • Monitoring: sites should enable the monitoring of the HTTP/WebDav and/or GridFTP endpoints
    • register the storage service endpoint as webdav and/or globus-GRIDFTP service type, with production flag disabled, providing respectively the URL field and the Extension Properties information as explained in the HOWTO21
    • check if the tests are ok
    • switch the production flag to "yes"

List of tickets


Site Ticket Notes
AEGIS03-ELEF-LEDA https://ggus.eu/index.php?mode=ticket_info&ticket_id=143152 webdav failures...
ATLAND https://ggus.eu/index.php?mode=ticket_info&ticket_id=143154 upgrading first the DPM at AstrogridPUC....
GARR-01-DIR https://ggus.eu/index.php?mode=ticket_info&ticket_id=143160
GRID-UNAM https://ggus.eu/index.php?mode=ticket_info&ticket_id=143161 scheduled a downtime in Feb for upgrade
INDIACMS-TIFR https://ggus.eu/index.php?mode=ticket_info&ticket_id=142245 some problems...
MA-01-CNRST https://ggus.eu/index.php?mode=ticket_info&ticket_id=143166
MA-04-CNRST-ATLAS https://ggus.eu/index.php?mode=ticket_info&ticket_id=143167 SRM disabled; waiting to fix webdav and gridftp failures
OBSPM https://ggus.eu/index.php?mode=ticket_info&ticket_id=143169 they asked for quattor documentation...
TASK https://ggus.eu/index.php?mode=ticket_info&ticket_id=143174 problem with starting xrootd...
TR-03-METU https://ggus.eu/index.php?mode=ticket_info&ticket_id=143175 to configure QT?
UA_ICYB_ARC https://ggus.eu/index.php?mode=ticket_info&ticket_id=143178
WCSS64 https://ggus.eu/index.php?mode=ticket_info&ticket_id=143182 they need some time to gain enough knowledge for doing the upgrade....
GR-07-UOI-HEPLAB https://ggus.eu/index.php?mode=ticket_info&ticket_id=143467 some problems with the upgrade....
Hephy-Vienna https://ggus.eu/index.php?mode=ticket_info&ticket_id=143277 DPM will be replaced with EOS during Q1 2020
HK-LCG2 https://ggus.eu/index.php?mode=ticket_info&ticket_id=143471 before the end of 2019...
ICM https://ggus.eu/index.php?mode=ticket_info&ticket_id=143091 before end of December...
IN2P3-IPNL https://ggus.eu/index.php?mode=ticket_info&ticket_id=143082 migration to EOS, dpm should be dismissed by mid 2020
IN2P3-IRES https://ggus.eu/index.php?mode=ticket_info&ticket_id=143070 January 2020
INFN-ROMA1 https://ggus.eu/index.php?mode=ticket_info&ticket_id=143276
NIKHEF-ELPROD https://ggus.eu/index.php?mode=ticket_info&ticket_id=143286 We won't upgrade. we plan to migrate to dCache before the end of 2019.
PSNC https://ggus.eu/index.php?mode=ticket_info&ticket_id=143474
ru-PNPI https://ggus.eu/index.php?mode=ticket_info&ticket_id=143281
TR-10-ULAKBIM https://ggus.eu/index.php?mode=ticket_info&ticket_id=143278 waiting for the SRR link
UKI-SCOTGRID-DURHAM https://ggus.eu/index.php?mode=ticket_info&ticket_id=143465 migration to CentOS7, first....
UKI-SCOTGRID-ECDF https://ggus.eu/index.php?mode=ticket_info&ticket_id=143077 upgrading to CentOS7 first without DOME
UKI-SCOTGRID-GLASGOW https://ggus.eu/index.php?mode=ticket_info&ticket_id=143076 plan in progress
UKI-SOUTHGRID-BRIS-HEP https://ggus.eu/index.php?mode=ticket_info&ticket_id=143083
UNIBE-LHEP https://ggus.eu/index.php?mode=ticket_info&ticket_id=143464 to enable DOME and the monitoring...


HTCondorCE integration

Link to procedure: https://wiki.egi.eu/wiki/PROC19

GGUS ticket: https://ggus.eu/index.php?mode=ticket_info&ticket_id=139377

Remaining steps:

SECMON failures

Several CEs are failing the job submission tests, preventing pakiti to check the vulnerabilities fixes on the WNs.

AOB

Next meeting

March 9th https://indico.egi.eu/event/4897/