Difference between revisions of "Agenda-2020-07-13"
Jump to navigation
Jump to search
Line 94: | Line 94: | ||
*** SARA-MATRIX: SRM not published in the BDII: planned a network change to solve the problem. | *** SARA-MATRIX: SRM not published in the BDII: planned a network change to solve the problem. | ||
** NGI_UA: https://ggus.eu/index.php?mode=ticket_info&ticket_id=147750 | ** NGI_UA: https://ggus.eu/index.php?mode=ticket_info&ticket_id=147750 | ||
*** UA-ISMA | *** UA-ISMA: migration to ARC6 and other planned software updates | ||
*sites suspended: | *sites suspended: | ||
Revision as of 12:35, 8 July 2020
Main | EGI.eu operations services | Support | Documentation | Tools | Activities | Performance | Technology | Catch-all Services | Resource Allocation | Security |
Documentation menu: | Home • | Manuals • | Procedures • | Training • | Other • | Contact ► | For: | VO managers • | Administrators |
Back to https://wiki.egi.eu/wiki/Operations_Meeting
General information
Middleware
UMD
- plans on CentOS8 STARTED
Preview repository
- released on 2020-05-08
- Preview 1.27.0 AppDB info (sl6): ARC 6.5.0 and 6.6.0, CVMFS 2.7.2, dCache 5.2.20, frontier-squid 4.11.2, gfal2 2.17.2, xrootd 4.11.3
- Preview 2.27.0 AppDB info (CentOS 7): ARC 6.5.0 and 6.6.0, CVMFS 2.7.2, dCache 5.2.20, frontier-squid 4.11.2, gfal2 2.17.2, xrootd 4.11.3
Operations
ARGO/SAM
- new metrics added to ARGO_MON_OPERATORS profile on May 27th: eu.egi.CREAMCE-JobSubmit, eu.egi.CREAMCE.WN-Csh, eu.egi.CREAMCE.WN-Softver
- results: 177 endpoints, 15 WARNING (Timeout occurred (900 sec) ), 53 CRITICAL. Success rate 70% (61.6% including the WARNING)
- When eu.egi.CREAMCE.WN-Softver is successful:
CREAM JobOutput OK: retrieved outputSandbox: ['std.err', 'std.out'] **** std.err **** **** std.out **** egee01 has UMD 3.14.4
When it fails:
CREAM JobOutput ERROR [DONE-OK, exitCode=1 ]: retrieved outputSandbox: ['std.err', 'std.out'] **** std.err **** **** std.out **** ERROR: unable to find glite, EMI, LCG or UMD WN version on n1037-amd
- HTCondor-CE probes included in the ARGO_MON_OPERATORS profile on May 13th: https://ggus.eu/index.php?mode=ticket_info&ticket_id=146949
- 56 endpoints, 15 CRITICAL, success rate is about 73.2%
FedCloud
Feedback from DMSU
Verify configuration records
On a yearly basis, the information registered into GOC-DB need to be verified. NGIs and RCs have been asked to check them. In particular:
- NGI managers should review the people registered and the roles assigned to them, and in particular check the following information:
- ROD E-Mail
- Security E-Mail
- NGI Managers should also review the status of the "not certified" RCs, in according to the RC Status Workflow;
- RCs administrators should review the people registered and the roles assigned to them, and in particular check the following information:
- telephone numbers
- CSIRT E-Mail
- RC administrators should also review the information related to the registered service endpoints.
The process should be completed by June 22nd.
Monthly Availability/Reliability
- Under-performed sites in the past A/R reports with issues not yet fixed:
- AfricaArabia: https://ggus.eu/index.php?mode=ticket_info&ticket_id=146877
- ZA-WITS-CORE: SE hardware problem, machine sent to the vendor; CREAM-CE failures due to a known issue with the classads library (GGUS 146979)
- AsiaPacific: https://ggus.eu/index.php?mode=ticket_info&ticket_id=142591
- NGI_DE: https://ggus.eu/index.php?mode=ticket_info&ticket_id=146871
- GoeGRID: CREAM-CE intermittent failures not affecting ATLAS; failures with ARC-CE
- NGI_DE: https://ggus.eu/index.php?mode=ticket_info&ticket_id=147313
- mainz: some problems in March and April, that could not be fixed easily; in May, the HPC infrastructure was attacked and the whole computer center was shut down; in downtime.
- wuppertalprod: SRM failures to to a BDII issue, fixed
- NGI_UK:
- UKI-NORTHGRID-SHEF-HEP: https://ggus.eu/index.php?mode=ticket_info&ticket_id=146455 ARC-CE re-installed, some condor problems to fix
- UKI-SOUTHGRID-SUSX: https://ggus.eu/index.php?mode=ticket_info&ticket_id=144720 Migration from CREAM to ARC, WN migration to CentOS7; SRM to be decommissioned; ARC-CE was failing the IGTF test, then solved; site-bdii failures.
- ROC_CANADA: https://ggus.eu/index.php?mode=ticket_info&ticket_id=146452
- CA-SFU-T2: SLURM problems caused failures to site-BDII freshness check due to some old jobs not properly cancelled; recovered
- CA-WATERLOO-T2: SRM failures not involving production VOs, fixed; some unscheduled downtime affected the the A/R figures; ARC-CE and Site-BDII back in production with a minimum set of resources; A/R figures are improving.
- AfricaArabia: https://ggus.eu/index.php?mode=ticket_info&ticket_id=146877
- Under-performed sites after 3 consecutive months, under-performed NGIs, QoS violations: (June 2020):
- NGI_BG: https://ggus.eu/index.php?mode=ticket_info&ticket_id=147747
- BG01-IPP
- AsiaPacific: https://ggus.eu/index.php?mode=ticket_info&ticket_id=147748
- HK-HKU-CC-01
- TW-NCUHEP
- NGI_NL: https://ggus.eu/index.php?mode=ticket_info&ticket_id=147749
- SARA-MATRIX: SRM not published in the BDII: planned a network change to solve the problem.
- NGI_UA: https://ggus.eu/index.php?mode=ticket_info&ticket_id=147750
- UA-ISMA: migration to ARC6 and other planned software updates
- NGI_BG: https://ggus.eu/index.php?mode=ticket_info&ticket_id=147747
- sites suspended:
IPv6 readiness plans
- please provide updates to the IPv6 assessment (ongoing) https://wiki.egi.eu/w/index.php?title=IPV6_Assessment
- if any relevant, information will be summarised at OMB
ARC Middleware 5 end of support, migration to ARC 6
- EGI Operations Broadcast
- PROC16 Decommission of unsupported software
- deadline: end of July
- Catalin is in contact with ARC team to get a webinar on ARC administration, scheduled (to be confirmed) for July 6th please contact operations@ for information
- Status
Date | Number of endpoints in BDII | Number of GGUS tickets | Issues |
---|---|---|---|
2020-06-08 | 75 | 42 | Some ARC endpoints publish a timestamp instead of a version like 5.X.Y; we can fairly assume they are ARC6 nightly builds, but we're going to close the corresponding tickets after explicit confirmation from the site admin. |
LCGDM end of support and migration to / enabling of DOME
- The DPM team has agreed to extend support for security updates for the DPM legacy functionality until 30 September 2019. However, affected service providers should still plan to disable legacy mode well before this date.
- more details: https://wiki.egi.eu/wiki/DPM_End_of_legacy-mode_support
- since DPM 1.10.3 release, it is possible enabling the non-legacy mode DOME (Disk operations Management Engine, see documentation)
- latest DPM/dmlite release 1.13.2 has been also released in UMD 4.10.0
- EGI Operations sent a broadcast to site-admins(first bunch and second bunch) and VO managers with a survey to fill in:
- site-admins: to know the upgrade plans: https://www.surveymonkey.com/r/2PBZSNB
- VO Managers: to know any barriers in moving away from using SRM protocol https://www.surveymonkey.com/r/2ZMZJVG
- Deployment statistics (Jun 5th):
$ ldapsearch -x -LLL -H ldap://egee-bdii.cnaf.infn.it:2170 -b "GLUE2GroupID=grid,o=glue" '(&(objectClass=GLUE2Manager)(GLUE2ManagerProductName=DPM))' GLUE2ManagerProductVersion GLUE2ManagerID | grep GLUE2ManagerProductVersion | sort | uniq -c 1 GLUE2ManagerProductVersion: 1.10.0 65 GLUE2ManagerProductVersion: 1.13.0 2 GLUE2ManagerProductVersion: 1.13.1 11 GLUE2ManagerProductVersion: 1.13.2 3 GLUE2ManagerProductVersion: 1.8.10 1 GLUE2ManagerProductVersion: 1.8.9 4 GLUE2ManagerProductVersion: 1.9.0
Liasing with WLCG to follow-up the upgrade. Opened GGUS tickets asking the following:
- all the sites with older DPM versions than 1.12 are suggested to upgrade to the latest DPM version , following the guide DPM upgrade (chapter 1 Upgrade to DPM 1.10.0 "Legacy Flavour" and chapter 2 Upgrade to DPM 1.10.0 "Dome Flavour")
- DOME and the old LCGDM (srm protocol) will coexist
- Monitoring: sites should enable the monitoring of the HTTP/WebDav and/or GridFTP endpoints
- register the storage service endpoint as webdav and/or globus-GRIDFTP service type, with production flag disabled, providing respectively the URL field and the Extension Properties information as explained in the HOWTO21
- check if the tests are ok
- switch the production flag to "yes"
List of tickets
- DPM upgrade, DOME enabling, and monitoring (33)
- enabling DOME and proper monitoring (6)
- tickets opened by WLCG: 13 and 7
SECMON failures
Several CEs are failing the job submission tests, preventing pakiti to check the vulnerabilities fixes on the WNs.
- original ticket: https://ggus.eu/index.php?mode=ticket_info&ticket_id=143837
- List of tickets to the sites
- https://ggus.eu/index.php?mode=ticket_info&ticket_id=144732
AOB
- Storage accounting: http://goc-accounting.grid-support.ac.uk/storagetest/storagesitesystems.html
- several sites stopped publishing storage accounting records: it needs to investigate on and fix it
Next meeting
July 13th, 2020 https://indico.egi.eu/event/4901/