The wiki is deprecated and due to be decommissioned by the end of September 2022.
The content is being migrated to other supports, new updates will be ignored and lost.
If needed you can get in touch with EGI SDIS team using operations @ egi.eu.
Difference between revisions of "Agenda-2020-04-20"
Jump to navigation
Jump to search
| (27 intermediate revisions by 2 users not shown) | |||
| Line 3: | Line 3: | ||
Back to https://wiki.egi.eu/wiki/Operations_Meeting | Back to https://wiki.egi.eu/wiki/Operations_Meeting | ||
= General information | = General information = | ||
= Middleware | = Middleware = | ||
== UMD == | == UMD == | ||
* | |||
** | * UMD 4.10.0 released in March http://repository.egi.eu/2020/03/26/release-umd-4-10-0/ (regular release) | ||
* UMD 4.10.2 released in April http://repository.egi.eu/2020/04/15/release-umd-4-10-2/ (HTCondor urgent update) | |||
* UMD 4.10.1 and 4.10.3 are fixes | |||
* CMD 1.3.0 released in February http://repository.egi.eu/2020/02/18/release-cmd-os-1-3-0/ (cloud info provider, APEL, caso 1.4.0) | |||
* plans on CentOS8 STARTED | * plans on CentOS8 STARTED | ||
** https://wiki.egi.eu/wiki/Next_middleware_release | ** https://wiki.egi.eu/wiki/Next_middleware_release | ||
* integration of HTCondor-CE still finalizing | * integration of HTCondor-CE still finalizing | ||
** still needed to contribute to the WN package | ** still needed to contribute to the WN package | ||
== CMD == | == CMD == | ||
| Line 24: | Line 25: | ||
** '''[[Preview 1.26.0]]''' [https://appdb.egi.eu/store/software/preview.repository/releases/1.0/1.26.0/ AppDB info] (sl6): ARC 6.4.1, CVMFS 2.7.0, dCache 5.2.13, gfal2 2.17.1, STORM 1.11.17, xrootd 4.11.1 | ** '''[[Preview 1.26.0]]''' [https://appdb.egi.eu/store/software/preview.repository/releases/1.0/1.26.0/ AppDB info] (sl6): ARC 6.4.1, CVMFS 2.7.0, dCache 5.2.13, gfal2 2.17.1, STORM 1.11.17, xrootd 4.11.1 | ||
** '''[[Preview 2.26.0]]''' [https://appdb.egi.eu/store/software/preview.repository/releases/2.0/2.26.0/ AppDB info] (CentOS 7): ARC 6.4.1, CVMFS 2.7.0, dCache 5.2.13, gfal2 2.17.1, STORM 1.11.17, xrootd 4.11.1 | ** '''[[Preview 2.26.0]]''' [https://appdb.egi.eu/store/software/preview.repository/releases/2.0/2.26.0/ AppDB info] (CentOS 7): ARC 6.4.1, CVMFS 2.7.0, dCache 5.2.13, gfal2 2.17.1, STORM 1.11.17, xrootd 4.11.1 | ||
* collecting information for the next update | |||
= Operations = | = Operations = | ||
== ARGO/SAM == | == ARGO/SAM == | ||
* | * Swift probe deployed in production https://ggus.eu/index.php?mode=ticket_info&ticket_id=143069 | ||
* | * new cream probe under test: https://ggus.eu/index.php?mode=ticket_info&ticket_id=144554 | ||
** general failures with eu.egi.CREAMCE.WN-Softver need to be investigated | |||
* HTCondor-CE probe [https://argo-mon-fedcloud.cro-ngi.hr/nagios/cgi-bin/status.cgi?servicegroup=SERVICE_org.opensciencegrid.htcondorce&style=detail deployed on the test instance]: https://ggus.eu/index.php?mode=ticket_info&ticket_id=141177 | |||
** | ** success rate is about 73% | ||
* | ** some WARNING status (''WARNING - ETF internal error while publishing WN results'') due to a bug in the probe; fixed in the new version, to be deployed | ||
** | |||
** | |||
== FedCloud == | == FedCloud == | ||
[https://ggus.eu/?mode=ticket_search&status=open&user=spinoso&date_type=creation+date&tf_radio=1&timeframe=any&keyword=Please%20upgrade%20to%20cASO%201.4.0&orderticketsby=REQUEST_ID&orderhow=desc&search_submit=GO cASO upgade campaign on OpenStack sites] | * [https://ggus.eu/?mode=ticket_search&status=open&user=spinoso&date_type=creation+date&tf_radio=1&timeframe=any&keyword=Please%20upgrade%20to%20cASO%201.4.0&orderticketsby=REQUEST_ID&orderhow=desc&search_submit=GO cASO upgade campaign on OpenStack sites] | ||
* [https://indico.egi.eu/event/4775/ badging] | |||
== Feedback from DMSU == | == Feedback from DMSU == | ||
| Line 53: | Line 54: | ||
***INFN-BARI: SRM failures | ***INFN-BARI: SRM failures | ||
**NGI_IT: https://ggus.eu/index.php?mode=ticket_info&ticket_id=145458 | **NGI_IT: https://ggus.eu/index.php?mode=ticket_info&ticket_id=145458 | ||
***GARR-01-DIR: failures during the DPM upgrade | ***GARR-01-DIR: failures during the DPM upgrade => '''tomorrow will be suspended''' | ||
**NGI_IT: https://ggus.eu/index.php?mode=ticket_info&ticket_id=144715 | **NGI_IT: https://ggus.eu/index.php?mode=ticket_info&ticket_id=144715 | ||
***INFN-CATANIA-STACK: failures with IGTF and OpenStack probes | ***INFN-CATANIA-STACK: failures with IGTF and OpenStack probes | ||
| Line 63: | Line 64: | ||
***RO-02-NIPNE: after long downtime, CREAM-CE still failing the tests, SRM endpoint to be removed from GOCDB... | ***RO-02-NIPNE: after long downtime, CREAM-CE still failing the tests, SRM endpoint to be removed from GOCDB... | ||
**NGI_UK: | **NGI_UK: | ||
***UKI-SOUTHGRID-SUSX: https://ggus.eu/index.php?mode=ticket_info&ticket_id=144720 Migration from CREAM to ARC, WN migration to CentOS7; SRM decommissioned; ARC-CE is failing the tests | ***UKI-SOUTHGRID-SUSX: https://ggus.eu/index.php?mode=ticket_info&ticket_id=144720 Migration from CREAM to ARC, WN migration to CentOS7; SRM to be decommissioned; ARC-CE is failing the tests (now the IGTF is failing) | ||
**ROC_LA https://ggus.eu/index.php?mode=ticket_info&ticket_id=145812 | **ROC_LA https://ggus.eu/index.php?mode=ticket_info&ticket_id=145812 | ||
***GRID-UNAM: DPM upgrade not yet completed | ***GRID-UNAM: DPM upgrade not yet completed | ||
| Line 70: | Line 71: | ||
***INFN-ROMA1-CMS | ***INFN-ROMA1-CMS | ||
**NGI_TR: https://ggus.eu/index.php?mode=ticket_info&ticket_id=146433 | **NGI_TR: https://ggus.eu/index.php?mode=ticket_info&ticket_id=146433 | ||
***AZ-IFAN | ***AZ-IFAN: experienced some network problems, now fixed | ||
***TR-FC1-ULAKBIM | ***TR-FC1-ULAKBIM: failures with the "old" OCCI protocol; they will move from VOMS to OIDC; some inconsistencies with eu.egi.cloud.OCCI-Context between the results in nagios and the report in ARGO | ||
**NGI_UA: https://ggus.eu/index.php?mode=ticket_info&ticket_id=146454 | **NGI_UA: https://ggus.eu/index.php?mode=ticket_info&ticket_id=146454 | ||
***UA-KNU: GlueSEUniqueID wasn't published due to missing DNS back-resolving record for IPv6 address of the SE. | ***UA-KNU: GlueSEUniqueID wasn't published due to missing DNS back-resolving record for IPv6 address of the SE. | ||
**NGI_UK: https://ggus.eu/index.php?mode=ticket_info&ticket_id=146455 | **NGI_UK: https://ggus.eu/index.php?mode=ticket_info&ticket_id=146455 | ||
***UKI-NORTHGRID-SHEF-HEP | ***UKI-NORTHGRID-SHEF-HEP: ARC-CE failures | ||
**ROC_CANADA: https://ggus.eu/index.php?mode=ticket_info&ticket_id=146452 | **ROC_CANADA: https://ggus.eu/index.php?mode=ticket_info&ticket_id=146452 | ||
***CA-SFU-T2: SLURM problems caused failures to site-BDII freshness check due to some old jobs not properly cancelled; A/R figures are now improving | ***CA-SFU-T2: SLURM problems caused failures to site-BDII freshness check due to some old jobs not properly cancelled; A/R figures are now improving | ||
| Line 84: | Line 85: | ||
== IPv6 readiness plans == | == IPv6 readiness plans == | ||
* | * please provide updates to the IPv6 assessment (ongoing) https://wiki.egi.eu/w/index.php?title=IPV6_Assessment | ||
* if any relevant, information will be summarised at OMB | * if any relevant, information will be summarised at OMB | ||
| Line 92: | Line 93: | ||
**more details: https://wiki.egi.eu/wiki/DPM_End_of_legacy-mode_support | **more details: https://wiki.egi.eu/wiki/DPM_End_of_legacy-mode_support | ||
* since [http://lcgdm.web.cern.ch/dpm-1103-released-epel DPM 1.10.3 release], it is possible enabling the non-legacy mode DOME (Disk operations Management Engine, [https://gitlab.cern.ch/lcgdm/dmlite/blob/master/doc/dome/dome.pdf see documentation]) | * since [http://lcgdm.web.cern.ch/dpm-1103-released-epel DPM 1.10.3 release], it is possible enabling the non-legacy mode DOME (Disk operations Management Engine, [https://gitlab.cern.ch/lcgdm/dmlite/blob/master/doc/dome/dome.pdf see documentation]) | ||
**latest [http://lcgdm.web.cern.ch/dmlitedpm-1132-released-epel DPM release 1.13.2] has been also released in [http://repository.egi.eu/ | **latest [http://lcgdm.web.cern.ch/dmlitedpm-1132-released-epel DPM/dmlite release 1.13.2] has been also released in [http://repository.egi.eu/2020/03/26/release-umd-4-10-0/ UMD 4.10.0] | ||
* '''EGI Operations sent a broadcast to site-admins([https://operations-portal.egi.eu/broadcast/archive/2426 first bunch] and [https://operations-portal.egi.eu/broadcast/archive/2438 second bunch]) and [https://operations-portal.egi.eu/broadcast/archive/2420 VO managers] with a survey to fill in''': | * '''EGI Operations sent a broadcast to site-admins([https://operations-portal.egi.eu/broadcast/archive/2426 first bunch] and [https://operations-portal.egi.eu/broadcast/archive/2438 second bunch]) and [https://operations-portal.egi.eu/broadcast/archive/2420 VO managers] with a survey to fill in''': | ||
**site-admins: to know the upgrade plans: https://www.surveymonkey.com/r/2PBZSNB | **site-admins: to know the upgrade plans: https://www.surveymonkey.com/r/2PBZSNB | ||
| Line 145: | Line 146: | ||
| INDIACMS-TIFR | | INDIACMS-TIFR | ||
| https://ggus.eu/index.php?mode=ticket_info&ticket_id=142245 | | https://ggus.eu/index.php?mode=ticket_info&ticket_id=142245 | ||
| | | upgrade delayed to to COVID-19 lockdown in Mumbai | ||
|- | |- | ||
| MA-01-CNRST | | MA-01-CNRST | ||
| Line 162: | Line 163: | ||
| https://ggus.eu/index.php?mode=ticket_info&ticket_id=143174 | | https://ggus.eu/index.php?mode=ticket_info&ticket_id=143174 | ||
| problem with starting xrootd... | | problem with starting xrootd... | ||
|- | |- | ||
| UA_ICYB_ARC | | UA_ICYB_ARC | ||
| Line 177: | Line 174: | ||
| GR-07-UOI-HEPLAB | | GR-07-UOI-HEPLAB | ||
| https://ggus.eu/index.php?mode=ticket_info&ticket_id=143467 | | https://ggus.eu/index.php?mode=ticket_info&ticket_id=143467 | ||
| some problems with the upgrade | | still on slc6, some problems with the upgrade; before the end of the year all the site will be migrated to CentOS 7 | ||
|- | |- | ||
| Hephy-Vienna | | Hephy-Vienna | ||
| Line 185: | Line 182: | ||
| HK-LCG2 | | HK-LCG2 | ||
| https://ggus.eu/index.php?mode=ticket_info&ticket_id=143471 | | https://ggus.eu/index.php?mode=ticket_info&ticket_id=143471 | ||
| | | April 2020 | ||
|- | |- | ||
| ICM | | ICM | ||
| Line 197: | Line 194: | ||
| IN2P3-IRES | | IN2P3-IRES | ||
| https://ggus.eu/index.php?mode=ticket_info&ticket_id=143070 | | https://ggus.eu/index.php?mode=ticket_info&ticket_id=143070 | ||
| | | testing the upgrade on the test infrastructure | ||
|- | |- | ||
| INFN-ROMA1 | | INFN-ROMA1 | ||
| https://ggus.eu/index.php?mode=ticket_info&ticket_id=143276 | | https://ggus.eu/index.php?mode=ticket_info&ticket_id=143276 | ||
| | | to enable the webdav monitoring | ||
|- | |- | ||
| NIKHEF-ELPROD | | NIKHEF-ELPROD | ||
| Line 249: | Line 246: | ||
**https://gitlab.cern.ch/etf/jess | **https://gitlab.cern.ch/etf/jess | ||
**ticket to ARGO to deploy the probes on the test instance: https://ggus.eu/index.php?mode=ticket_info&ticket_id=141177 | **ticket to ARGO to deploy the probes on the test instance: https://ggus.eu/index.php?mode=ticket_info&ticket_id=141177 | ||
***probe [https://argo-mon-fedcloud.cro-ngi.hr/nagios/cgi-bin/status.cgi?servicegroup=SERVICE_org.opensciencegrid.htcondorce&style=detail deployed on the test instance]: success rate is about 73%, some WARNING status to | ***probe [https://argo-mon-fedcloud.cro-ngi.hr/nagios/cgi-bin/status.cgi?servicegroup=SERVICE_org.opensciencegrid.htcondorce&style=detail deployed on the test instance]: success rate is about 73%, some WARNING status (''WARNING - ETF internal error while publishing WN results'') due to a bug in the probe; fixed in the new version, to be deployed | ||
== SECMON failures == | == SECMON failures == | ||
| Line 258: | Line 255: | ||
= AOB = | = AOB = | ||
* Storage accounting: http://goc-accounting.grid-support.ac.uk/storagetest/storagesitesystems.html | |||
** several sites stopped publishing storage accounting records: it needs to investigate on and fix it | |||
== Next meeting == | == Next meeting == | ||
May 11th, 2020 https://indico.egi.eu/event/4899/ | |||
Latest revision as of 15:20, 20 April 2020
| Main | EGI.eu operations services | Support | Documentation | Tools | Activities | Performance | Technology | Catch-all Services | Resource Allocation | Security |
| Documentation menu: | Home • | Manuals • | Procedures • | Training • | Other • | Contact ► | For: | VO managers • | Administrators |
Back to https://wiki.egi.eu/wiki/Operations_Meeting
General information
Middleware
UMD
- UMD 4.10.0 released in March http://repository.egi.eu/2020/03/26/release-umd-4-10-0/ (regular release)
- UMD 4.10.2 released in April http://repository.egi.eu/2020/04/15/release-umd-4-10-2/ (HTCondor urgent update)
- UMD 4.10.1 and 4.10.3 are fixes
- CMD 1.3.0 released in February http://repository.egi.eu/2020/02/18/release-cmd-os-1-3-0/ (cloud info provider, APEL, caso 1.4.0)
- plans on CentOS8 STARTED
- integration of HTCondor-CE still finalizing
- still needed to contribute to the WN package
CMD
- CMD-OS released http://repository.egi.eu/2020/02/18/release-cmd-os-1-3-0/
Preview repository
- released on 2020-01-21:
- Preview 1.26.0 AppDB info (sl6): ARC 6.4.1, CVMFS 2.7.0, dCache 5.2.13, gfal2 2.17.1, STORM 1.11.17, xrootd 4.11.1
- Preview 2.26.0 AppDB info (CentOS 7): ARC 6.4.1, CVMFS 2.7.0, dCache 5.2.13, gfal2 2.17.1, STORM 1.11.17, xrootd 4.11.1
- collecting information for the next update
Operations
ARGO/SAM
- Swift probe deployed in production https://ggus.eu/index.php?mode=ticket_info&ticket_id=143069
- new cream probe under test: https://ggus.eu/index.php?mode=ticket_info&ticket_id=144554
- general failures with eu.egi.CREAMCE.WN-Softver need to be investigated
- HTCondor-CE probe deployed on the test instance: https://ggus.eu/index.php?mode=ticket_info&ticket_id=141177
- success rate is about 73%
- some WARNING status (WARNING - ETF internal error while publishing WN results) due to a bug in the probe; fixed in the new version, to be deployed
FedCloud
Feedback from DMSU
Monthly Availability/Reliability
- Under-performed sites in the past A/R reports with issues not yet fixed:
- AsiaPacific: https://ggus.eu/index.php?mode=ticket_info&ticket_id=142591
- NGI_CH: https://ggus.eu/index.php?mode=ticket_info&ticket_id=145818
- UNIGE-DPNC: only the site-bdii is in production. The site is undergoing re-organisation, currently in downtime till Apr 30th; new ARC-CE under test.
- NGI_IT: https://ggus.eu/index.php?mode=ticket_info&ticket_id=145815
- INFN-BARI: SRM failures
- NGI_IT: https://ggus.eu/index.php?mode=ticket_info&ticket_id=145458
- GARR-01-DIR: failures during the DPM upgrade => tomorrow will be suspended
- NGI_IT: https://ggus.eu/index.php?mode=ticket_info&ticket_id=144715
- INFN-CATANIA-STACK: failures with IGTF and OpenStack probes
- NGI_PL: https://ggus.eu/index.php?mode=ticket_info&ticket_id=143945
- ICM: qcg and SRM failures; statistics decreasing again...
- NGI_PL: https://ggus.eu/index.php?mode=ticket_info&ticket_id=140557
- TASK: QCG problems was fixed; some CREAM-CE failures; the SRM tests are still failing and the DPM upgrade wasn't completed yet
- NGI_RO: https://ggus.eu/index.php?mode=ticket_info&ticket_id=142160
- RO-02-NIPNE: after long downtime, CREAM-CE still failing the tests, SRM endpoint to be removed from GOCDB...
- NGI_UK:
- UKI-SOUTHGRID-SUSX: https://ggus.eu/index.php?mode=ticket_info&ticket_id=144720 Migration from CREAM to ARC, WN migration to CentOS7; SRM to be decommissioned; ARC-CE is failing the tests (now the IGTF is failing)
- ROC_LA https://ggus.eu/index.php?mode=ticket_info&ticket_id=145812
- GRID-UNAM: DPM upgrade not yet completed
- Under-performed sites after 3 consecutive months, under-performed NGIs, QoS violations: (March 2020):
- NGI_IT: https://ggus.eu/index.php?mode=ticket_info&ticket_id=146453
- INFN-ROMA1-CMS
- NGI_TR: https://ggus.eu/index.php?mode=ticket_info&ticket_id=146433
- AZ-IFAN: experienced some network problems, now fixed
- TR-FC1-ULAKBIM: failures with the "old" OCCI protocol; they will move from VOMS to OIDC; some inconsistencies with eu.egi.cloud.OCCI-Context between the results in nagios and the report in ARGO
- NGI_UA: https://ggus.eu/index.php?mode=ticket_info&ticket_id=146454
- UA-KNU: GlueSEUniqueID wasn't published due to missing DNS back-resolving record for IPv6 address of the SE.
- NGI_UK: https://ggus.eu/index.php?mode=ticket_info&ticket_id=146455
- UKI-NORTHGRID-SHEF-HEP: ARC-CE failures
- ROC_CANADA: https://ggus.eu/index.php?mode=ticket_info&ticket_id=146452
- CA-SFU-T2: SLURM problems caused failures to site-BDII freshness check due to some old jobs not properly cancelled; A/R figures are now improving
- CA-WATERLOO-T2: SRM failures not involving production VOs, fixed, figures improving
- NGI_IT: https://ggus.eu/index.php?mode=ticket_info&ticket_id=146453
- sites suspended:
IPv6 readiness plans
- please provide updates to the IPv6 assessment (ongoing) https://wiki.egi.eu/w/index.php?title=IPV6_Assessment
- if any relevant, information will be summarised at OMB
LCGDM end of support and migration to / enabling of DOME
- The DPM team has agreed to extend support for security updates for the DPM legacy functionality until 30 September 2019. However, affected service providers should still plan to disable legacy mode well before this date.
- more details: https://wiki.egi.eu/wiki/DPM_End_of_legacy-mode_support
- since DPM 1.10.3 release, it is possible enabling the non-legacy mode DOME (Disk operations Management Engine, see documentation)
- latest DPM/dmlite release 1.13.2 has been also released in UMD 4.10.0
- EGI Operations sent a broadcast to site-admins(first bunch and second bunch) and VO managers with a survey to fill in:
- site-admins: to know the upgrade plans: https://www.surveymonkey.com/r/2PBZSNB
- VO Managers: to know any barriers in moving away from using SRM protocol https://www.surveymonkey.com/r/2ZMZJVG
- Deployment statistics (Apr 17th):
$ ldapsearch -x -LLL -H ldap://egee-bdii.cnaf.infn.it:2170 -b "GLUE2GroupID=grid,o=glue" '(&(objectClass=GLUE2Manager)(GLUE2ManagerProductName=DPM))' GLUE2ManagerProductVersion GLUE2ManagerID | grep GLUE2ManagerProductVersion | sort | uniq -c
1 GLUE2ManagerProductVersion: 1.10.0
67 GLUE2ManagerProductVersion: 1.13.0
2 GLUE2ManagerProductVersion: 1.13.1
8 GLUE2ManagerProductVersion: 1.13.2
4 GLUE2ManagerProductVersion: 1.8.10
1 GLUE2ManagerProductVersion: 1.8.8
1 GLUE2ManagerProductVersion: 1.8.9
4 GLUE2ManagerProductVersion: 1.9.0
Liasing with WLCG to follow-up the upgrade. Opened GGUS tickets asking the following:
- all the sites with older DPM versions than 1.12 are suggested to upgrade to the latest DPM version , following the guide DPM upgrade (chapter 1 Upgrade to DPM 1.10.0 "Legacy Flavour" and chapter 2 Upgrade to DPM 1.10.0 "Dome Flavour")
- DOME and the old LCGDM (srm protocol) will coexist
- Monitoring: sites should enable the monitoring of the HTTP/WebDav and/or GridFTP endpoints
- register the storage service endpoint as webdav and/or globus-GRIDFTP service type, with production flag disabled, providing respectively the URL field and the Extension Properties information as explained in the HOWTO21
- check if the tests are ok
- switch the production flag to "yes"
List of tickets
- DPM upgrade, DOME enabling, and monitoring (33)
- enabling DOME and proper monitoring (6)
- tickets opened by WLCG: 13 and 7
HTCondorCE integration
Link to procedure: https://wiki.egi.eu/wiki/PROC19
GGUS ticket: https://ggus.eu/index.php?mode=ticket_info&ticket_id=139377
Remaining steps:
- (To do) OPS Dashboard:
- To do when Conf Mgmt and Monitoring are completed.
- (in progress) Information System: the info-provider is already available: https://github.com/opensciencegrid/htcondor-ce/tree/master/contrib/bdii
- to test during the staged rollout
- (in progress) Monitoring: some probes are already available that submit jobs to CondorCe, CREAM and ARC-CE:
- https://gitlab.cern.ch/etf/jess
- ticket to ARGO to deploy the probes on the test instance: https://ggus.eu/index.php?mode=ticket_info&ticket_id=141177
- probe deployed on the test instance: success rate is about 73%, some WARNING status (WARNING - ETF internal error while publishing WN results) due to a bug in the probe; fixed in the new version, to be deployed
SECMON failures
Several CEs are failing the job submission tests, preventing pakiti to check the vulnerabilities fixes on the WNs.
- original ticket: https://ggus.eu/index.php?mode=ticket_info&ticket_id=143837
- List of tickets to the sites
- https://ggus.eu/index.php?mode=ticket_info&ticket_id=144732
AOB
- Storage accounting: http://goc-accounting.grid-support.ac.uk/storagetest/storagesitesystems.html
- several sites stopped publishing storage accounting records: it needs to investigate on and fix it
Next meeting
May 11th, 2020 https://indico.egi.eu/event/4899/