General information
Middleware
UMD
- CentOS Stream 8 now the recommended OS for new installations
- C8->CS8 migrations recommended
- CS9 will be supported by CERN and FNAL
- middleware: recommended path is C7->CS9 (we will probabily skip CS8)
- new release https://repository.egi.eu/UMD/4.15.1.html
- ARC-CE 6.13.0 bug fixes release
- Xrootd 5.3.1 bug fixes release
- CERN EOS 5.0.2 new release of EOS Open Storage which provides a storage solution large amounts of physics data and user files, with a focus on interactive and batch analysis.
- dCache 6.2.31 security vulnerability fix
- Infrastructure Manager Nagios probe 1.3.1
- GridFTP 13.21.1 minor bug fix of some Globus packages
- gfal2 2.19.2 regular update of the gfal clientes
- gfal2-utils 1.6.0 regular update of the gfal2-utils clientes
- EGI CVMFS 3.3.16 new release for the EGI default configuration meta-package configured for EGI.
- CVMFS 2.8.2 patch release containing bug fixes for clients and new diagnostics commands for the client.
- HTCondor 9.0.1 New major release of HTCondor
- HTCondor-CE 5.1.3 New Major Reelase of the HTCondor-CE
Preview repository
- released on 2021-06-10
- Preview 2.34.0 (CentOS 7): ARC 6.12.0, CVMFS 2.8.1, xrootd 5.2.0
- released on 2021-08-11
- Preview 2.35.0 (CentOS 7): APEL SSM 3.2.1, DPM/DMLite 1.15.0 and 1.15.1, frontier-squid 4.15.2, xrootd 5.3.0
- We plan to stop the release of Preview since it doesn't seem to be used very much, and it is also easier to catch the last version of the products from EPEL or the product teams repos, prior the release in UMD.
Operations
ARGO/SAM
- probe for checking the HTCondorCE host certificate validity deployed in production (GGUS 147386):
- checks on expiration date, CN, and CA:
- it is working fine (very few failures)
- In January we will make the request to included the metric in the A/R profile
- Memory limits set by the ARC-CE probe: https://ggus.eu/index.php?mode=ticket_info&ticket_id=155081
- the default is 512MB, but they were increased because failures on some sites
- 1GB for normal test jobs, 1.5GB for security jobs
- these limits seem to high for a simple test jobs that is expected to run fast and with low demand
- request to come back to the default limits and let the probe use particular settings in CEs if any
- the default is 512MB, but they were increased because failures on some sites
- a proposal could be:
- sites with particular environment settings can define the values on GOCDB using the extension properties
- the probe is executed with its default values unless there is something else defined on GOCDB
FedCloud
Feedback from DMSU
New Known Error Database (KEDB)
The KEDB has been moved to Jira+Confluence: https://confluence.egi.eu/display/EGIKEDB/EGI+Federation+KEDB+Home
- problems are tracked with Jira tickets to better follow-up their evoulution
- problems can be registered by DMSU staff and EGI Operations team
Monthly Availability/Reliability
- Under-performed sites in the past A/R reports with issues not yet fixed:
- AfricaArabia: https://ggus.eu/index.php?mode=ticket_info&ticket_id=154295
- MA-01-CNRST: ARC-CE failures: jobs are submitted but don't manage to finish; failures with org.nordugrid.ARC-CE-SRM-result
- NGI_IT: https://ggus.eu/index.php?mode=ticket_info&ticket_id=150818
- INFN-PISA: HTCondorCE failures fixed; SRM failures not yet; webdav failures
- NGI_PL: https://ggus.eu/index.php?mode=ticket_info&ticket_id=153659
- TASK: in the process of replacing QCG with ARC-CE
- NGI_RO: https://ggus.eu/index.php?mode=ticket_info&ticket_id=154746
- GRIDIFIN: jobs don;t manage to finish since 15th July
- NGI_PL: https://ggus.eu/index.php?mode=ticket_info&ticket_id=154747
- PSNC: storage backend issues affecting the HPC cluster and DPM, causing also ARC-CE instability; DPM issues were fixed, working on HPC cluster
- ROC_LA: https://ggus.eu/index.php?mode=ticket_info&ticket_id=153658
- SUPERCOMPUTO-UNAM: jobs cannot be submitted
- Russia: https://ggus.eu/index.php?mode=ticket_info&ticket_id=154748
- RU-SARFTI: ARC-CE failures, problem with hard drives, fixed
- AfricaArabia: https://ggus.eu/index.php?mode=ticket_info&ticket_id=154295
- Under-performed sites after 3 consecutive months, under-performed NGIs, QoS violations: (November 2021):
- AsiaPacific: https://ggus.eu/index.php?mode=ticket_info&ticket_id=155177
- INDIACMS-TIFR: major power outages, network issues; currently failures with HTCondorCE
- TW-NCHC
- NGI_DE: https://ggus.eu/index.php?mode=ticket_info&ticket_id=155178
- LRZ-LMU problem with retrieving the SURL
- MPPMU
- NGI_IBERGRID: https://ggus.eu/index.php?mode=ticket_info&ticket_id=155180
- NCG-INGRID-PT: problems with the webdav probes affecting only ops VO (see also GGUS 151396
- AsiaPacific: https://ggus.eu/index.php?mode=ticket_info&ticket_id=155177
- sites suspended:
Documentation
- plan to decommission MediaWiki
- content to be moved to different locations (confluence and https://docs.egi.eu/)
- confluence space hosting policies and procedures: EGI Policies and Procedures
- EGI Federation Operations
- Change Management, Release and Deployment Management, Incident and Service Request Management, Problem Management, Information Security Management
- Manuals, How-Tos, Troubleshooting, FAQs:
- huge number of material need to be reviewed and in case updated when moved to the new place
- location will be https://docs.egi.eu/providers/operations-manuals/
IPv6 readiness plans
- please provide updates to the IPv6 assessment (ongoing) https://wiki.egi.eu/w/index.php?title=IPV6_Assessment
- if any relevant, information will be summarised at OMB
Transition from X509 to federated identities (AARC profile token)
- WLCG is testing aai tokens (WLCG profile) as authz system for accessing the middleware, with Indigo IAM as a replacement of VOMS
- In Feb 2022 OSG will fully move to token-based AAI, abandoning X509 certificates
- HTCondorCE: replacement of Grid Community Toolkit
- The long-term support series (9.0.x) from the CHTC repositories will support X509/VOMS authentication through Sep 2022
- Starting in 9.3.0 (released in October), the HTCondor feature releases does NOT contain this support
- EGI sites are recommended to stay with the long-term support series for the time being
What we need to know in preparation of the transition:
Checking the middleware compliance with the AARC Profile token:
- ARC-CE: https://ggus.eu/index.php?mode=ticket_info&ticket_id=154958
- So far focusing on the WLCG profile, which is built upon the AARC profile, so this should cover everything.
- Argus: https://ggus.eu/index.php?mode=ticket_info&ticket_id=154959
- dCache: https://ggus.eu/index.php?mode=ticket_info&ticket_id=154960
- DPM: https://ggus.eu/index.php?mode=ticket_info&ticket_id=154961
- The DPM is in maintenance mode to be phased out by ~2024. There is no effort for implementing new functionality, which furthermore would be short-lived.
- HTCondor-CE: https://ggus.eu/index.php?mode=ticket_info&ticket_id=154962
- STORM: https://ggus.eu/index.php?mode=ticket_info&ticket_id=154963
Need to check the awareness and readiness of users communities:
- which GRID services do they use
- are they familiar with AAI identities
- are they ready for the switch
Migration of the VOs from VOMS to Check-in
- transition period where both X509 and tokens cam be used
- delays in updating the GRID elements to the latest version compliant with tokens
- not all if the middleware products can be complianat with tokens at the same time
- the same VO has to interact with element supporting different authentications
AOB
- DPM migration
- Transition from X509 to federated identities
Next meeting
Jan