Agenda-2021-10-11

Main

EGI.eu operations services

Support

Documentation

Tools

Activities

Performance

Technology

Catch-all Services

Resource Allocation

Security

Documentation menu:

Home •

Manuals •

Procedures •

Training •

Other •

Contact ►

For:

VO managers •

Administrators

Back to https://wiki.egi.eu/wiki/Operations_Meeting

Middleware

UMD

• no news related to CentOS8

Preview repository

• released on 2021-06-10
  • Preview 2.34.0 (CentOS 7): ARC 6.12.0, CVMFS 2.8.1, xrootd 5.2.0
• released on 2021-08-11
  • Preview 2.35.0 (CentOS 7): APEL SSM 3.2.1, DPM/DMLite 1.15.0 and 1.15.1, frontier-squid 4.15.2, xrootd 5.3.0

Operations

ARGO/SAM

• probe for checking the HTCondorCE host certificate validity (GGUS 147386):
  • checks on expiration date, CN, and CA:
    • https://argo-mon-devel.egi.eu/nagios/cgi-bin/status.cgi?servicegroup=SERVICE_org.opensciencegrid.htcondorce&style=detail
  • to be deployed in production once new condor client is released in UMD

FedCloud

• cASO upgade campaign on OpenStack sites
• badging

New Known Error Database (KEDB)

The KEDB has been moved to Jira+Confluence: https://confluence.egi.eu/display/EGIKEDB/EGI+Federation+KEDB+Home

• problems are tracked with Jira tickets to better follow-up their evoulution
• problems can be registered by DMSU staff and EGI Operations team

Monthly Availability/Reliability

• Under-performed sites in the past A/R reports with issues not yet fixed:
  • NGI_FRANCE: https://ggus.eu/index.php?mode=ticket_info&ticket_id=152253
    • AUVERGRID: Long downtime connected to IN2P3-LPC site. Some problems with org.nordugrid.ARC-CE-result and org.nordugrid.ARC-CE-srm metrics: even if the sub-metrics complete successfully, the test jobs don't manage to get the "ending" status, producing an UNKNOWN status in the A/R computation; problem fixed after updating ARC-CE and condor to a newer version.
  • NGI_IT: https://ggus.eu/index.php?mode=ticket_info&ticket_id=150818
    • INFN-PISA: HTCondorCE failures fixed; SRM failures not yet
  • NGI_PL: https://ggus.eu/index.php?mode=ticket_info&ticket_id=153659
    • TASK: oin the process of replacing QCG with ARC-CE
  • NGI_UA: https://ggus.eu/index.php?mode=ticket_info&ticket_id=152841
    • UA-NSCMBR: problem during the DPM update: conflict between xrootd 5 and dmlite 1.13. Unscheduled downtime due to power failure in the computing centre. NFS configuration issue affected ARC-CE. Accounting data republished using the ARC accountng functionalities.
  • NGI_UK: https://ggus.eu/index.php?mode=ticket_info&ticket_id=153660
    • UKI-SOUTHGRID-SUSX: CE configuration issues; some other failures occurred.
  • ROC_LA: https://ggus.eu/index.php?mode=ticket_info&ticket_id=153658
    • SUPERCOMPUTO-UNAM: some network issues
• Under-performed sites after 3 consecutive months, under-performed NGIs, QoS violations: (September 2021):
  • AfricaArabia: https://ggus.eu/index.php?mode=ticket_info&ticket_id=154295
    • MA-01-CNRST
  • AsiaPacific: https://ggus.eu/index.php?mode=ticket_info&ticket_id=154294
    • JP-KEK-CRC-02
  • NGI_BG: https://ggus.eu/index.php?mode=ticket_info&ticket_id=154293
    • BG05-SUGrid: unforeseen power outage, nodes recovered.
  • NGI_NDGF: https://ggus.eu/index.php?mode=ticket_info&ticket_id=154297
    • SE-SNIC-T2
  • NGI_UA: https://ggus.eu/index.php?mode=ticket_info&ticket_id=154298
    • UA-IFBG
  • ROC_LA: https://ggus.eu/index.php?mode=ticket_info&ticket_id=154292
    • AstrogridPUC

• sites suspended:

Documentation

• plan to decommission MediaWiki
• content to be moved to different locations (confluence and https://docs.egi.eu/)
• confluence space hosting policies and procedures: EGI Policies and Procedures
  • EGI Federation Operations
  • Change Management, Release and Deployment Management, Incident and Service Request Management, Problem Management, Information Security Management
• Manuals, How-Tos, Troubleshooting, FAQs:
  • huge number of material need to be reviewed and in case updated when moved to the new place
  • location will be https://docs.egi.eu/providers/operations-manuals/

IPv6 readiness plans

• please provide updates to the IPv6 assessment (ongoing) https://wiki.egi.eu/w/index.php?title=IPV6_Assessment
• if any relevant, information will be summarised at OMB

APEL migration from ActiveMQ to ARGO Message Service (AMS)

• ActiveMQ dismissed on July 8th: for security reasons it is not possible maintain it any longer.
  • Scheduled Donwtime: https://goc.egi.eu/portal/index.php?Page_Type=Downtime&id=30888 https://goc.egi.eu/portal/index.php?Page_Type=Downtime&id=30889
• Migration insructions (HTCondorCE, Storage, and Cloud accounting): https://github.com/apel/ssm/blob/dev/migrating_to_ams.md
• ARC 6.12.0 released, instructions:
  • http://www.nordugrid.org/arc/releases/6.12/release_notes_6.12.html
  • all the sites with ARC-CE need to update to this version
• Recommended versions:
  • Apel Clien: 1.9.0
  • APEL SSM: 3.2.1
• Cloud accounting campaign:
  • list of tickets
  • 2 tickets (out of 21) not solved yet
    • UA-BITP 150857: old records associated to "None" VO (due to a configuration issue) needs to be removed from the repository
    • CYFRONET-CLOUD 150845:
• HTCondorCE and Storage accounting campaign:
  • list of tickets
  • 9 tickets (out of 53) not solved yet
    • Hephy-Vienna 150961: they have a problem with the apel parser under investigation.
    • ICN-UNAM 151374:
    • INFN-COSENZA 151183: using an old SSM version
    • INFN-NAPOLI-ATLAS 150976: some problems similar to https://ggus.eu/index.php?mode=ticket_info&ticket_id=154347
    • INFN-PISA 150977:
    • INFN-ROMA3 150978:
    • INFN-TRIESTE 151981:
    • SUPERCOMPUTO-UNAM 151375:
    • UA-BITP 150991: accounting records not properly parsed
• ARC-CE and storage accounting campaign:
  • list of tickets
  • 11 tickets (out of 112) not solved yet
    • Australia-ATLAS152428 and Australia-T2 152429: they stilll have ARC-CE 5.4; moving to a Cloudscheduler based compute system and will be removing the ARC-CE's in the near future
    • CA-SFU-T2 152433: CEs updated, check the accounting publication in the coming days... some errors with the benchmark which seem harmful. Duplicated records for the previous months, it was suggested to set `apel_messages = summaries` in the arc conf file.
    • IN2P3-IPNL 152460: CE not yet in production
    • JP-KEK-CRC-02 152471: installed new CEs, some authz failures with sending the records...
    • RU-SPbSU 152491: some discrepancy between local database and central repository, involved ARC developers, see GGUS 154090...
    • Taiwan-LCG2 152497: setting up the ARC6 server...
    • TASK 152498:
    • TW-FTT 152503:
    • UA-MHI 152509: downtime until 22nd Sept...
    • UKI-NORTHGRID-MAN-HEP 152521: HTC data in May and June seems higher than expected... there are duplicated records that need to deleted in the central repository...

Prerequisites for using AMS

• A valid host certificate from an IGTF Accredited CA.
• A GOCDB 'Site' entry flagged as 'Production'.
• A GOCDB 'Service' entry of the correct service type flagged as 'Production'. The following service types are used:
  • For Grid accounting use 'gLite-APEL'.
  • For Cloud accounting use 'eu.egi.cloud.accounting'.
  • For Storage accounting use 'eu.egi.storage.accounting'.
• The 'Host DN' listed in the GOCDB 'Service' entry must exactly match the certificate DN of the host used for accounting. Make sure there are no leading or trailing spaces in the 'Host DN' field.

Monitoring of the accounting data

To ensure the monitoring of the publication of the accounting data, one CE per site needs to be registered as "APEL" service endpoint.

• http://goc-accounting.grid-support.ac.uk/rss/SITE-NAME_Pub.html
• http://goc-accounting.grid-support.ac.uk/rss/SITE-NAME_Sync.html

AOB

Next meeting

Nov