Revision as of 18:30, 12 November 2021

Main

EGI.eu operations services

Support

Documentation

Tools

Activities

Performance

Technology

Catch-all Services

Resource Allocation

Security

Documentation menu:

Home •

Manuals •

Procedures •

Training •

Other •

Contact ►

For:

VO managers •

Administrators

Back to https://wiki.egi.eu/wiki/Operations_Meeting

General information

Middleware

UMD

including EOS in UMD

Preview repository

released on 2021-06-10
- Preview 2.34.0 (CentOS 7): ARC 6.12.0, CVMFS 2.8.1, xrootd 5.2.0
released on 2021-08-11
- Preview 2.35.0 (CentOS 7): APEL SSM 3.2.1, DPM/DMLite 1.15.0 and 1.15.1, frontier-squid 4.15.2, xrootd 5.3.0
We plan to stop the release of Preview since it doesn't seem to be used very much, and it is also easier to catch the last version of the products from EPEL or the product teams repos, prior the release in UMD.

Operations

ARGO/SAM

probe for checking the HTCondorCE host certificate validity deployed in production (GGUS 147386):
- checks on expiration date, CN, and CA:
  - https://argo-mon.egi.eu/nagios/cgi-bin/status.cgi?servicegroup=SERVICE_org.opensciencegrid.htcondorce&style=detail
- it is working fine (very few failures)
- to be included in the A/R profile

FedCloud

badging

Feedback from DMSU

New Known Error Database (KEDB)

The KEDB has been moved to Jira+Confluence: https://confluence.egi.eu/display/EGIKEDB/EGI+Federation+KEDB+Home

problems are tracked with Jira tickets to better follow-up their evoulution
problems can be registered by DMSU staff and EGI Operations team

Monthly Availability/Reliability

Under-performed sites in the past A/R reports with issues not yet fixed:
- AfricaArabia: https://ggus.eu/index.php?mode=ticket_info&ticket_id=154295
  - MA-01-CNRST: ARC-CE failures
- NGI_IT: https://ggus.eu/index.php?mode=ticket_info&ticket_id=150818
  - INFN-PISA: HTCondorCE failures fixed; SRM failures not yet
- NGI_PL: https://ggus.eu/index.php?mode=ticket_info&ticket_id=153659
  - TASK: in the process of replacing QCG with ARC-CE
- NGI_UA: https://ggus.eu/index.php?mode=ticket_info&ticket_id=152841
  - UA-NSCMBR: problem during the DPM update: conflict between xrootd 5 and dmlite 1.13. Unscheduled downtime due to power failure in the computing centre. NFS configuration issue affected ARC-CE. Accounting data republished using the ARC accountng functionalities.
- NGI_UK: https://ggus.eu/index.php?mode=ticket_info&ticket_id=153660
  - UKI-SOUTHGRID-SUSX: CE configuration issues; some other failures occurred.
- ROC_LA: https://ggus.eu/index.php?mode=ticket_info&ticket_id=153658
  - SUPERCOMPUTO-UNAM: some network issues
Under-performed sites after 3 consecutive months, under-performed NGIs, QoS violations: (October 2021):
- NGI_DE: https://ggus.eu/index.php?mode=ticket_info&ticket_id=154745
  - GoeGrid: relocation of the cluster to a different building on the campus and subsequent network issues; handover to new staff; problems fixed.
- NGI_IBERGRID: https://ggus.eu/index.php?mode=ticket_info&ticket_id=154750
  - UAM-LCG2
- NGI_RO: https://ggus.eu/index.php?mode=ticket_info&ticket_id=154746
  - GRIDIFIN
- NGI_PL: https://ggus.eu/index.php?mode=ticket_info&ticket_id=154747
  - PSNC: storage backend issues affecting the HPC cluster and DPM, causing also ARC-CE instability; DPM issues were fixed, working on HPC cluster
- Russia: https://ggus.eu/index.php?mode=ticket_info&ticket_id=154748
  - RU-SARFTI: ARC-CE failures, problem with hard drives, fixed
- NGI_UA: https://ggus.eu/index.php?mode=ticket_info&ticket_id=154749
  - UA-KNU: failures with IGTF metric, now fixed.

sites suspended:

Documentation

plan to decommission MediaWiki
content to be moved to different locations (confluence and https://docs.egi.eu/)
confluence space hosting policies and procedures: EGI Policies and Procedures
- EGI Federation Operations
- Change Management, Release and Deployment Management, Incident and Service Request Management, Problem Management, Information Security Management
Manuals, How-Tos, Troubleshooting, FAQs:
- huge number of material need to be reviewed and in case updated when moved to the new place
- location will be https://docs.egi.eu/providers/operations-manuals/

IPv6 readiness plans

please provide updates to the IPv6 assessment (ongoing) https://wiki.egi.eu/w/index.php?title=IPV6_Assessment
if any relevant, information will be summarised at OMB

AOB

EGI Conference 18 - 21 Oct 2021: https://indico.egi.eu/event/5464/overview

Next meeting

Dec

@@ Line 84: / Line 84: @@
 * please provide updates to the IPv6 assessment (ongoing) https://wiki.egi.eu/w/index.php?title=IPV6_Assessment
 * if any relevant, information will be summarised at  OMB
-== APEL migration from ActiveMQ to ARGO Message Service (AMS) ==
-* '''ActiveMQ dismissed on July 8th''': for security reasons it is not possible maintain it any longer.
-** Scheduled Donwtime: https://goc.egi.eu/portal/index.php?Page_Type=Downtime&id=30888 https://goc.egi.eu/portal/index.php?Page_Type=Downtime&id=30889
-* Migration insructions (HTCondorCE, Storage, and Cloud accounting): https://github.com/apel/ssm/blob/dev/migrating_to_ams.md
-* ARC 6.12.0 released, instructions:
-** http://www.nordugrid.org/arc/releases/6.12/release_notes_6.12.html
-** all the sites with ARC-CE need to update to this version
-* Recommended versions:
-** Apel Clien: 1.9.0
-** APEL SSM: 3.2.1
-* Cloud accounting campaign:
-** [https://ggus.eu/index.php?mode=ticket_search&su_hierarchy=0&status=all&date_type=creation+date&tf_radio=1&timeframe=any&from_date=05+Mar+2021&to_date=06+Mar+2021&ticket_category=all&typeofproblem=all&specattrib=none&user=paolini&keyword=APEL+migration+from+ActiveMQ+to+AMS&orderticketsby=REQUEST_ID&orderhow=desc&ticket_per_page=50&show_columns_check%5B0%5D=TICKET_TYPE&show_columns_check%5B1%5D=AFFECTED_VO&show_columns_check%5B2%5D=AFFECTED_SITE&show_columns_check%5B3%5D=PRIORITY&show_columns_check%5B4%5D=RESPONSIBLE_UNIT&show_columns_check%5B5%5D=STATUS&show_columns_check%5B6%5D=DATE_OF_CHANGE&show_columns_check%5B7%5D=SHORT_DESCRIPTION&show_columns_check%5B8%5D=SCOPE&search_submit=Search list of tickets]
-** 2 tickets (out of 21) not solved yet
-*** '''UA-BITP''' [https://ggus.eu/index.php?mode=ticket_info&ticket_id=150857 150857]: old records associated to "None" VO (due to a configuration issue) needs to be removed from the repository
-*** '''CYFRONET-CLOUD''' [https://ggus.eu/index.php?mode=ticket_info&ticket_id=150845 150845]:
-* HTCondorCE and Storage accounting campaign:
-** [https://ggus.eu/index.php?mode=ticket_search&su_hierarchy=0&status=all&date_type=creation+date&tf_radio=1&timeframe=any&from_date=15+Mar+2021&to_date=16+Mar+2021&ticket_category=all&typeofproblem=all&specattrib=none&user=paolini&keyword=APEL+migration+from+ActiveMQ+to+AMS+-+new+settings&orderticketsby=REQUEST_ID&orderhow=desc&ticket_per_page=50&show_columns_check%5B0%5D=TICKET_TYPE&show_columns_check%5B1%5D=AFFECTED_VO&show_columns_check%5B2%5D=AFFECTED_SITE&show_columns_check%5B3%5D=PRIORITY&show_columns_check%5B4%5D=RESPONSIBLE_UNIT&show_columns_check%5B5%5D=STATUS&show_columns_check%5B6%5D=DATE_OF_CHANGE&show_columns_check%5B7%5D=SHORT_DESCRIPTION&show_columns_check%5B8%5D=SCOPE&search_submit=Search list of tickets]
-** 8 tickets (out of 53) not solved yet
-*** '''ICN-UNAM''' [https://ggus.eu/index.php?mode=ticket_info&ticket_id=151374 151374]:
-*** '''INFN-COSENZA''' [https://ggus.eu/index.php?mode=ticket_info&ticket_id=151183 151183]: using an old SSM version
-*** '''INFN-NAPOLI-ATLAS''' [https://ggus.eu/index.php?mode=ticket_info&ticket_id=150976 150976]: some problems similar to https://ggus.eu/index.php?mode=ticket_info&ticket_id=154347
-*** '''INFN-PISA''' [https://ggus.eu/index.php?mode=ticket_info&ticket_id=150977 150977]:
-*** '''INFN-ROMA3''' [https://ggus.eu/index.php?mode=ticket_info&ticket_id=150978 150978]:
-*** '''INFN-TRIESTE''' [https://ggus.eu/index.php?mode=ticket_info&ticket_id=151981 151981]:
-*** '''SUPERCOMPUTO-UNAM''' [https://ggus.eu/index.php?mode=ticket_info&ticket_id=151375 151375]:
-*** '''UA-BITP''' [https://ggus.eu/index.php?mode=ticket_info&ticket_id=150991 150991]: accounting records not properly parsed
-* ARC-CE and storage accounting campaign:
-** [https://ggus.eu/index.php?mode=ticket_search&su_hierarchy=0&status=all&date_type=creation+date&tf_radio=1&timeframe=any&from_date=11+Jun+2021&to_date=12+Jun+2021&ticket_category=all&typeofproblem=all&specattrib=none&user=paolini&keyword=APEL+migration+from+ActiveMQ+to+AMS+-+ARC-CE+new+settings&orderticketsby=REQUEST_ID&orderhow=asc&ticket_per_page=120&show_columns_check%5B0%5D=TICKET_TYPE&show_columns_check%5B1%5D=AFFECTED_SITE&show_columns_check%5B2%5D=PRIORITY&show_columns_check%5B3%5D=RESPONSIBLE_UNIT&show_columns_check%5B4%5D=STATUS&show_columns_check%5B5%5D=DATE_OF_CHANGE&show_columns_check%5B6%5D=SHORT_DESCRIPTION&search_submit=Search list of tickets]
-** 11 tickets (out of 112) not solved yet
-*** '''Australia-ATLAS'''[https://ggus.eu/index.php?mode=ticket_info&ticket_id=152428 152428] and '''Australia-T2''' [https://ggus.eu/index.php?mode=ticket_info&ticket_id=152429 152429]: they stilll have ARC-CE 5.4; moving to a Cloudscheduler based compute system and will be removing the ARC-CE's in the near future
-*** '''CA-SFU-T2''' [https://ggus.eu/index.php?mode=ticket_info&ticket_id=152433 152433]: CEs updated; some errors with the benchmark which seem harmful. Duplicated records for the previous months were cleaned, it was suggested to set `apel_messages = summaries` in the arc conf file; investigating on some inconsistencies.
-*** '''IN2P3-IPNL''' [https://ggus.eu/index.php?mode=ticket_info&ticket_id=152460 152460]: CE not yet in production
-*** '''JP-KEK-CRC-02''' [https://ggus.eu/index.php?mode=ticket_info&ticket_id=152471 152471]: installed new CEs, some authz failures with sending the records...
-*** '''RU-SPbSU''' [https://ggus.eu/index.php?mode=ticket_info&ticket_id=152491 152491]: some discrepancy between local database and central repository, involved ARC developers, see [https://ggus.eu/index.php?mode=ticket_info&ticket_id=154090 GGUS 154090]...
-*** '''Taiwan-LCG2''' [https://ggus.eu/index.php?mode=ticket_info&ticket_id=152497 152497]: setting up the ARC6 server...
-*** '''TASK''' [https://ggus.eu/index.php?mode=ticket_info&ticket_id=152498 152498]:
-*** '''TW-FTT''' [https://ggus.eu/index.php?mode=ticket_info&ticket_id=152503 152503]:
-*** '''UA-MHI''' [https://ggus.eu/index.php?mode=ticket_info&ticket_id=152509 152509]: downtime until 22nd Sept...
-*** '''UKI-NORTHGRID-MAN-HEP''' [https://ggus.eu/index.php?mode=ticket_info&ticket_id=152521 152521]: HTC data in May and June seems higher than expected... there are duplicated records that need to deleted in the central repository...
-=== Prerequisites for using AMS ===
-* A valid host certificate from an IGTF Accredited CA.
-* A GOCDB 'Site' entry flagged as 'Production'.
-* A GOCDB 'Service' entry of the correct service type flagged as 'Production'. The following service types are used:
-** For Grid accounting use 'gLite-APEL'.
-** For Cloud accounting use 'eu.egi.cloud.accounting'.
-** For Storage accounting use 'eu.egi.storage.accounting'.
-* The 'Host DN' listed in the GOCDB 'Service' entry must exactly match the certificate DN of the host used for accounting. Make sure there are no leading or trailing spaces in the 'Host DN' field.
-=== Monitoring of the accounting data ===
-To ensure the monitoring of the publication of the accounting data, '''one CE per site''' needs to be registered as "'''''APEL'''''" service endpoint.
-* http://goc-accounting.grid-support.ac.uk/rss/SITE-NAME_Pub.html
-* http://goc-accounting.grid-support.ac.uk/rss/SITE-NAME_Sync.html
 = AOB  =

Difference between revisions of "Agenda-2021-11-15"