Agenda-2021-03-08

Main

EGI.eu operations services

Support

Documentation

Tools

Activities

Performance

Technology

Catch-all Services

Resource Allocation

Security

Documentation menu:

Home •

Manuals •

Procedures •

Training •

Other •

Contact ►

For:

VO managers •

Administrators

Back to https://wiki.egi.eu/wiki/Operations_Meeting

Middleware

Software repository maintenance

• unscheduled downtime from 03-Mar-21 17:00:00 (UTC) to 05-Mar-21 14:30:06 (UTC)
• Broadcasts announcing the unexpected downtime and the recovery
• Migration from IASA to IBERGRID
• Warning downtime until 09-Mar-21 14:30:00 (UTC)
• repository is completely online; the related website not yet (only some very simple html pages)
• more details of the incident will be circulated soon

UMD

• CentOS8 discussion still ongoing
• migration of Software Provisioning infrastructure to IBERGRID still ongoing
  • in particular, administration portal used for release creation done successfully
• still planning next release, delay due to repo migration

Preview repository

• released on 2021-02-22
  • Preview 2.31.0 AppDB info (CentOS 7): APEL-SSM 3.1.1, ARC 6.10.1, CVMFS 2.8.0 and egi-cvmfs-3-1.13, davix 0.7.6, dCache 5.2.38, gfal2 2.18.2

Operations

ARGO/SAM

• Site-BDII metrics org.bdii.Entries and org.bdii.Freshness removed from ARGO_MON_CRITICAL profile
  • the metrics are still kept in the ARGO_MON_OPERATORS profiles
  • it is still an important service to support infrastructure oversight activities
• HTCondor-CE probes
  • deployed on secmon and pakiti: GGUS 150006
  • working on the probe for the host certificate validity check: GGUS 147386
    • With 8.9.12 installed (expected the week of Mar 15), you should be able to query remote HTCondor-CEs for their host certificate using the following:

$ python -c 'import htcondor; ad = htcondor.Collector("collector2.opensciencegrid.org:9619").locate(htcondor.DaemonTypes.Schedd, "hosted-ce10.opensciencegrid.org"); print htcondor.SecMan().ping(ad, "READ")["ServerPublicCert"]' | openssl x509 -noout -subject -enddate
subject= /CN=hosted-ce10.opensciencegrid.org
notAfter=Apr 26 12:26:42 2021 GMT

• CREAM-CE metrics removed from ARGO_MON, ARGO_MON_OPERATIONS and ARGO_MON_CRITICAL (GGUS 149778)
  • emi.cream.CREAMCE*
  • eu.egi.CREAM*

FedCloud

• cASO upgade campaign on OpenStack sites
• badging

Monthly Availability/Reliability

• Under-performed sites in the past A/R reports with issues not yet fixed:
  • AsiaPacific: https://ggus.eu/index.php?mode=ticket_info&ticket_id=150109
    • INDIACMS-TIFR failures with HTCondor-CE and webdav; additional failures with SRM tests
    • KR-KNU-T3: migration from CREAM-CE to HTCondor-CE
  • NGI_HR: https://ggus.eu/index.php?mode=ticket_info&ticket_id=148518
    • egee.irb.hr: in the process of a major upgrade from CentOS 6 to CentOS 7, some delays.
  • NGI_UA: https://ggus.eu/index.php?mode=ticket_info&ticket_id=150469
    • UA-MHI: upgrade to CentOS7 and ARC 6, tests are ok
  • ROC_LA: https://ggus.eu/index.php?mode=ticket_info&ticket_id=148515
    • ATLAND: ARC-CE misconfiguration
  • ROC_LA: https://ggus.eu/index.php?mode=ticket_info&ticket_id=148956
    • CBPF: SRM failures due to information not properly published. Physical access to facilities restricted due to COVID measures; planned a DPM update.
  • ROC_LA https://ggus.eu/index.php?mode=ticket_info&ticket_id=149355
    • SUPERCOMPUTO-UNAM: scheduled a downtime for upgrading the site.
• Under-performed sites after 3 consecutive months, under-performed NGIs, QoS violations: (February 2021):
  • NGI_DE: https://ggus.eu/index.php?mode=ticket_info&ticket_id=150816
    • GoeGrid
  • NGI_IT: https://ggus.eu/index.php?mode=ticket_info&ticket_id=150818
    • INFN-PISA
  • ROC_LA: https://ggus.eu/index.php?mode=ticket_info&ticket_id=150817
    • ICN-UNAM: replacing CREAM-CE
• sites suspended:
  • GARR-01-DIR (NGI_IT), TW-NCUHEP (AsiaPacific)

IPv6 readiness plans

• please provide updates to the IPv6 assessment (ongoing) https://wiki.egi.eu/w/index.php?title=IPV6_Assessment
• if any relevant, information will be summarised at OMB

APEL migration from ActiveMQ to ARGO Message Service (AMS)

• Migration insructions: https://github.com/apel/ssm/blob/dev/migrating_to_ams.md
• ActiveMQ is going to be dismissed at the end of March with the end of EOSC-hub
  • https://ggus.eu/index.php?mode=ticket_info&ticket_id=140318
• Currently an issue with apel client prevent SSM to send properly the records through AMS
  • it doesn't affect cloud and storage accounting
  • ARC-CE might not work if using an old bundled version of SSM - but new ARC versions may work if set to use standalone SSM
  • With CondorCE it may work, we will find some sites to test it
  • by mid-March a fix will be released; then the sites with ARC-CE/HTCondorCE can implement the change
• starting the migration with FedCloud sites
  • list of tickets

Feedback from NGI_FRANCE

On the Cloud infra, several tickets have been open to switch to the new messaging system. It would nice to have the following RPMs made available from CMD repo, and not only from UMD:

• apel-ssm-2.4.1-1.el7.noarch
• python-argo-ams-library-0.5.1-1.el7.noarch

In addition, many Cloud sites are now using OpenStack Stein or newer. These version are provided with python-daemon = 2.2.3-1.el7. It conflicts with the requirement of apel-ssm ( python-daemon < 2.2.0)

Feedback from URT:

• there is new apel-ssm 3.0.0 version under untested repo and this new version solves the dependency issue of requirement python-daemon <= 2.2.0.
• there is also the new python-argo-ams 0.54 library

ARC-CE probe failing due to UMD repositories being down

• The unavailability of UMD repository caused a failure with the ARC-CE IGTF probes (org.nordugrid.ARC-CE-result-ops)
  • https://ggus.eu/index.php?mode=ticket_info&ticket_id=150827

Job terminated as Failed. - Failed in data staging: Failed checking source replica http://repository.egi.eu:80/sw/production/cas/1/current/meta/ca-policy-egi-core.list: Failed to obtain information about file: Failed to connect to repository.egi.eu(IPv4):80 - JID: gsiftp://alex4.nipne.ro:2811/jobs/yq0NDmskJcynuvw3Vp3UrRNqABFKDmABFKDm8hJKDmABFKDmxx7PPm 

• Asked the ARC-CE developers to remove this dependency from the probe:
  • https://ggus.eu/index.php?mode=ticket_info&ticket_id=150833
• It will be asked a recomputation to exclude these failures from the A/R figures

CREAM-CE Decommission

• End of Security Updates and Support: 31st Dec 2020
  • Original broadcast: https://operations-portal.egi.eu/broadcast/archive/2293
• Decommissioning deadline: 31st Jan 2021
• PROC16 Decommission of unsupported software
• Decommissioning start date: Oct 1st 2020
  • a probe detecting CREAM-CE endpoints will be run, returning WARNING status
  • GGUS ticket: https://ggus.eu/index.php?mode=ticket_info&ticket_id=148715
  • eu.egi.sec.CREAMCE
• Nov 1st: probe returns CRITICAL status, alarms created on the ROD dashboard, ROD teams start to create tickets
  • https://ggus.eu/index.php?mode=ticket_info&ticket_id=149312
• 1st Feb 2021: EGI Ops will start chasing the sites still providing CREAM-CE endpoints
  • By this time service end-points which couldn't be upgraded should be put into downtime by site admin or ROD
• 1st March 2021: Sites still deploying unsupported service endpoints risk suspension, unless documented technical reasons prevent a Site Admin from updating these endpoints.
• Tickets opened: 49
  • link to the list
• Please note that at least one CE endpoint should be associated to the APEL service type in order to monitor the publication of the accounting data, as explained here
  • If the CE you are going to remove was also registered as APEL service type, do not forget to move the APEL service type to a different CE endpoint.

VOMS upgrade to CentOS 7

• VOMS for CentOS 7 released Nov 23rd with UMD 4.12.13
  • VOMS Admin 3.8.0, VOMS Server 2.0.15
• VOMS endpoints registered on GOCDB as production and monitored: 41
  • Provided by 33 sites
• list of ticket opened: GGUS
• the VOMS servers need to be published in the BDII in order to easily collect the deployed version

AOB

Next meeting

12th Apr 2021