Agenda-2019-06-17

Main

EGI.eu operations services

Support

Documentation

Tools

Activities

Performance

Technology

Catch-all Services

Resource Allocation

Security

Documentation menu:

Home •

Manuals •

Procedures •

Training •

Other •

Contact ►

For:

VO managers •

Administrators

Middleware

CREAM End Of Support Notice

• have a look at this topic in the EGI Community Forum https://community.egi.eu/t/cream-end-of-support-notice/438
• EGI Conference in May has a dedicated afternoon on May 7th https://indico.egi.eu/indico/event/4431/timetable/#20190507

Preview repository

• released on 2019-05-23:
  • Preview 1.23.0 AppDB info (sl6): dCache 4.2.33, DPM 1.12.1, FTS 3.8.4, xrootd 4.9.1
  • Preview 2.23.0 AppDB info (CentOS 7): dCache 4.2.33, DPM 1.12.1, FTS 3.8.4, xrootd 4.9.1

Operations

Notifications from ARGO about the nagios probes failures

In the process of implementing the notification system in ARGO, it was introduced the following changes in GOC-DB:

• Notifications flag at the site level
• Notifications flag at the service endpoint level

In this way ARGO will retrieve from GOC-DB the information about the sites and services whom sending the email notification and the related recipients.

The logic of the notifications is the following:

Site	Service	Notify?
Y	Y	Y
Y	N	N
N	Y	N
N	N	N

If there isn't any email contact defined at the service endpoint level, it will be used the site contact. Please review your contacts.

You can enable the notifications.

NOTE: It is not mandatory for the sites

Verify configuration records

On a yearly basis, the information registered into GOC-DB need to be verified. NGIs and RCs have been asked to check them. In particular:

1. NGI managers should review the people registered and the roles assigned to them, and in particular check the following information:
   • E-Mail
   • ROD E-Mail
   • Security E-Mail

NGI Managers should also review the status of the "not certified" RCs, in according to the RC Status Workflow;

1. RCs administrators should review the people registered and the roles assigned to them, and in particular check the following information:
   • E-Mail
   • telephone numbers
   • CSIRT E-Mail

RC administrators should also review the information related to the registered service endpoints.

The process should be completed by June 21th.

Tickets to track the process are listed here https://wiki.egi.eu/wiki/Verify_Configuration_Records#2019-05

• 12 SOLVED, 4 In progress (out of 31)

Monthly Availability/Reliability

• Under-performed sites in the past A/R reports with issues not yet fixed:
  • AfricaArabia: https://ggus.eu/index.php?mode=ticket_info&ticket_id=139056
    • ZA-WITS-CORE: authentication on CE and SE fixed, site-BDII, CE and SE information fixed; A/R figures are improving.
  • AsiaPacific: https://ggus.eu/index.php?mode=ticket_info&ticket_id=141089
    • IR-IPM-HEP: installed new host certificates, figures are improving...
  • NGI_AEGIS: https://ggus.eu/index.php?mode=ticket_info&ticket_id=140555
    • AEGIS01-IPB-SCL: CE host certificate renewd; SRM not yet published in the BDII; CAs package to update
  • NGI_GRNET: https://ggus.eu/index.php?mode=ticket_info&ticket_id=140556
    • GR-12-TEIKAV: SRM instable; some investigations are on-going
  • NGI_PL: https://ggus.eu/index.php?mode=ticket_info&ticket_id=140557
    • TASK: still problems: SRM failures; QCG bug;
  • NGI_PL: https://ggus.eu/index.php?mode=ticket_info&ticket_id=141090
    • CYFRONET-CLOUD
    • CYFRONET-PROMETHEUS
    • PSNC
  • NGI_UA: https://ggus.eu/index.php?mode=ticket_info&ticket_id=141096
    • UA-IMBG
• Under-performed sites after 3 consecutive months, under-performed NGIs, QoS violations: (May 2019):
  • NGI_AEGIS: https://ggus.eu/index.php?mode=ticket_info&ticket_id=141596
    • AEGIS03-ELEF-LEDA
    • AEGIS11-MISANU
  • NGI_DE: https://ggus.eu/index.php?mode=ticket_info&ticket_id=141598
    • LRZ-LMU: problem with ARC-CE IGTF probe after moving all the jobs to singularity: the X509_CERT_DIR was not mounted in the container.
  • NGI_HR: https://ggus.eu/index.php?mode=ticket_info&ticket_id=141597
    • egee.irb.hr

suspended sites:

IPv6 readiness plans

• please provide updates to the IPv6 assessment (ongoing) https://wiki.egi.eu/w/index.php?title=IPV6_Assessment information will be summarised for the OMB

LCGDM end of support and migration to / enabling of DOME

• The DPM team has agreed to extend support for security updates for the DPM legacy functionality until 30 September 2019. However, affected service providers should still plan to disable legacy mode well before this date.
  • more details: https://wiki.egi.eu/wiki/DPM_End_of_legacy-mode_support
• since DPM 1.10.3 release, it is possible enabling the non-legacy mode DOME (Disk operations Management Engine, see documentation)
  • latest DPM release 1.12.1
• EGI Operations sent a broadcast to site-admins(first bunch and second bunch) and VO managers with a survey to fill in:
  • site-admins: to know the upgrade plans: https://www.surveymonkey.com/r/2PBZSNB
  • VO Managers: to know any barriers in moving away from using SRM protocol https://www.surveymonkey.com/r/2ZMZJVG

• Deployment statistics: 93 sites (108 servers)

$ ldapsearch -x -LLL -H ldap://bdii.marie.hellasgrid.gr:2170 -b "Mds-Vo-Name=local,o=grid" '(&(objectClass=GlueSE)(GlueSEImplementationName=DPM))' GlueSEImplementationVersion | grep -i ^glue | sort | uniq -c
    24 GlueSEImplementationVersion: 1.10.0
    34 GlueSEImplementationVersion: 1.12.0
     1 GlueSEImplementationVersion: 1.12.1
    16 GlueSEImplementationVersion: 1.8.10
     2 GlueSEImplementationVersion: 1.8.11
     2 GlueSEImplementationVersion: 1.8.7
     4 GlueSEImplementationVersion: 1.8.8
     5 GlueSEImplementationVersion: 1.8.9
    20 GlueSEImplementationVersion: 1.9.0

• all the sites with older DPM versions than 1.12 are suggested to upgrade to the latest DPM version , following the guide DPM upgrade (chapter 1 Upgrade to DPM 1.10.0 "Legacy Flavour" and chapter 2 Upgrade to DPM 1.10.0 "Dome Flavour")
  • DOME and the old LCGDM (srm protocol) will coexist
• the upgrade will be followed-up mainly by WLCG
• EGI Operations will liaise with the non-WLCG sites

HTCondorCE integration

Link to procedure: https://wiki.egi.eu/wiki/PROC19

GGUS ticket: https://ggus.eu/index.php?mode=ticket_info&ticket_id=139377

Steps status:

• (Completed) Underpinning Agreement:
• (Completed) Configuration management:
  • Service type already present: org.opensciencegrid.htcondorce
• (To do) OPS Dashboard:
  • To do when Conf Mgmt and Monitoring are completed.
• (in progress) Information System: the info-provider is already available: https://github.com/opensciencegrid/htcondor-ce/tree/master/contrib/bdii
• (in progress) Monitoring: some probes are already available that submit jobs to CondorCe, CREAM and ARC-CE:
  • https://gitlab.cern.ch/etf/jess
  • ticket to ARGO to deploy the probes on the test instance: https://ggus.eu/index.php?mode=ticket_info&ticket_id=141177
• (Completed) Support: the support unit is https://wiki.egi.eu/wiki/GGUS:HTCondor-CE_FAQ
• (in progress) Accounting:
  • Some work has been done: https://twiki.cern.ch/twiki/bin/view/LCG/HtCondorCeAccounting
  • ticket https://ggus.eu/index.php?mode=ticket_info&ticket_id=139403 to track the progress on this topic
    • APEL 1.8.0-1 adds support for a cputmult scaling factor.
    • Liverpool site managed to publish accounting records
    • To involve more (UK) sites in the testing
    • INFN-T1 joined the tests
• (in progress) Documentation
• (Completed) Security
  • Condor team filled in the questionnaire and sent it to Linda
• (in progress) UMD
  • suggested to act in advance because the functional tests could take a lot of time
  • Ticket for the inclusion: https://ggus.eu/index.php?mode=ticket_info&ticket_id=139764
    • HTCondor-CE is released with HTCondor in a version without the dependencies on other OSG software. Moreover, HTCondor-CE depends on HTCondor (though not for use as a batch scheduler), so both packages will need to be distributed.
  • Asked for any automated deployment (ansible, puppets or other) configuration files supported by HTcondor team
    • It is available a copy of the ansible files that OSG uses to configure HTCondor-CE on their internal testbed machines. They can provide help for a non-OSG version of the files
  • Provided information for the UMD card: https://wiki.egi.eu/wiki/UMD_products_ID_cards

webdav probes in CRITICAL profile

In June the webdav probes have been included in the ARGO_MON_CRITICAL profile, so the results of these probes will be taken into account for the A/R figures: https://ggus.eu/index.php?mode=ticket_info&ticket_id=141173

• webdav endpoints registered in GOC-DB: https://goc.egi.eu/gocdbpi/public/?method=get_service&&service_type=webdav
• link to nagios results: https://argo-mon.egi.eu/nagios/cgi-bin/status.cgi?servicegroup=SERVICE_webdav&style=detail

Still to fix:

• NGI_HR: egee.irb.hr https://ggus.eu/index.php?mode=ticket_info&ticket_id=134458
  • webdav protocol disabled from the SE and removed from the information system
  • to delete the service endpoint registered on GOCDB: https://goc.egi.eu/portal/index.php?Page_Type=Site&id=475
• AEGIS01-IPB-SCL https://ggus.eu/index.php?mode=ticket_info&ticket_id=139929 : SRM currently unreachable

Storage accounting deployment

During the September 2017 meeting, OMB has approved the full-scale deployment of storage accounting. The APEL team tested it with a group of early adopters sites, and the results prove that storage accounting is now production-ready.

Storage accounting is currently supported only for the DPM and dCache storage elements therefore only the resource centres deploying these kind of storage elements are requested to publish storage accounting data.

In order to properly install and configure the storage accounting scripts, please follow the instructions reported in the wiki: https://wiki.egi.eu/wiki/APEL/Storage

IMPORTANT: be sure to have installed the star-accounting.py script v1.0.4 (http://svnweb.cern.ch/world/wsvn/lcgdm/lcg-dm/trunk/scripts/StAR-accounting/star-accounting.py)

After setting up a daily cron job and running the accounting software, look for your data in the Accounting Portal: http://goc-accounting.grid-support.ac.uk/storagetest/storagesitesystems.html. If it does not appear within 24 hours, or there are other errors, please open a GGUS ticket to APEL who will help debug the process.

Test portal: http://accounting-devel.egi.eu/storage.php

IMPORTANT: Do not encrypt the storage records with your host certificate, please comment out the “server_cert” variable in sender.cfg

List of sites already publishing and of tickets opened is reported here.

Several sites are not publishing the storage accounting data yet. NGIs please follow-up with the sites the configuration of the script in order to speed-up the process.

• AsiaPacific: TW-NCUHEP https://ggus.eu/index.php?mode=ticket_info&ticket_id=131426 (Site Suspended)
• NGI_AEGIS: AEGIS04-KG https://ggus.eu/index.php?mode=ticket_info&ticket_id=131431 (Site Suspended)
• NGI_CH:
  • CSCS-LCG2 https://ggus.eu/index.php?mode=ticket_info&ticket_id=131432 (dcache instructions has been updated, configuring STAR should be easier)
• NGI_DE:
  • DESY-HH https://ggus.eu/index.php?mode=ticket_info&ticket_id=131439 (postponed)
  • GoeGrid https://ggus.eu/index.php?mode=ticket_info&ticket_id=131443
• NGI_IT:
  • INFN-ROMA1-CMS https://ggus.eu/index.php?mode=ticket_info&ticket_id=131482 (new site-admin will work on it)

AOB

Next meeting

July https://indico.egi.eu/indico/event/4709/