Alert.png The wiki is deprecated and due to be decommissioned by the end of September 2022.
The content is being migrated to other supports, new updates will be ignored and lost.
If needed you can get in touch with EGI SDIS team using operations @ egi.eu.

Difference between revisions of "Agenda-2019-09-09"

From EGIWiki
Jump to navigation Jump to search
(Undo revision 102407 by Spinoso (talk))
 
(30 intermediate revisions by 2 users not shown)
Line 1: Line 1:
{{Template:Op menubar}} {{Template:Doc_menubar}} {{TOC_right}}  
{{Template:Op menubar}} {{Template:Doc_menubar}} {{TOC_right}}  
[[Category:Grid Operations Meetings]]
[[Category:Grid Operations Meetings]]
TO UPDATE
 
= General information =
= General information =


= Middleware  =
= Middleware  =


== CREAM End Of Support Notice ==  
== UMD 4.8.5 ==
 
* "DPM-dedicated" release to support migration to 1.13.0 (see [https://wiki.egi.eu/wiki/Agenda-2019-09-09#LCGDM_end_of_support_and_migration_to_.2F_enabling_of_DOME dedicated section])
* have a look at this topic in the EGI Community Forum https://community.egi.eu/t/cream-end-of-support-notice/438
* release candidate ready, integration tests in progress, TBR very soon
* EGI Conference in May has a dedicated afternoon on May 7th https://indico.egi.eu/indico/event/4431/timetable/#20190507


== Preview repository  ==
== Preview repository  ==
Line 24: Line 23:


== Feedback from DMSU  ==
== Feedback from DMSU  ==
IMPORTANT
Known Error Database: https://wiki.egi.eu/wiki/KEDB


*[https://ggus.eu/index.php?mode=ticket_info&ticket_id=142022 GGUS 142022]: Decision on preferred storage access protocols, required monitoring probes (assigned to Operations, on hold)
Tickets handled by DMSU:
*'''SRM argo/nagios probes''' [https://ggus.eu/index.php?mode=ticket_info&ticket_id=142022 GGUS 142022]: Decision on preferred storage access protocols, required monitoring probes (assigned to Operations, on hold)
** SRM probes need to be updated since they are still using the old lcg-utils which have been replaced by GFAL2
** considering the SRM end of support for DPM, the reference protocol will be HTTP/webdav
*'''CREAM cannot initialize connection to BLAH BLParserClient''' [https://ggus.eu/index.php?mode=ticket_info&ticket_id=142621 GGUS #142621]:
**CREAM/BLAH supports only single version of HTCondor (8.6.3) currently, does not work with 8.8.4. No fixes for HTCondor 8.6 line are expected anymore since 8.8 is more than six months old already. CREAM team may be able to respond in case of actual emergency (required changes could be very small) but has not commented yet.
*'''BDII udpate & base64 encoded ldif entries''' [https://ggus.eu/index.php?mode=ticket_info&ticket_id=142928 GGUS #142928]:
**bdii-update does not handle base64-encoded values in LDIF correctly, non-ASCCII characters or passwords invoking mandatory base64 use are not present in BDII information normally, but bdii-update should handle it anyway (developers have not commented yet).
*'''CREAM-CE at CentOS 7: could not create connection to database server''' [https://ggus.eu/index.php?mode=ticket_info&ticket_id=142425 GGUS #142425] '''(SOLVED)''':
**Puppet recipe for CREAM-CE requests presence of DB server, but does not handle its configuration fully (even the parameters set by Puppet on the CREAM side), manual checking and adjustment may be necessary
ERROR org.glite.ce.commonj.db.DatasourceManager
- Cannot create PoolableConnectionFactory
Could not create connection to database server.
Attempted reconnect 3 times. Giving up.


SOME SITES MAY FIND IT IMPORTANT
**solved by setting in /etc/glite-ce-cream/cream-config.xml:
url="jdbc:mysql://localhost:3306/..."
(instead of url="jdbc:mysql://lgdce01.jinr.ru:3306/...")


*[https://ggus.eu/index.php?mode=ticket_info&ticket_id=142621 GGUS #142621]: CREAM/BLAH supports only single version of HTCondor (8.6.3) currently, does not work with 8.8.4. No fixes for HTCondor 8.6 line are expected anymore since 8.8 is more than six months old already. CREAM team may be able to respond in case of actual emergency (required changes could be very small) but has not commented yet.
*'''changes in VOMS server certificate''' [https://ggus.eu/index.php?mode=ticket_info&ticket_id=142627 GGUS #142627], [https://ggus.eu/index.php?mode=ticket_info&ticket_id=142820 GGUS #142820]:
 
**whene there is a change of DN of VOMS server's certificate, all the clients need to update their configuration
*[https://ggus.eu/index.php?mode=ticket_info&ticket_id=142928 GGUS #142928]: bdii-update does not handle base64-encoded values in LDIF correctly, non-ASCCII characters or passwords invoking mandatory base64 use are not present in BDII information normally, but bdii-update should handle it anyway (developers have not commented yet).
*'''OpenStack-VM-ops probe failed since automatic image deletion''' [https://ggus.eu/index.php?mode=ticket_info&ticket_id=142905 GGUS #142905]:
 
**AppDB entry of VM image used by monitoring probe has expired and manual refresh was necessary, the probe does not check (and issue warning in advance) for this itself (regular maintaineer of the image was on vacation), likely repeat yearly
SW/PROCEDURE IMPROVEMENT POSSIBLE
 
*[https://ggus.eu/index.php?mode=ticket_info&ticket_id=142425 GGUS #142425]: Puppet recipe for CREAM-CE requests presence of DB server, but does not handle its configuration fully (even the parameters set by Puppet on the CREAM side), manual checking and adjustment may be necessary.
 
*[https://ggus.eu/index.php?mode=ticket_info&ticket_id=142627 GGUS #142627], [https://ggus.eu/index.php?mode=ticket_info&ticket_id=142820 GGUS #142820]: Change of DN of VOMS server's certificate (could be forced by CA upon renewal) is too complicated/poorly supported by established procedures.
 
*[https://ggus.eu/index.php?mode=ticket_info&ticket_id=142905 GGUS #142905] AppDB entry of VM image used by monitoring probe has expired and manual refresh was necessary, the probe does not check (and issue warning in advance) for this itself (regular maintaineer of the image was on vacation), likely repeat yearly


== Notifications from ARGO about the nagios probes failures ==
== Notifications from ARGO about the nagios probes failures ==
Line 111: Line 118:
== IPv6 readiness plans  ==
== IPv6 readiness plans  ==


* please provide updates to the IPv6 assessment (ongoing) https://wiki.egi.eu/w/index.php?title=IPV6_Assessment information will be summarised for the OMB
* INFORMATION ARE 6 MONTHS OLD, '''please provide updates''' to the IPv6 assessment (ongoing) https://wiki.egi.eu/w/index.php?title=IPV6_Assessment  
* if any relevant, information will be summarised at OMB


== LCGDM end of support and migration to / enabling of DOME ==
== LCGDM end of support and migration to / enabling of DOME ==
Line 123: Line 131:
**VO Managers: to know any barriers in moving away from using SRM protocol https://www.surveymonkey.com/r/2ZMZJVG
**VO Managers: to know any barriers in moving away from using SRM protocol https://www.surveymonkey.com/r/2ZMZJVG


* '''Deployment statistics: 93 sites (108 servers)'''
* '''Deployment statistics: 85 sites (100 servers)'''
  $ ldapsearch -x -LLL -H ldap://bdii.marie.hellasgrid.gr:2170 -b "Mds-Vo-Name=local,o=grid" '(&(objectClass=GlueSE)(GlueSEImplementationName=DPM))' GlueSEImplementationVersion | grep -i ^glue | sort | uniq -c
  $ ldapsearch -x -LLL -H ldap://bdii.marie.hellasgrid.gr:2170 -b "Mds-Vo-Name=local,o=grid" '(&(objectClass=GlueSE)(GlueSEImplementationName=DPM))' GlueSEImplementationVersion | grep -i ^glue | sort | uniq -c
     24 GlueSEImplementationVersion: 1.10.0
     14 GlueSEImplementationVersion: 1.10.0
     34 GlueSEImplementationVersion: 1.12.0
     21 GlueSEImplementationVersion: 1.12.0
       1 GlueSEImplementationVersion: 1.12.1
       2 GlueSEImplementationVersion: 1.12.1
     16 GlueSEImplementationVersion: 1.8.10
     22 GlueSEImplementationVersion: 1.13.0
    17 GlueSEImplementationVersion: 1.8.10
       2 GlueSEImplementationVersion: 1.8.11
       2 GlueSEImplementationVersion: 1.8.11
       2 GlueSEImplementationVersion: 1.8.7
       2 GlueSEImplementationVersion: 1.8.7
       4 GlueSEImplementationVersion: 1.8.8
       4 GlueSEImplementationVersion: 1.8.8
       5 GlueSEImplementationVersion: 1.8.9
       3 GlueSEImplementationVersion: 1.8.9
     20 GlueSEImplementationVersion: 1.9.0
     14 GlueSEImplementationVersion: 1.9.0


* all the sites with older DPM versions than 1.12 are suggested to upgrade to the latest DPM version , following the guide [https://twiki.cern.ch/twiki/bin/view/DPM/DpmSetup1100 DPM upgrade] (chapter 1 Upgrade to DPM 1.10.0 "Legacy Flavour" and chapter 2 Upgrade to DPM 1.10.0 "Dome Flavour")
* all the sites with older DPM versions than 1.12 are suggested to upgrade to the latest DPM version , following the guide [https://twiki.cern.ch/twiki/bin/view/DPM/DpmSetup1100 DPM upgrade] (chapter 1 Upgrade to DPM 1.10.0 "Legacy Flavour" and chapter 2 Upgrade to DPM 1.10.0 "Dome Flavour")
Line 158: Line 167:
**To do when Conf Mgmt and Monitoring are completed.
**To do when Conf Mgmt and Monitoring are completed.
*(in progress) Information System: the info-provider is already available: https://github.com/opensciencegrid/htcondor-ce/tree/master/contrib/bdii  
*(in progress) Information System: the info-provider is already available: https://github.com/opensciencegrid/htcondor-ce/tree/master/contrib/bdii  
**to test during the staged rollout
*(in progress) Monitoring: some probes are already available that submit jobs to CondorCe, CREAM and ARC-CE:
*(in progress) Monitoring: some probes are already available that submit jobs to CondorCe, CREAM and ARC-CE:
**https://gitlab.cern.ch/etf/jess
**https://gitlab.cern.ch/etf/jess
Line 169: Line 179:
***To involve more (UK) sites in the testing
***To involve more (UK) sites in the testing
***INFN-T1 joined the tests
***INFN-T1 joined the tests
*(in progress) Documentation
***problems should have been fixed with [https://github.com/apel/apel/releases APEL 1.8.1]
*(Completed) Documentation
*('''Completed''') Security
*('''Completed''') Security
**Condor team filled in the questionnaire and sent it to Linda
**Condor team filled in the questionnaire and sent it to Linda
Line 179: Line 190:
***It is available a copy of the ansible files that OSG uses to configure HTCondor-CE on their internal testbed machines. They can provide help for a non-OSG version of the files
***It is available a copy of the ansible files that OSG uses to configure HTCondor-CE on their internal testbed machines. They can provide help for a non-OSG version of the files
**Provided information for the UMD card: https://wiki.egi.eu/wiki/UMD_products_ID_cards
**Provided information for the UMD card: https://wiki.egi.eu/wiki/UMD_products_ID_cards
== Storage accounting deployment  ==
During the [https://indico.egi.eu/indico/event/3241/ September 2017 meeting], OMB has approved the full-scale deployment of storage accounting. The APEL team [[Storage accounting testing|tested it with a group of early adopters sites]], and the results prove that storage accounting is now production-ready.
Storage accounting is currently supported '''only for the DPM and dCache storage elements''' therefore only the resource centres deploying these kind of storage elements are requested to publish storage accounting data.
In order to properly install and configure the storage accounting scripts, please follow the instructions reported in the wiki: https://wiki.egi.eu/wiki/APEL/Storage
'''IMPORTANT''': be sure to have installed the star-accounting.py script v1.0.4 (http://svnweb.cern.ch/world/wsvn/lcgdm/lcg-dm/trunk/scripts/StAR-accounting/star-accounting.py)
After setting up a daily cron job and running the accounting software, look for your data in the Accounting Portal: http://goc-accounting.grid-support.ac.uk/storagetest/storagesitesystems.html. If it does not appear within 24 hours, or there are other errors, please open a GGUS ticket to APEL who will help debug the process.
'''Test portal''': http://accounting-devel.egi.eu/storage.php
'''IMPORTANT''': Do not encrypt the storage records with your host certificate, please comment out the “server_cert” variable in sender.cfg
'''List of sites already publishing and of tickets opened is [[Storage accounting deployment|reported here]]'''.
Several sites are not publishing the storage accounting data yet. '''NGIs please follow-up with the sites the configuration of the script in order to speed-up the process.'''
*AsiaPacific: TW-NCUHEP https://ggus.eu/index.php?mode=ticket_info&ticket_id=131426 (Site Suspended)
*NGI_AEGIS: AEGIS04-KG https://ggus.eu/index.php?mode=ticket_info&ticket_id=131431 (Site Suspended)
*NGI_CH:
**CSCS-LCG2 https://ggus.eu/index.php?mode=ticket_info&ticket_id=131432 (dcache instructions has been updated, configuring STAR should be easier)
*NGI_DE:
**DESY-HH https://ggus.eu/index.php?mode=ticket_info&ticket_id=131439 (postponed)
**GoeGrid https://ggus.eu/index.php?mode=ticket_info&ticket_id=131443
*NGI_IT:
**INFN-ROMA1-CMS https://ggus.eu/index.php?mode=ticket_info&ticket_id=131482 (new site-admin will work on it)


= AOB  =
= AOB  =
Line 214: Line 195:
== Next meeting  ==
== Next meeting  ==


July https://indico.egi.eu/indico/event/4709/
October https://indico.egi.eu/indico/event/4782/

Latest revision as of 13:00, 9 September 2019

Main EGI.eu operations services Support Documentation Tools Activities Performance Technology Catch-all Services Resource Allocation Security


Documentation menu: Home Manuals Procedures Training Other Contact For: VO managers Administrators


General information

Middleware

UMD 4.8.5

  • "DPM-dedicated" release to support migration to 1.13.0 (see dedicated section)
  • release candidate ready, integration tests in progress, TBR very soon

Preview repository

  • released on 2019-08-16
    • Preview 1.24.0 AppDB info (sl6): APEL Client/Server 1.8.1, APEL-SSM 2.4.0, ARC 6.1.0, davix 0.7.4, dCache 4.2.40, DMLite/DPM 1.13.1, Dynafed 1.5.0, frontier-squid 4.8.1, gfal2 2.16.3
    • Preview 2.24.0 AppDB info (CentOS 7): APEL Client/Server 1.8.1, APEL-SSM 2.4.0, ARC 6.1.0, davix 0.7.4, dCache 4.2.40, DMLite/DPM 1.13.1, Dynafed 1.5.0, frontier-squid 4.8.1, gfal2 2.16.3

Operations

ARGO/SAM

FedCloud

Feedback from DMSU

Known Error Database: https://wiki.egi.eu/wiki/KEDB

Tickets handled by DMSU:

  • SRM argo/nagios probes GGUS 142022: Decision on preferred storage access protocols, required monitoring probes (assigned to Operations, on hold)
    • SRM probes need to be updated since they are still using the old lcg-utils which have been replaced by GFAL2
    • considering the SRM end of support for DPM, the reference protocol will be HTTP/webdav
  • CREAM cannot initialize connection to BLAH BLParserClient GGUS #142621:
    • CREAM/BLAH supports only single version of HTCondor (8.6.3) currently, does not work with 8.8.4. No fixes for HTCondor 8.6 line are expected anymore since 8.8 is more than six months old already. CREAM team may be able to respond in case of actual emergency (required changes could be very small) but has not commented yet.
  • BDII udpate & base64 encoded ldif entries GGUS #142928:
    • bdii-update does not handle base64-encoded values in LDIF correctly, non-ASCCII characters or passwords invoking mandatory base64 use are not present in BDII information normally, but bdii-update should handle it anyway (developers have not commented yet).
  • CREAM-CE at CentOS 7: could not create connection to database server GGUS #142425 (SOLVED):
    • Puppet recipe for CREAM-CE requests presence of DB server, but does not handle its configuration fully (even the parameters set by Puppet on the CREAM side), manual checking and adjustment may be necessary
ERROR org.glite.ce.commonj.db.DatasourceManager
- Cannot create PoolableConnectionFactory
Could not create connection to database server.
Attempted reconnect 3 times. Giving up.
    • solved by setting in /etc/glite-ce-cream/cream-config.xml:
url="jdbc:mysql://localhost:3306/..."
(instead of url="jdbc:mysql://lgdce01.jinr.ru:3306/...")
  • changes in VOMS server certificate GGUS #142627, GGUS #142820:
    • whene there is a change of DN of VOMS server's certificate, all the clients need to update their configuration
  • OpenStack-VM-ops probe failed since automatic image deletion GGUS #142905:
    • AppDB entry of VM image used by monitoring probe has expired and manual refresh was necessary, the probe does not check (and issue warning in advance) for this itself (regular maintaineer of the image was on vacation), likely repeat yearly

Notifications from ARGO about the nagios probes failures

In the process of implementing the notification system in ARGO, it was introduced the following changes in GOC-DB:

  • Notifications flag at the site level
  • Notifications flag at the service endpoint level

In this way ARGO will retrieve from GOC-DB the information about the sites and services whom sending the email notification and the related recipients.

The logic of the notifications is the following:

Site Service Notify?
Y Y Y
Y N N
N Y N
N N N

If there isn't any email contact defined at the service endpoint level, it will be used the site contact. Please review your contacts.

You can enable the notifications.

NOTE: It is not mandatory for the sites

Monthly Availability/Reliability

suspended sites: AEGIS11-MISANU (NGI_AEGIS), IMBG (NGI_UA),

IPv6 readiness plans

LCGDM end of support and migration to / enabling of DOME

  • Deployment statistics: 85 sites (100 servers)
$ ldapsearch -x -LLL -H ldap://bdii.marie.hellasgrid.gr:2170 -b "Mds-Vo-Name=local,o=grid" '(&(objectClass=GlueSE)(GlueSEImplementationName=DPM))' GlueSEImplementationVersion | grep -i ^glue | sort | uniq -c
    14 GlueSEImplementationVersion: 1.10.0
    21 GlueSEImplementationVersion: 1.12.0
     2 GlueSEImplementationVersion: 1.12.1
    22 GlueSEImplementationVersion: 1.13.0
    17 GlueSEImplementationVersion: 1.8.10
     2 GlueSEImplementationVersion: 1.8.11
     2 GlueSEImplementationVersion: 1.8.7
     4 GlueSEImplementationVersion: 1.8.8
     3 GlueSEImplementationVersion: 1.8.9
    14 GlueSEImplementationVersion: 1.9.0
  • all the sites with older DPM versions than 1.12 are suggested to upgrade to the latest DPM version , following the guide DPM upgrade (chapter 1 Upgrade to DPM 1.10.0 "Legacy Flavour" and chapter 2 Upgrade to DPM 1.10.0 "Dome Flavour")
    • DOME and the old LCGDM (srm protocol) will coexist
  • the upgrade will be followed-up mainly by WLCG
  • EGI Operations will liaise with the non-WLCG sites by opening GGUS tickets after DPM 1.13 is reeleased into UMD
  • Monitoring: sites should enable the monitoring of the HTTP/WebDav endpoint
    • register the storage service endpoint as WebDav service type, with production flag disabled, providing the URL field (see the HOWTO21
    • check if the tests are ok
    • switch the production flag to "yes"

HTCondorCE integration

Link to procedure: https://wiki.egi.eu/wiki/PROC19

GGUS ticket: https://ggus.eu/index.php?mode=ticket_info&ticket_id=139377

Steps status:

AOB

Next meeting

October https://indico.egi.eu/indico/event/4782/