Alert.png The wiki is deprecated and due to be decommissioned by the end of September 2022.
The content is being migrated to other supports, new updates will be ignored and lost.
If needed you can get in touch with EGI SDIS team using operations @ egi.eu.

Difference between revisions of "Agenda-2018-09-10"

From EGIWiki
Jump to navigation Jump to search
(Created page with "{{Template:Op menubar}} {{Template:Doc_menubar}} {{TOC_right}} Category:Grid Operations Meetings = General information = = Middleware = UMD 4.7.0 released CentOS7 ...")
 
 
(36 intermediate revisions by 2 users not shown)
Line 6: Line 6:
= Middleware  =
= Middleware  =


UMD 4.7.0 released
* UMD 4.7.0 released in July https://wiki.egi.eu/wiki/Agenda-2018-07-16#Middleware
 
* ''there is an issue with CREAM/SL6 related to this new UMD 4.7.0 release for canL-java and voms-api-java''. A fix was released, but it seems not to handle properly everything, and CREAM team needs more time.  
CentOS7
** to prevent this from happening in the future, the UMD team will improve the Staged-Rollout step (under discussion)
 
** broadcast sent in July https://operations-portal.egi.eu/broadcast/archive/2141
    davix 0.6.7 - various improvements and bug fixes, see release notes http://dmc.web.cern.ch/release/davix-067
** reference ticket: https://ggus.eu/index.php?mode=ticket_info&ticket_id=136074
    ARC 15.03.18 - bugfix release, addressing 3 bugs discovered since release 15.03u17, see release notes http://www.nordugrid.org/arc/releases/15.03u18/release_notes_15.03u18.html
    APEL 1.6.2-1 - various bug fixes, see release notes https://github.com/apel/apel/releases/tag/1.6.2-1
    APEL-SSM 2.2.1 - Added a check that certificates have not expired before starting SSM. SSL errors now propagated out properly and saved for received messages. Trimmed down the number of log messages generated for receivers. Added python-devel build requirement for non fedora-packager OSs (CentOS).
    gfal2 2.15.4 - various bug fixes, see release notes https://dmc.web.cern.ch/release/gfal2-2154
    srm-ifce 1.24.3 - various bug fixes, se release notes http://dmc.web.cern.ch/release/srm-ifce-1.24.3
    FTS 3 7.8 - several bug fixes and improvements https://fts.web.cern.ch/sites/fts.web.cern.ch/themes/fts-webpage/releases-jekyll/releases/2018/01/30/FTS_3_7_8/
    XROOTD 4.8.2 - several bug fixes and improvements https://github.com/xrootd/xrootd/blob/v4.8.2/docs/ReleaseNotes.txt
    dCache 3.2.21 - several bug fixes and improvements https://www.dcache.org/downloads/1.9/release-notes-3.2.shtml#21
    VOMS clients 3.3.0 - see release notes http://italiangrid.github.io/voms/release-notes/voms-clients/3.0.7/
    MyProxy 6.1.29 - Fix -Werror=format-security errors
    CERN Frontier squid 3.5.28 - see release notes
    CernVM-FS server 2.5.0 - see release notes https://cvmfs.readthedocs.io/en/2.5/cpt-releasenotes.html
    yaim-core 5.1.6 - This release provides a backward compatible fix for GT4_INSTALL script name change and for the broken "join" command on CentOS/EL 7. Aslo added recognition of CentOS/SL/EL 7
    globus gridftp 12.5.1
 
SL6
 
    davix 0.6.7 - various improvements and bug fixes, see release notes http://dmc.web.cern.ch/release/davix-067
    ARC 15.03.18 - bugfix release, addressing 3 bugs discovered since release 15.03u17, see release notes http://www.nordugrid.org/arc/releases/15.03u18/release_notes_15.03u18.html
    APEL 1.6.2-1 - various bug fixes, see release notes https://github.com/apel/apel/releases/tag/1.6.2-1
    APEL-SSM 2.2.1 - Added a check that certificates have not expired before starting SSM. SSL errors now propagated out properly and saved for received messages. Trimmed down the number of log messages generated for receivers. Added python-devel build requirement for non fedora-packager OSs (CentOS).
    gfal2 2.15.4 - various bug fixes, see release notes https://dmc.web.cern.ch/release/gfal2-2154
    srm-ifce 1.24.3 - various bug fixes, se release notes http://dmc.web.cern.ch/release/srm-ifce-1.24.3
    FTS 3 7.8 - several bug fixes and improvements https://fts.web.cern.ch/sites/fts.web.cern.ch/themes/fts-webpage/releases-jekyll/releases/2018/01/30/FTS_3_7_8/
    StoRM 1.11.13 - fixes to some outstanding bugs, see details in the release notes http://italiangrid.github.io/storm/2018/02/19/storm-v1.11.13-released.html
    XROOTD 4.8.2 - several bug fixes and improvements https://github.com/xrootd/xrootd/blob/v4.8.2/docs/ReleaseNotes.txt
    dCache 3.2.21 - several bug fixes and improvements https://www.dcache.org/downloads/1.9/release-notes-3.2.shtml#21
    MyProxy 6.1.29 - Fix -Werror=format-security errors
    VOMS clients 3.3.0 - see release notes http://italiangrid.github.io/voms/release-notes/voms-clients/3.0.7/
    yaim-core 5.1.6 - Fixed CentOS 6 line
    CERN Frontier squid 3.5.28 - see release notes
    CernVM-FS server 2.5.0 - see release notes https://cvmfs.readthedocs.io/en/2.5/cpt-releasenotes.html
    globus gridftp 12.5.1
    canl 2.5.0
    bouncycastle 1.58-1
 
* issue with canL-java, in progress https://ggus.eu/index.php?mode=ticket_info&ticket_id=136074
 


== Preview repository  ==
== Preview repository  ==
* released on 2018-06-22:
*2018-08-06
** '''[[Preview 1.18.0]]''' [https://appdb.egi.eu/store/software/preview.repository/releases/1.0/1.18.0/ AppDB info] (sl6): APEL Client/Server 1.7.0, dCache 3.2.22, DPM 1.10.3, frontier-squid 3.5.27-4.2, gfal2 2.15.4, xrootd 4.8.3
** '''[[Preview 1.19.0]]''' [https://appdb.egi.eu/store/software/preview.repository/releases/1.0/1.19.0/ AppDB info] (sl6): ARGUS 1.7.2, CGSI-gSOAP 1.3.11, CREAM 1.16.7, davix 0.6.8, dCache 3.2.27, frontier-squid 3.5.27-5.2, STORM 1.11.14, xrootd 4.8.4
** '''[[Preview 2.18.0]]''' [https://appdb.egi.eu/store/software/preview.repository/releases/2.0/2.18.0/ AppDB info] (CentOS 7): APEL Client/Server 1.7.0, dCache 3.2.22, DPM 1.10.3, frontier-squid 3.5.27-4.2, gfal2 2.15.4, xrootd 4.8.3
** '''[[Preview 2.19.0]]''' [https://appdb.egi.eu/store/software/preview.repository/releases/2.0/2.19.0/ AppDB info] (CentOS 7): CGSI-gSOAP 1.3.11, CREAM 1.16.7, davix 0.6.8, dCache 3.2.27, frontier-squid 3.5.27-5.2, xrootd 4.8.4


= Operations  =
= Operations  =
Line 60: Line 22:


== FedCloud  ==
== FedCloud  ==
* 14 sites running up to date CMF
* cASO installation campaign is over
* 6 sites running CMF version no longer supported; upgrade plans provided, EGI Operations to contact them and follow the execution of the plan provided
** few sites still 1.1.1 because using appliance, to be forwarded to Enol
* REMINDER: sites can be SUSPENDED if running an outdated version of the software!
** http://goc-accounting.grid-support.ac.uk/cloudtest/cloudsites2.html
* 2 OpenNebula sites switching to OpenStack from OpenNebula
* accounting upgrades available
** upgrade to cASO 1.2.0 for OpenStack (available in CMD)
** if not yet done, OpenNebula sites must align to oneacct 0.4.6-1 (available in CMD)
** upgrade to latest APEL-SSM, available in CMD only for C7 (Ubuntu packages are coming)


== Feedback from Helpdesk  ==
== Feedback from Helpdesk  ==
Line 95: Line 52:
To track the process, a [https://wiki.egi.eu/wiki/Verify_Configuration_Records series of tickets] has been opened.
To track the process, a [https://wiki.egi.eu/wiki/Verify_Configuration_Records series of tickets] has been opened.


'''Status on Jul 16th''': 20 Solved, 3 in progress. out of 32 NGIs
'''Status on Sep 7th''': 26 Solved, 3 in progress, 3 not replied (NGI_BY, NGI_DE, NGI_GRNET).


== Notifications from ARGO about the nagios probes failures ==
== Notifications from ARGO about the nagios probes failures ==
Line 137: Line 94:
== Monthly Availability/Reliability  ==
== Monthly Availability/Reliability  ==


*Underperformed sites in the past A/R reports with issues not yet fixed:  
*Underperformed sites in the past A/R reports with issues not yet fixed:
**NGI_BG BG05-SUGrid https://ggus.eu/index.php?mode=ticket_info&ticket_id=133884 : CE problems solved, outdated CAs; now jobs submission failures
**NGI_IT: https://ggus.eu/index.php?mode=ticket_info&ticket_id=136523
**NGI_UA: https://ggus.eu/index.php?mode=ticket_info&ticket_id=134933
***INFN-MILANO-ATLASC: recovered
***UA-BITP: hardware problems and authentication issues with the SE, now solved.
***INFN-PERUGIA: downtime longer than a month, to suspend
***UA-KNU: hardware problems; the services are back online, still failures with ARC-CE.
*Underperformed sites after 3 consecutive months, underperformed NGIs, QoS violations: ('''June 2018'''):
**NGI_DE: https://ggus.eu/index.php?mode=ticket_info&ticket_id=135905
***FZK-LCG2
***UNI-FREIBURG
**NGI_GRNET: https://ggus.eu/index.php?mode=ticket_info&ticket_id=135906
***GR-01-AUTH
**NGI_IT: https://ggus.eu/index.php?mode=ticket_info&ticket_id=135907
***INFN-TORINO
**NGI_NL: dismissing several sites https://ggus.eu/index.php?mode=ticket_info&ticket_id=135908
**NGI_PL: https://ggus.eu/index.php?mode=ticket_info&ticket_id=135909
**NGI_PL: https://ggus.eu/index.php?mode=ticket_info&ticket_id=135909
***CYFRONET-PROMETHEUS: UNKNOWN status returned by pl.plgrid.QCG-Computing probe
***CYFRONET-PROMETHEUS: UNKNOWN status returned by pl.plgrid.QCG-Computing probe, fixed a problem on ARGO, statistics improving...
**NGI_RO: https://ggus.eu/index.php?mode=ticket_info&ticket_id=135910
**NGI_UA: https://ggus.eu/index.php?mode=ticket_info&ticket_id=134933 some other failures in July
***RO-07-NIPNE
***UA-BITP: SE problems solved, perfomance not completely good yet
***RO-14-ITIM
***UA-KNU: problems with local storage and clock skew; another storage array failure due to broken air conditioner and overheating, statistics are improving;
**NGI_UA: https://ggus.eu/index.php?mode=ticket_info&ticket_id=135911
*Underperformed sites after 3 consecutive months, underperformed NGIs, QoS violations: ('''August 2018'''):
***UA-IMBG
**AsiaPacific: https://ggus.eu/index.php?mode=ticket_info&ticket_id=137041
***UA-NSCMBR
***TW-NTU-HEP
**NGI_AEGIS: https://ggus.eu/index.php?mode=ticket_info&ticket_id=137042
***AEGIS01-IPB-SCL
***AEGIS04-KG
**NGI_ARMGRID: https://ggus.eu/index.php?mode=ticket_info&ticket_id=137043
***AM-01-IIAP: CREAM-CE problems investigating into [https://ggus.eu/index.php?mode=ticket_info&ticket_id=136194 GGUS 136194]
**NGI_FI: https://ggus.eu/index.php?mode=ticket_info&ticket_id=137044
***FI_AA
**NGI_FRANCE: https://ggus.eu/index.php?mode=ticket_info&ticket_id=137045 (QoS)
**NGI_GRNET: https://ggus.eu/index.php?mode=ticket_info&ticket_id=137046
***HG-05-FORTH


suspended sites: none for OLA violation
suspended sites: BG05-SUGrid (Aug 1st),


== Decommissioning EMI WMS  ==
== Decommissioning EMI WMS  ==
Line 174: Line 131:
'''[https://goc.egi.eu/portal/index.php?Page_Type=Downtimes_Calendar&view=month&certStatus=Certified&service_type=WMS WMS servers in downtime on GOC-DB]'''  
'''[https://goc.egi.eu/portal/index.php?Page_Type=Downtimes_Calendar&view=month&certStatus=Certified&service_type=WMS WMS servers in downtime on GOC-DB]'''  


[https://goc.egi.eu/portal/index.php?Page_Type=Services&serviceType=WMS&selectItemserviceType=WMS&ngi=&searchTerm=&production=TRUE&monitored=TRUE&certStatus=Certified&scopeMatch=all&servKeyNames=&servKeyValue= 8 WMS still registered on GOC-DB as production and monitored] (1 month ago they were 16)
[https://goc.egi.eu/portal/index.php?Page_Type=Services&serviceType=WMS&selectItemserviceType=WMS&ngi=&searchTerm=&production=TRUE&monitored=TRUE&certStatus=Certified&scopeMatch=all&servKeyNames=&servKeyValue= 9 WMS still registered on GOC-DB as production and monitored]


'''[https://argo-mon.egi.eu/nagios/cgi-bin/status.cgi?servicegroup=SERVICE_WMS&style=detail Status on nagios]'''
'''[https://argo-mon.egi.eu/nagios/cgi-bin/status.cgi?servicegroup=SERVICE_WMS&style=detail Status on nagios]'''
Line 225: Line 182:
*NGI_DE:
*NGI_DE:
**DESY-HH https://ggus.eu/index.php?mode=ticket_info&ticket_id=131439
**DESY-HH https://ggus.eu/index.php?mode=ticket_info&ticket_id=131439
**GoeGrid https://ggus.eu/index.php?mode=ticket_info&ticket_id=131443 (to do by the end of January)
**GoeGrid https://ggus.eu/index.php?mode=ticket_info&ticket_id=131443 (postponed)
*NGI_FRANCE: OBSPM https://ggus.eu/index.php?mode=ticket_info&ticket_id=131456
*NGI_IT:
*NGI_IT:
**INFN-ROMA1-CMS https://ggus.eu/index.php?mode=ticket_info&ticket_id=131482 (cannot do it now, Jun 19th)
**INFN-ROMA1-CMS https://ggus.eu/index.php?mode=ticket_info&ticket_id=131482 (cannot do it now, Jun 19th)
Line 234: Line 190:


= AOB  =
= AOB  =
* is MPI used in EGI? if so, are sites interested in having MPI supported by EGI on the grid?
 
* Meltdown / SPECTRE issue https://wiki.egi.eu/wiki/SVG:Advisory-SVG-CVE-2017-5753
== Next meeting  ==
== Next meeting  ==


*'''September 10th, 2018''' https://indico.egi.eu/indico/event/4166/
*'''October 8th, 2018''' https://indico.egi.eu/indico/event/4168/

Latest revision as of 12:56, 10 September 2018

Main EGI.eu operations services Support Documentation Tools Activities Performance Technology Catch-all Services Resource Allocation Security


Documentation menu: Home Manuals Procedures Training Other Contact For: VO managers Administrators


General information

Middleware

Preview repository

  • 2018-08-06
    • Preview 1.19.0 AppDB info (sl6): ARGUS 1.7.2, CGSI-gSOAP 1.3.11, CREAM 1.16.7, davix 0.6.8, dCache 3.2.27, frontier-squid 3.5.27-5.2, STORM 1.11.14, xrootd 4.8.4
    • Preview 2.19.0 AppDB info (CentOS 7): CGSI-gSOAP 1.3.11, CREAM 1.16.7, davix 0.6.8, dCache 3.2.27, frontier-squid 3.5.27-5.2, xrootd 4.8.4

Operations

ARGO/SAM

FedCloud

Feedback from Helpdesk

yearly review of the information registered into GOC-DB

2018-05-09

On a yearly basis, the information registered into GOC-DB need to be verified. NGIs and RCs have been asked to check them. In particular:

  1. NGI managers should review the people registered and the roles assigned to them, and in particular check the following information:
    • E-Mail
    • ROD E-Mail
    • Security E-Mail
NGI Managers should also review the status of the "not certified" RCs, in according to the RC Status Workflow;
  1. RCs administrators should review the people registered and the roles assigned to them, and in particular check the following information:
    • E-Mail
    • telephone numbers
    • CSIRT E-Mail
RC administrators should also review the information related to the registered service endpoints.

The process should be completed by June 6th.

To track the process, a series of tickets has been opened.

Status on Sep 7th: 26 Solved, 3 in progress, 3 not replied (NGI_BY, NGI_DE, NGI_GRNET).

Notifications from ARGO about the nagios probes failures

In the process of implementing the notification system in ARGO, it was introduced the following changes in GOC-DB:

  • Notifications flag at the site level
  • Notifications flag at the service endpoint level

In this way ARGO will retrieve from GOC-DB the information about the sites and services whom sending the email notification and the related recipients.

The logic of the notifications is the following:

Site Service Notify?
Y Y Y
Y N N
N Y N
N N N

If there isn't any email contact defined at the service endpoint level, it will be used the site contact. Please review your contacts.

You can enable the notifications.

NOTE: It is not mandatory for the sites

Monthly Availability/Reliability

suspended sites: BG05-SUGrid (Aug 1st),

Decommissioning EMI WMS

WMS servers can be decommissioned. Please follow the procedure PROC12. The plan is:

  • Starting from January 2018, put the WMS servers in draining: this will block the submission of new jobs and will allow the jobs previously submitted to finish
    • inform in advance your users that you are going to put in draining and then dismiss the WMS servers (as per PROC12)
    • there might be several VOs enabled on your WMS servers: in case only few of them need to use the service for few weeks more, you might disable the other VOs
  • On Dec 14th EGI Operations sent a new broadcast to the VOs reminding the users the forthcoming WMS decommission
  • From March, the nagios probe eu.egi.sec.WMS will return a CRITICAL status for the servers still in production, and the ROD teams will open a ticket to the sites that haven't finished the decommission process yet

WMS servers in downtime on GOC-DB

9 WMS still registered on GOC-DB as production and monitored

Status on nagios

IPv6 readiness plans

webdav probes in OPERATORS profile

The webdav probes was included in the ARGO_MON_OPERATORS profile after the approval in the January OMB: in this way the failures will generate an alarm on the dashboard, and the ROD teams can open a ticket to the failing sites. If no particular issue occurs, and if at least 75% of webdav endpoint are passing the tests, the probes will be added in the ARGO_MON_CRITICAL profile, so the results of these probes will be taken into account for the A/R figures.

List of sites that not have completed the configuration yet:

List of sites that disabled webdav: UNIGE-DPNC, GR-01-AUTH, HG-03-AUTH, CETA-GRID, WUT

Storage accounting deployment

During the September 2017 meeting, OMB has approved the full-scale deployment of storage accounting. The APEL team tested it with a group of early adopters sites, and the results prove that storage accounting is now production-ready.

Storage accounting is currently supported only for the DPM and dCache storage elements therefore only the resource centres deploying these kind of storage elements are requested to publish storage accounting data.

In order to properly install and configure the storage accounting scripts, please follow the instructions reported in the wiki: https://wiki.egi.eu/wiki/APEL/Storage

IMPORTANT: be sure to have installed the star-accounting.py script v1.0.4 (http://svnweb.cern.ch/world/wsvn/lcgdm/lcg-dm/trunk/scripts/StAR-accounting/star-accounting.py)

After setting up a daily cron job and running the accounting software, look for your data in the Accounting Portal: http://goc-accounting.grid-support.ac.uk/storagetest/storagesitesystems.html. If it does not appear within 24 hours, or there are other errors, please open a GGUS ticket to APEL who will help debug the process.

Test portal: http://accounting-devel.egi.eu/storage.php

IMPORTANT: Do not encrypt the storage records with your host certificate, please comment out the “server_cert” variable in sender.cfg

List of sites already publishing and of tickets opened is reported here.

Several sites are not publishing the storage accounting data yet. NGIs please follow-up with the sites the configuration of the script in order to speed-up the process.

AOB

Next meeting