|
|
| Line 1: |
Line 1: |
| {{Template:Op menubar}} {{Template:Doc_menubar}} {{TOC_right}}
| |
| [[Category:Grid Operations Meetings]]
| |
|
| |
|
| = General information =
| |
|
| |
| = Middleware =
| |
|
| |
| * UMD4.6.1 released http://repository.egi.eu/2018/03/14/release-umd-4-6-1/
| |
| ** APEL-SSM 2.2.0 - Added a check that certificates have not expired before starting SSM. SSL errors now propagated out properly and saved for received messages. Trimmed down the number of log messages generated for receivers. Added python-devel build requirement for non fedora-packager OSs (CentOS)
| |
| ** DPM 1.9.2 - This update includes releases 1.9.2 and lcgdm 0.19.0. It contains a fix for dpm-listspaces wrongly reporting 0 free space for pools See http://lcgdm.web.cern.ch/dpm-192-release
| |
| ** XRootD 4.7.1 - various bug fixes https://github.com/xrootd/xrootd/blob/v4.7.1/docs/ReleaseNotes.txt
| |
| ** CVMFS server 2.4.4 - fix registration of chunk hashes without bulk hash and non-SHA1 hash algorithm: fix on geoapi. See http://cvmfs.readthedocs.io/en/2.4/cpt-releasenotes.html
| |
| ** CVMFS 2.4.4 - react to a change of DNS server on macOS. See http://cvmfs.readthedocs.io/en/2.4/cpt-releasenotes.html
| |
| ** APEL 1.6.1 - various fixes and improvements https://github.com/apel/apel/releases/tag/1.6.1-1
| |
| ** frontier-squid 3.5.27-3.1 - includes an important configuration change to enable frontier clients to clear out certain types of cached errors
| |
|
| |
| * Next release is regular release UMD 4.7.0 (April/May)
| |
|
| |
| * '''UMD3 shutdown end of April'''
| |
| ** tests with umd-release upgrade are OK
| |
| ** broadcast sent to NGIs/sites/VOs https://operations-portal.egi.eu/broadcast/archive/2023
| |
| ** updates will no longer enter UMD3
| |
| ** feedback: mpi-start package missing in UMD4/SL6 -> maintenance and usage on grid to be evaluated in order to include it
| |
|
| |
| *CMD-OS update still in preparation: found SR for cloudkeeper, but unable to get packages for Ubuntu; no way to include yet user id isolatin patch for Mitaka/Ubuntu
| |
|
| |
| == Preview repository ==
| |
| *Release on 2018-03-07:
| |
| ** '''[[Preview 1.17.0]]''' [https://appdb.egi.eu/store/software/preview.repository/releases/1.0/1.17.0/ AppDB info] (sl6): APEL Client/Server 1.6.1, frontier-squid 3.5.27-3.1, FTS 3.7.8, srm-ifce 1.24.3, STORM 1.11.13, xrootd 4.8.1
| |
| ** '''[[Preview 2.17.0]]''' [https://appdb.egi.eu/store/software/preview.repository/releases/2.0/2.17.0/ AppDB info] (CentOS 7): APEL Client/Server 1.6.1, frontier-squid 3.5.27-3.1, FTS 3.7.8, srm-ifce 1.24.3, xrootd 4.8.1
| |
|
| |
| = Operations =
| |
|
| |
| == ARGO/SAM ==
| |
|
| |
| <br>
| |
|
| |
| == FedCloud ==
| |
|
| |
| *hardening FedCloud Appliances in App-DB (in progress)<br>
| |
| *cloudkeeper for OpenStack (the vmcatcher replacement) cannot yet be distributed for Mitaka/Ubuntu through CMD (missing packages), proposed to FedCloud TF an installation campaign to bypass (this time) the UMD process
| |
|
| |
| == Feedback from Helpdesk ==
| |
|
| |
| == Notifications from ARGO about the nagios probes failures ==
| |
|
| |
| In the process of implementing the notification system in ARGO, it has been recently introduced the following changes in GOC-DB:
| |
| *Notifications flag at the site level
| |
| *Notifications flag at the service endpoint level
| |
|
| |
| In this way ARGO will retrieve from GOC-DB the information about the sites and services whom sending the email notification and the related recipients.
| |
|
| |
| The logic of the notifications is the following:
| |
|
| |
| {| class="wikitable"
| |
| ! Site
| |
| ! Service
| |
| ! Notify?
| |
| |-
| |
| | Y
| |
| | Y
| |
| | '''Y'''
| |
| |-
| |
| | Y
| |
| | N
| |
| | '''N'''
| |
| |-
| |
| | N
| |
| | Y
| |
| | '''N'''
| |
| |-
| |
| | N
| |
| | N
| |
| | '''N'''
| |
| |}
| |
|
| |
| If there isn't any email contact defined at the service endpoint level, it will be used the site contact. '''Please review your contacts'''. (the notifications are not implemented yet in ARGO)
| |
|
| |
| '''NOTE''': It is not mandatory for the sites
| |
|
| |
| == Monthly Availability/Reliability ==
| |
|
| |
| *Underperformed sites in the past A/R reports with issues not yet fixed:
| |
| **NGI_BG BG05-SUGrid https://ggus.eu/index.php?mode=ticket_info&ticket_id=133884
| |
| **NGI_CH: https://ggus.eu/index.php?mode=ticket_info&ticket_id=133885
| |
| ***T3_CH_PSI (problems have been solved)
| |
| ***UNIBE-ID (problems with the "org.nordugrid.ARC-CE-sw-csh" probe: we need to discuss if this kind of test is still necessary
| |
| ***UNIGE-DPNC (reorganising, in long downtime: re-installing the storage from scratch, ARC-CE will be deployed again)
| |
| **NGI_IT https://ggus.eu/index.php?mode=ticket_info&ticket_id=133285 (IGI-BOLOGNA INFN-BOLOGNA-T3 INFN-CNAF-LHCB INFN-T1) the sites are slowly coming back online
| |
| ***Apr 6th: IGI-BOLOGNA has been decommissioned, and INFN-T1 is performing quite good, INFN-CNAF-LHCB waiting for new hosts certificates, INFN-BOLOGNA-T3 moving the servers from the deposit to the data center
| |
| **NGI_PL:
| |
| ***PSNC https://ggus.eu/index.php?mode=ticket_info&ticket_id=132815 (intermittent failures with QCG and SRM)
| |
| ***CYFRONET-LCG2 https://ggus.eu/index.php?mode=ticket_info&ticket_id=133886
| |
|
| |
| *Underperformed sites after 3 consecutive months, underperformed NGIs, QoS violations: ('''new tickets to be opened''')
| |
|
| |
|
| |
|
| |
| suspended sites:
| |
|
| |
| == NGI ARGUS servers not properly configured ==
| |
|
| |
| '''IMPORTANT: The ARGUS version that fixes the PAP permissions has been released in UMD 4.6: http://repository.egi.eu/2017/12/18/release-umd-4-6-0/'''
| |
| '''Please update to this version.'''
| |
|
| |
| As reported by EGI-CSIRT during the [https://indico.egi.eu/indico/event/3849/ February OMB], several NGI ARGUS servers are not properly configured and they are not passing the nagios tests (eu.egi.Argus-DNs probe):
| |
|
| |
| https://argo-mon.egi.eu/nagios/cgi-bin/status.cgi?servicegroup=SERVICE_ngi.ARGUS&style=detail
| |
|
| |
| How the probe works:
| |
| *Central Argus server suspend 1 fake DN each day
| |
| *Argo looks for this DN in NGI Arguses
| |
| Requirements for passing the test:
| |
| *NGI Argus must be configured to fetch DNs
| |
| *Argo must be able to connect to NGI Argus
| |
| *Argo DN must be authorized to query DNs
| |
|
| |
| Information about the nagios errors and possible solutions: https://wiki.egi.eu/wiki/EGI_CSIRT:Central_emergency_suspension#NGI_Argus_Monitoring
| |
|
| |
| Information on global banning set-up: https://wiki.nikhef.nl/grid/Argus_Global_Banning_Setup_Overview#NGI_Argus
| |
|
| |
| '''Authorization error''':
| |
| *NGI_ARMGRID https://ggus.eu/index.php?mode=ticket_info&ticket_id=133859
| |
|
| |
| '''Connection error''':
| |
|
| |
| *NGI_FI https://ggus.eu/index.php?mode=ticket_info&ticket_id=133872 (the service has been marked as not production and not monitored, asked an explanation).
| |
| *NGI_HR https://ggus.eu/index.php?mode=ticket_info&ticket_id=133867 (the server will be reinstalled on CentOS7)
| |
| *ROC_LA https://ggus.eu/index.php?mode=ticket_info&ticket_id=133866 (problem in issuing the new the host certificate)
| |
|
| |
| == Decommissioning EMI WMS ==
| |
|
| |
| WMS servers can be decommissioned. Please '''follow the procedure [[PROC12]]'''. The plan is:
| |
|
| |
| *Starting from January 2018, put the WMS servers in draining: this will block the submission of new jobs and will allow the jobs previously submitted to finish
| |
| **inform in advance your users that you are going to put in draining and then dismiss the WMS servers (as per [[PROC12]])
| |
| **there might be several VOs enabled on your WMS servers: in case only few of them need to use the service for few weeks more, you might disable the other VOs
| |
| *On Dec 14th EGI Operations sent a new broadcast to the VOs reminding the users the forthcoming WMS decommission
| |
| *From March, the nagios probe [https://wiki.egi.eu/wiki/MW_SAM_tests#General_tests eu.egi.sec.WMS] will return a CRITICAL status for the servers still in production, and the ROD teams will open a ticket to the sites that haven't finished the decommission process yet
| |
|
| |
| '''[https://goc.egi.eu/portal/index.php?Page_Type=Downtimes_Calendar&view=month&certStatus=Certified&service_type=WMS WMS servers in downtime on GOC-DB]'''
| |
|
| |
| [https://goc.egi.eu/portal/index.php?Page_Type=Services&serviceType=WMS&selectItemserviceType=WMS&ngi=&searchTerm=&production=TRUE&monitored=TRUE&certStatus=Certified&scopeMatch=all&servKeyNames=&servKeyValue= 16 WMS still registered on GOC-DB as production and monitored] (1 month ago they were 32)
| |
|
| |
| '''[https://argo-mon.egi.eu/nagios/cgi-bin/status.cgi?servicegroup=SERVICE_WMS&style=detail Status on nagios]'''
| |
|
| |
| == IPv6 readiness plans ==
| |
|
| |
| *assessment ongoing https://wiki.egi.eu/w/index.php?title=IPV6_Assessment
| |
| *still missing NGIs/ROCs
| |
| *added column in FedCloud wiki to monitor IPv6 readiness of cloud sites https://wiki.egi.eu/wiki/Federated_Cloud_infrastructure_status#Status_of_the_Federated_Cloud
| |
|
| |
| == webdav probes in OPERATORS profile ==
| |
|
| |
| The webdav probes was included in the [https://poem.egi.eu/poem/admin/poem/profile/4/ ARGO_MON_OPERATORS] profile after the approval in the [https://indico.egi.eu/indico/event/3616/ January OMB]: in this way the '''failures will generate an alarm on the dashboard, and the ROD teams can open a ticket to the failing sites'''. If no particular issue occurs, and if at least 75% of webdav endpoint are passing the tests, the probes will be added in the [https://poem.egi.eu/poem/admin/poem/profile/3/ ARGO_MON_CRITICAL] profile, so the results of these probes will be taken into account for the A/R figures.
| |
|
| |
| *webdav endpoints registered in GOC-DB: https://goc.egi.eu/gocdbpi/public/?method=get_service&&service_type=webdav
| |
| *link to nagios results: https://argo-mon.egi.eu/nagios/cgi-bin/status.cgi?servicegroup=SERVICE_webdav&style=detail
| |
| ** '''3 CRITICAL, 2 WARNING out of 24 (OK status: 79.2%)'''
| |
|
| |
| List of sites that not have completed the configuration yet:
| |
|
| |
| *NGI_HR: '''egee.srce.hr''' https://ggus.eu/index.php?mode=ticket_info&ticket_id=131041 (webdav URL missing in GOC-DB...)
| |
| *NGI_HR: '''egee.irb.hr''' https://ggus.eu/index.php?mode=ticket_info&ticket_id=134458 (webdav URL missing in GOC-DB...)
| |
|
| |
| List of sites that disabled webdav: UNIGE-DPNC, [https://ggus.eu/index.php?mode=ticket_info&ticket_id=131037 GR-01-AUTH], [https://ggus.eu/index.php?mode=ticket_info&ticket_id=131038 HG-03-AUTH], CETA-GRID, [https://ggus.eu/index.php?mode=ticket_info&ticket_id=131052 WUT]
| |
|
| |
| == Storage accounting deployment ==
| |
|
| |
| During the [https://indico.egi.eu/indico/event/3241/ September meeting], OMB has approved the full-scale deployment of storage accounting. The APEL team [[Storage accounting testing|has tested it with a group of early adopters sites]], and the results prove that storage accounting is now production-ready.
| |
|
| |
| Storage accounting is currently supported '''only for the DPM and dCache storage elements''' therefore only the resource centres deploying these kind of storage elements are requested to publish storage accounting data.
| |
|
| |
| In order to properly install and configure the storage accounting scripts, please follow the instructions reported in the wiki: https://wiki.egi.eu/wiki/APEL/Storage
| |
|
| |
| '''IMPORTANT''': be sure to have installed the star-accounting.py script v1.0.4 (http://svnweb.cern.ch/world/wsvn/lcgdm/lcg-dm/trunk/scripts/StAR-accounting/star-accounting.py)
| |
|
| |
| After setting up a daily cron job and running the accounting software, look for your data in the Accounting Portal: http://goc-accounting.grid-support.ac.uk/storagetest/storagesitesystems.html. If it does not appear within 24 hours, or there are other errors, please open a GGUS ticket to APEL who will help debug the process.
| |
|
| |
| '''IMPORTANT''': Do not encrypt the storage records with your host certificate, please comment out the “server_cert” variable in sender.cfg
| |
|
| |
| '''List of sites already publishing and of tickets opened is [[Storage accounting deployment|reported here]]'''. Several sites are not publishing the storage accounting data yet. '''NGIs please follow-up with the sites the configuration of the script in order to speed-up the process.'''
| |
|
| |
| *AsiaPacific: TW-NCUHEP https://ggus.eu/index.php?mode=ticket_info&ticket_id=131426
| |
| *NGI_AEGIS: AEGIS04-KG https://ggus.eu/index.php?mode=ticket_info&ticket_id=131431
| |
| *NGI_CH:
| |
| **CSCS-LCG2 https://ggus.eu/index.php?mode=ticket_info&ticket_id=131432
| |
| **T3_CH_PSI https://ggus.eu/index.php?mode=ticket_info&ticket_id=131433 (in progress, need to update dcache)
| |
| *NGI_DE:
| |
| **DESY-HH https://ggus.eu/index.php?mode=ticket_info&ticket_id=131439
| |
| **GoeGrid https://ggus.eu/index.php?mode=ticket_info&ticket_id=131443 (to do by the end of January)
| |
| **UNI-FREIBURG https://ggus.eu/index.php?mode=ticket_info&ticket_id=131445
| |
| *NGI_FRANCE: OBSPM https://ggus.eu/index.php?mode=ticket_info&ticket_id=131456
| |
| *NGI_IT:
| |
| **INFN-ROMA1-CMS https://ggus.eu/index.php?mode=ticket_info&ticket_id=131482
| |
| **INFN-ROMA1 https://ggus.eu/index.php?mode=ticket_info&ticket_id=131490 (postponed due to T1 problems)
| |
| *NGI_NDGF: FI_HIP_T2 https://ggus.eu/index.php?mode=ticket_info&ticket_id=131491
| |
| *NGI_NL:
| |
| **LSG-AMC https://ggus.eu/index.php?mode=ticket_info&ticket_id=131494 (in progress...)
| |
| **LSG-AMS https://ggus.eu/index.php?mode=ticket_info&ticket_id=131495 (in progress...)
| |
| **LSG-BCBR https://ggus.eu/index.php?mode=ticket_info&ticket_id=131496 (in progress...)
| |
| **LSG-EMC https://ggus.eu/index.php?mode=ticket_info&ticket_id=131497 (in progress...)
| |
| **LSG-KUN https://ggus.eu/index.php?mode=ticket_info&ticket_id=131498 (in progress...)
| |
| **LSG-LUMC https://ggus.eu/index.php?mode=ticket_info&ticket_id=131499 (in progress...)
| |
| **LSG-RUG https://ggus.eu/index.php?mode=ticket_info&ticket_id=131500 (in progress...)
| |
| **LSG-TUD https://ggus.eu/index.php?mode=ticket_info&ticket_id=131501 (in progress...)
| |
| **LSG-UM https://ggus.eu/index.php?mode=ticket_info&ticket_id=131502 (in progress...)
| |
| **LSG-VU https://ggus.eu/index.php?mode=ticket_info&ticket_id=131503 (in progress...)
| |
| **LSG-WUR https://ggus.eu/index.php?mode=ticket_info&ticket_id=131504 (in progress...)
| |
| **NIKHEF-ELPROD https://ggus.eu/index.php?mode=ticket_info&ticket_id=131506 (in progress, [https://ggus.eu/index.php?mode=ticket_info&ticket_id=131740 problems with star-accounting scripts])
| |
| *Russia: RRC-KI https://ggus.eu/index.php?mode=ticket_info&ticket_id=131546 (need to upgrade the DPM)
| |
|
| |
| = AOB =
| |
|
| |
| *NGI_FRANCE reorganizing the '''national operations following a''' '''distributed model''' (not anymore operated by <span class="im">CC-IN2P3</span>); are other NGIs reorganising as well? how? do you have suggestions?
| |
| *do you have suggestions to '''improve the EGI Operations''' meeting itself or to '''improve the distribution of the items discussed/reported between EGI Operations and OMB'''?
| |
|
| |
| == Next meeting ==
| |
|
| |
| *'''May 14th, 2018'''
| |
The wiki is deprecated and due to be decommissioned by the end of September 2022.