|Main||EGI.eu operations services||Support||Documentation||Tools||Activities||Performance||Technology||Catch-all Services||Resource Allocation||Security|
|Documentation menu:||Home •||Manuals •||Procedures •||Training •||Other •||Contact ►||For:||VO managers •||Administrators|
- CMD-OS 1.2.0 released http://repository.egi.eu/2018/05/23/release-cmd-os-1-2-0/
- cloudkeeper 1.6.0 (NEW) - Cloudkeeper checks the EGI App DB for new or updated images that need to be supported on the site. It downloads images and registers them with OpenNebula, so that they can be used in resource instantiation.
- cloudkeeper-os 0.9.9 (NEW) - Cloudkeeper-os is able to manage OpenStack cloud - upload, update and remove images and templates representing EGI AppDB appliances. cloudkeeper-os runs as a server listening for gRPC communication usually from core cloudkeeper component.
- cloud-info-provider 0.9.1 - makes OCCI optional, avoiding failures if no OCCI endpoint is configured; updates on documentation, now stored on a dedicated EGI Foundation github store
- found dependency problem with cloudkeeper-os (UMD team on it) https://ggus.eu/index.php?mode=ticket_info&ticket_id=135539
- UMD 4.7.0 scheduled for June
- FTS 3.7.8
- gfal2 major release ( 2.15.4 )
- dCache 3.2 (3.0 is being decommissioned, worth including the latest in UMD4)
- ARC 5.4.2 (15.03u18)
- glite-yaim-core-5.1.6 (fix https://ggus.eu/index.php?mode=ticket_info&ticket_id=127321#update#14 )
- CMD-ONE upgrade in preparation
- cloudkeeper, cloudkeeper-one, cloud-info-provider need Early Adopters on OpenNebula 5
to be released a new update containing: frontier squid 3.5.27, xrootd 4.8.3, gfal2 2.15.5, APEL 1.6.2
- starting decommissioning campaign for old OpenStack and OpenNebula versions: all software must follow the Service Operations Security Policy, so out of support software/versions should go through decommissioning procedure https://wiki.egi.eu/wiki/PROC16
- 23 sites: 15 Openstack, 7 OpenNebula; OpenNebula 4 to be decommissioned, OpenStack <Ocata and !=Mitaka/Xenial LTS to be decommissioned
- EGI Operations is going to asking sites about plans (no hard suspensions) for the update to a supported version (timeline, which version, feedback (using CMD? Using dockers? Using FedCloud appliance?) )
Feedback from Helpdesk
yearly review of the information registered into GOC-DB
On a yearly basis, the information registered into GOC-DB need to be verified. NGIs and RCs have been asked to check them. In particular:
- NGI managers should review the people registered and the roles assigned to them, and in particular check the following information:
- ROD E-Mail
- Security E-Mail
- NGI Managers should also review the status of the "not certified" RCs, in according to the RC Status Workflow;
- RCs administrators should review the people registered and the roles assigned to them, and in particular check the following information:
- telephone numbers
- CSIRT E-Mail
- RC administrators should also review the information related to the registered service endpoints.
The process should be completed by June 6th.
To track the process, a series of tickets has been opened.
Status on June 11th: 12 Solved, 4 in progress. out of 32 NGIs
Notifications from ARGO about the nagios probes failures
In the process of implementing the notification system in ARGO, it was introduced the following changes in GOC-DB:
- Notifications flag at the site level
- Notifications flag at the service endpoint level
In this way ARGO will retrieve from GOC-DB the information about the sites and services whom sending the email notification and the related recipients.
The logic of the notifications is the following:
If there isn't any email contact defined at the service endpoint level, it will be used the site contact. Please review your contacts.
You can enable the notifications.
NOTE: It is not mandatory for the sites
- Underperformed sites in the past A/R reports with issues not yet fixed:
- NGI_BG BG05-SUGrid https://ggus.eu/index.php?mode=ticket_info&ticket_id=133884 : CE problems solved, outdated CAs; now jobs submission failures
- NGI_CZ: praguelcg2 https://ggus.eu/index.php?mode=ticket_info&ticket_id=134931 arc-ce probes returned UNKNOWN status; Problem with org.nordugrid.ARC-CE-submit-ops test was caused by missing "ENV/PROXY" in the ARC-CE runtime (see http://wiki.nordugrid.org/wiki/Service_Monitoring)
- NGI_IT https://ggus.eu/index.php?mode=ticket_info&ticket_id=133285 (IGI-BOLOGNA INFN-BOLOGNA-T3 INFN-CNAF-LHCB INFN-T1) the sites are slowly coming back online
- Apr 6th: IGI-BOLOGNA has been decommissioned, and INFN-T1 is performing quite good, INFN-CNAF-LHCB waiting for new hosts certificates, INFN-BOLOGNA-T3 moving the servers from the deposit to the data center
- NGI_UA: https://ggus.eu/index.php?mode=ticket_info&ticket_id=134933
- UA-BITP: hardware problems and authentication issues with the SE, now solved.
- UA-KNU: hardware problems; the services are back online, still failures with ARC-CE and site-BDII.
- Underperformed sites after 3 consecutive months, underperformed NGIs, QoS violations: (May 2018):
- no new A/R violations, several Qos Violation: decided not to open new tickets:
- AsiaPacific: very urgent and top priority
- NGI_ARMGRID: urgent
- NGI_CHINA: top priority
- NGI_FRANCE: very urgent and top priority
- NGI_IL: urgent
- NGI_IT: top priority
- NGI_NL: top priority
- NGI_PL: very urgent and top priority
- NGI_UK: top priority
- ROC_CANADA: very urgent
- no new A/R violations, several Qos Violation: decided not to open new tickets:
suspended sites: none for OLA violation
Decommissioning EMI WMS
WMS servers can be decommissioned. Please follow the procedure PROC12. The plan is:
- Starting from January 2018, put the WMS servers in draining: this will block the submission of new jobs and will allow the jobs previously submitted to finish
- inform in advance your users that you are going to put in draining and then dismiss the WMS servers (as per PROC12)
- there might be several VOs enabled on your WMS servers: in case only few of them need to use the service for few weeks more, you might disable the other VOs
- On Dec 14th EGI Operations sent a new broadcast to the VOs reminding the users the forthcoming WMS decommission
- From March, the nagios probe eu.egi.sec.WMS will return a CRITICAL status for the servers still in production, and the ROD teams will open a ticket to the sites that haven't finished the decommission process yet
8 WMS still registered on GOC-DB as production and monitored (1 month ago they were 16)
IPv6 readiness plans
- assessment ongoing https://wiki.egi.eu/w/index.php?title=IPV6_Assessment
- still missing NGIs/ROCs
- added column in FedCloud wiki to monitor IPv6 readiness of cloud sites https://wiki.egi.eu/wiki/Federated_Cloud_infrastructure_status#Status_of_the_Federated_Cloud
webdav probes in OPERATORS profile
The webdav probes was included in the ARGO_MON_OPERATORS profile after the approval in the January OMB: in this way the failures will generate an alarm on the dashboard, and the ROD teams can open a ticket to the failing sites. If no particular issue occurs, and if at least 75% of webdav endpoint are passing the tests, the probes will be added in the ARGO_MON_CRITICAL profile, so the results of these probes will be taken into account for the A/R figures.
- webdav endpoints registered in GOC-DB: https://goc.egi.eu/gocdbpi/public/?method=get_service&&service_type=webdav
- link to nagios results: https://argo-mon.egi.eu/nagios/cgi-bin/status.cgi?servicegroup=SERVICE_webdav&style=detail
- 5 CRITICAL, 2 WARNING out of 33 (OK status: 78.8%)
List of sites that not have completed the configuration yet:
- NGI_HR: egee.srce.hr https://ggus.eu/index.php?mode=ticket_info&ticket_id=131041 (webdav URL missing in GOC-DB...)
- NGI_HR: egee.irb.hr https://ggus.eu/index.php?mode=ticket_info&ticket_id=134458 (webdav URL missing in GOC-DB...)
Storage accounting deployment
During the September meeting, OMB has approved the full-scale deployment of storage accounting. The APEL team has tested it with a group of early adopters sites, and the results prove that storage accounting is now production-ready.
Storage accounting is currently supported only for the DPM and dCache storage elements therefore only the resource centres deploying these kind of storage elements are requested to publish storage accounting data.
In order to properly install and configure the storage accounting scripts, please follow the instructions reported in the wiki: https://wiki.egi.eu/wiki/APEL/Storage
IMPORTANT: be sure to have installed the star-accounting.py script v1.0.4 (http://svnweb.cern.ch/world/wsvn/lcgdm/lcg-dm/trunk/scripts/StAR-accounting/star-accounting.py)
After setting up a daily cron job and running the accounting software, look for your data in the Accounting Portal: http://goc-accounting.grid-support.ac.uk/storagetest/storagesitesystems.html. If it does not appear within 24 hours, or there are other errors, please open a GGUS ticket to APEL who will help debug the process.
Test portal: http://accounting-devel.egi.eu/storage.php
IMPORTANT: Do not encrypt the storage records with your host certificate, please comment out the “server_cert” variable in sender.cfg
List of sites already publishing and of tickets opened is reported here. Several sites are not publishing the storage accounting data yet. NGIs please follow-up with the sites the configuration of the script in order to speed-up the process.
- AsiaPacific: TW-NCUHEP https://ggus.eu/index.php?mode=ticket_info&ticket_id=131426
- NGI_AEGIS: AEGIS04-KG https://ggus.eu/index.php?mode=ticket_info&ticket_id=131431
- DESY-HH https://ggus.eu/index.php?mode=ticket_info&ticket_id=131439
- GoeGrid https://ggus.eu/index.php?mode=ticket_info&ticket_id=131443 (to do by the end of January)
- NGI_FRANCE: OBSPM https://ggus.eu/index.php?mode=ticket_info&ticket_id=131456
- INFN-ROMA1-CMS https://ggus.eu/index.php?mode=ticket_info&ticket_id=131482
- NGI_NDGF: FI_HIP_T2 https://ggus.eu/index.php?mode=ticket_info&ticket_id=131491
- NIKHEF-ELPROD https://ggus.eu/index.php?mode=ticket_info&ticket_id=131506 (in progress, problems with star-accounting scripts)
- Russia: RRC-KI https://ggus.eu/index.php?mode=ticket_info&ticket_id=131546 (need to upgrade the DPM)
Sites with wrong configuration (2018-05-22):
- AEGIS03-ELEF-LEDA https://ggus.eu/index.php?mode=ticket_info&ticket_id=135262 (SOLVED)
- AM-01-IIAP https://ggus.eu/index.php?mode=ticket_info&ticket_id=135263 (SOLVED, site suspended)
- CNR-ILC-PISA https://ggus.eu/index.php?mode=ticket_info&ticket_id=135264 (SOLVED)
- CYFRONET-LCG2 https://ggus.eu/index.php?mode=ticket_info&ticket_id=135265 (system to be reconfigured)
- RO-07-NIPNE https://ggus.eu/index.php?mode=ticket_info&ticket_id=135266 (SOLVED)
- UKI-NORTHGRID-LIV-HEP https://ggus.eu/index.php?mode=ticket_info&ticket_id=135267 (SOLVED)
- UNIBE-LHEP https://ggus.eu/index.php?mode=ticket_info&ticket_id=135268 (SOLVED)
- July 9th, 2018