Alert.png The wiki is deprecated and due to be decommissioned by the end of September 2022.
The content is being migrated to other supports, new updates will be ignored and lost.
If needed you can get in touch with EGI SDIS team using operations @ egi.eu.

EGI-InSPIRE:UNICORE integration task force

From EGIWiki
Jump to navigation Jump to search
EGI Activity groups Special Interest groups Policy groups Virtual teams Distributed Competence Centres


Alert.png This article is Deprecated and should no longer be used, but is still available for reasons of reference.


Interoperations menu: Home Unicore Globus Mapper/QCG ARC Desktop Grids EGI Federated Cloud XSEDE Contact



Coordinator: M. Krakowian/EGI.eu

 Meetings page: https://www.egi.eu/indico/categoryDisplay.py?categId=49

Mailing list: unicore-integration-tf@mailman.egi.eu

Mailing list page: https://www.egi.eu/sso/groupView/unicore-integration-tf

NGIs with UNICORE RC's: NGI_PL, NGI_DE, NGI_BY


Current status

Status of integration

Status Comments
Management

Completed

  • unicore6.Registry
  • unicore6.Gateway
  • unicore6.TargetSystemFactory
  • unicore6.StorageFactory
  • unicore6.StorageManagement
  • unicore6.ServiceOrchestrator
  • unicore6.WorkflowFactory
  • unicore6.UVOSAssertionQueryService
Monitoring

Completed

  • unicore6.Gateway - emi.unicore.Gateway
  • unicore6.Registry - emi.unicore.Registry
  • unicore6.ServiceOrchestrator - emi.unicore.ServiceOrchestrator
  • unicore6.StorageManagement - emi.unicore.GlobalStorage, emi.unicore.GlobalStorage-FreeSpace
  • unicore6.TargetSystemFactory - emi.unicore.TargetSystemFactory, emi.unicore.UNICORE-Job
  • unicore6.UVOSAssertionQueryService - emi.unicore.UVOS
  • unicore6.WorkflowFactory - emi.unicore.WorkflowService
  • unicore6.StorageFactory - emi.unicore.StorageFactory

EGI OPS Availability and Reliability Profile:

  • unicore6.TargetSystemFactory - emi.unicore.TargetSystemFactory, emi.unicore.UNICORE-Job
Accounting

In progress

Accounting services for UNICORE have been developed by NGI_PL and NGI_BY. UNICORE resource centres can publish accounting information through a UNICORE RUS server

Support Completed Support is provided by the UNICORE product team through the EGI Helpdesk with medium level of service support
Dashboard

Completed

  • GGUS-81278
    • emi.unicore.Gateway
    • emi.unicore.Registry
    • emi.unicore.ServiceOrchestrator
    • emi.unicore.UVOS
    • emi.unicore.TargetSystemFactory
    • emi.unicore.GlobalStorage
    • emi.unicore.GlobalStorage-FreeSpace
    • emi.unicore.WorkflowService
Documentation

Completed

Information System


In progress




Legend

Actions

Open action points

September 2012

Action

Descripton Assigned to Deadline Concerning
ACTION 1 Open a JIRA ticket for adding available CIS probes into next SAM release. Emir Imamagic
Monitoring
ACTION 2 Open a SAM JIRA ticket for integration of the new version of complex UNICORE job test. Emir Imamagic
Monitoring
ACTION 3
Check the status of UNICORE probes on NGI_DE SAM instance. Emir Imamagic
Monitoring
ACTION 4
Add NGI_BY UNICORE services to GOCDB. Serge Salamanka
Management


June 2012

Action

Descripton Assigned to Deadline Concerning
ACTION 1 Clarify certification procedures from perspective of other middleware together with PL-Grid. Malgorzata Krakowian
Documentation


Closed action points

June 2012

ACTION 2: Notify German and Poland via RT tickets once the SR of SAM Update-17 starts (Mario David)

  • Poland participated in SR of SAM Update-17

ACTION 3: Integration of atomic job test in SAM Update-19: https://tomtools.cern.ch/jira/browse/SAM-2722 (Emir Imamagic)

  • Integrated two new tests: emi.unicore.StorageFactory and emi.unicore.UNICORE-Job.

ACTION 4: Provide update once new probes based on UCC 6.5.0 are available. (Krzysztof Benedyczak)

  • Probes will be provided in the next EMI release. It was agreed to use solution with local registry in Update-19. The next SAM release will use new version of UCC.

ACTION 5 (ADDITIONAL): Added new UNICORE service types to GOCDB (Emir Imamagic):

  • org.ogf.bes.BESFactory: BES job submission entry point
  • unicore6.Catalogue: service providing unified view of files in multiple unicore6.StorageManagement
  • unicore6.CISInfoProvider: provides data for CIS
  • unicore6.CISRegistryPortType: provides human friendly interface for browsing the UNICORE Grid status
  • unicore6.GridBeanService: enables download of GridBeans (modules providing application-specific GUI).

March 2012

ACTION 1: Test using local registry for probes. (Krzysztof Benedyczak)

  • Testing with local registries was implemented in SAM Update-19. In the next release of UCC (6.5.0), probes do not require Registry at all.

ACTION 2: Ask NGIs on interest in UNICORE deployment. (Tiziana Ferrari & Emir Imamagic)

  • Asked at the OMB in Munich, no-one expressed interest.

Old action points

2011

  • AP: Add more UNICORE services into GOCDB.
Update: https://goc.egi.eu/portal/index.php?Page_Type=View_Object&object_id=22973&grid_id=0
https://goc.egi.eu/portal/index.php?Page_Type=View_Object&object_id=14727&grid_id=0
110704 Found problems with entering all RFC 3986 chars parsing of some characters in Service Point URL field, fixed by DM 110705.
Update: Still some minor things: description of the ServiceEndPointURL input field should contain the now correct RFC 3986 characters and initial letters (R and G) of unicore6.Registry and unicore6.Gateway should be capital.


Update: KB to make an update that the probes will be maintained by EMI, EI to close the ticket after the corresponding release.
Update: KB has updated the ticket.
Update: UNICORE probes integration delay due to misjudging the amount of effort needed for actual integration. Deadline for SAM for the Update-12 release missed. Next deadline is SAM Update-13 which will be released around the end of July
Update: SAM Update-13 pushed back to August 12 (from end of July), we think we manage until then, even if the integration requires more changes on the EGI side than initially expected.


  • AP: "Keep an eye on accounting": See that EMI follows its roadmap and we get the possibility to send accounting different from gLite to APEL. Related to get updated on the work going on in the new EMI Workinggroup led by Andrea Guarese https://twiki.cern.ch/twiki/bin/view/EMI/ComputeAccounting
Update: Important deadlines end of June and end of July. A wiki should come out of it and a discussion group. JG will take initiative to organize something when the time has come to discuss the details on how to best integrate the current UNICORE accounting solution.
Update: The wiki and the mailinglist have been created. A new production system is online based on messaging, which can be used to test against by UNICORE. Existing direct database insert clients like SGAS, DGAS,.. will also have to be moved to this new system. JG planning a meeting with Hannover within the next two weeks to get an impression of the German D-Grid solution. Later on we should have a functional comparison between the Polish and the German solution and see what fits best to EGI in a taskforce meeting focused on accounting.
Background information http://dl.dropbox.com/u/9548018/APEL%20SSM%20background.txt and links to wiki and mailinglist: http://dl.dropbox.com/u/9548018/APEL%20SSM%20testing.txt
  • AP: "Best practices" KB and FR to send documentation links to operational-documentation-best-practices@mailman.egi.eu as soon as considered sufficiently completed, suggested procedure which could be of interest in this context: how to install more than one UNICORE service on one host.
Update: PL-Grid restructuring whole UNICORE documentation system otherwise concentration on EMI 1 release. MB will now start by pointing them to standard unicore.eu documents which conclude good documentation and installation guides and see if they would like to already include a link to them. What is missing: real best practices, the very solutions to different real integration problems. Some material exists in Polish language.
Update from Mathilde: UNICORE installation documentation (not yet covering the EMI UNICORE rpms) can be found at http://www.unicore.eu/documentation/manuals/unicore6/files/manual_installation.pdf The basic scenario (section 2) should answer the question on how to install multiple services on one server. It is not perfect for EGI purposes but maybe it is of interest.
Update: Michaela sent those links to the best practices mailinglist, but hasn't got any feedback yet.
Update: MB will try to contact them again.
  • AP on TF: to bring up site-information system (BDII) discussion up at the TCB
  • AP: "Reformulation of OLA" MB to formulate a more highlevel and middleware independent version of point 10.3 in the OLA https://documents.egi.eu/secure/ShowDocument?docid=650 and create a new ticket in the noc-managers queue for it and assign it to Tiziana, so it can be discussed at the OMB
Update: https://rt.egi.eu/rt/Ticket/Display.html?id=2574
Update: Planned roadmap according to COO: discussion for an update at the Nov OMB, with the purpose of having a new version in december of January.

after meeting on 110715

  • AP: "EMI registry use case" Find out more about the EMI registry and the static info contained: How is the distinction between the static data GOCDB and EMI registry, how can we propagate downtime info for just one Service Endpoint (URLs are needed to distungish between different instances)? (contact Laurence Field)
Progress: MB will report on her email discussion with Laurence Field and plans to push the GLUE 2.0 use case in propagating downtime information
Update: EMI registry in very early planning stage, XML solution in the meantime. Second solution for GOCDB (the GLUE 2.0 based approach) again in discussion. MB to bring forward and discuss the usecase in the next OGF PGI wg meeting.
Update: Too late to be part of V1 of http://www.ogf.org/documents/GFD.180.pdf waiting together with 2 other use cases for V2.
Update: currently reviewing related document EMI Execution Service Specification https://twiki.cern.ch/twiki/pub/EMI/EmiExecutionService/EMI-ES-Specification_v1.0.odt
Update: EMI ESS reviewed, currently no PGI wg meetings.
Update: PGI working group start again with regular phoneconferences now after OGF 32, but EMI registry currently no longer directly relevant to us now with ServicePointURLs working in GOCDB.


  • AP: "List of service types to be integrated." MB compiling a list of the next service types which should be integrated into GOCDB and sending it to our mailinglist for discussion
Update: Some discussion of this list during the meeting, AP on KB to send his list to the mailinglist as well and to continue discussing there and to put the output of this discussion in ticket https://rt.egi.eu/rt/Ticket/Display.html?id=944
Update: remaining service name description discussed during the meeting. MB to update the ticket. Decision that currently no more are needed.
Update: MB updated the ticket.
Closed: List successfully taken over and we are fine with the current set of services for the moment.
GOCDB/Input System User Documentation#Service_types


  • AP: Start a more tight collaboration with Belarus. AL to keep us updated on efforts to go open source or other possibilities to collaborate, KB to send info on new release with all major features.
Update: A new release of UNICORE RUS accounting system is available from sourceforge. It is major milestone as it completes our move to JMS based architecture with a full fail-over support. It is also updated to USE and 6.4.0 release of UNICORE. http://unicore-dev.zam.kfa-juelich.de/documentation/rus-accounting-1.3.1/
Update: The NGI_BY UNICORE Accounting service AL presented at the EGI UF 2011 got permission to go Open Source. An English version and translated documentation is being prepared for initial export to Sourceforge. NGI-DE already became a non open source version to have some tests and do some comparison.
Update: Allowance to go open source was drawn back after a week, AP on AL to send contact information do deciding director to closer investigate the reasons behind this decision.
Update: AL sent contact details to MB. MB asked Tiziana Ferrari and Steven Newhouse for help. Reply from SN: "Clearly, if an NGI refuses to distribute a tool they have developed they are free to do so." New questions: Will it still be possible to deploy the NGI_BY accounting solution and use it to publish accounting records? Will it still be translated? How interested will some NGIs be to use a closed source solution?
Update: Already translated to English, but it cannot easily distributed with closed source because it's written in PHP. TF contacted the responsible director: Not so easy to make it legally open source, because it was bound to the funding by Belarus and Russia within the SKIF-GRID program and if not putting in our own funding, we don't have chances to make it OpenSource. Money allocation in the near future within EGI-InSPIRE very unlikely, though. Suggestion to try to get money somewhere else. NGI_BY of course still welcome in our task force.
Closed.



(* AP: KB to create an RT requirement ticket about integration of NGI specific service types into the regional dashboard. Update: Ticket has been created and will be handled next OTAG 110720)

older

  • AP: "ServiceEndpointURLs in GOCDB" Reopen GOCDB requirement ticket for EndPointServiceURL https://rt.egi.eu/rt/Ticket/Display.html?id=975 and ask for second solution (Add URL field for ServiceEndpoint and repeat ServiceEndpoint to represent different URLs /Associate a ServiceEndpoint with a new „1-to-many‟ EndpointLocation entity), with a proposed timeline of 6 months.
Progress: David provided us with some updated slides on the possible implementation with more detail: https://wiki.egi.eu/wiki/File:GocdbGlue2Unicore.pdf
Update: Personal discussion at the EGI UF, Vilnius with John Casson: A new proposed solution with getting rid of the primary key constraint and a new timeline of one month has been discussed. It fulfills all our requirements. This requirement has now top-priority within GOCDB development.
Update: Requirement has been fixed and seems to work: https://goc.gridops.org/portal/index.php?Page_Type=View_Object&object_id=22973&grid_id=0
Update: closed. New AP created to add more UNICORE services into GOCDB.


  • AP: EI to contact Maciej Pawlik and Paweł Wolniewicz to discuss sensible Nagios configuration default definitions.
Update: redundant because of fast GOCDB integration, will eventually be done anyway. Closed.


  • AP: "Next Meeting": MB making again a doodlepoll for next meeting with Global Timezones and maybe function enabled.
Update: http://www.doodle.com/7xcgzsqxadp45v6q



  • AP: official requirement towards APEL (accounting monitoring must be sufficient to decide whether all deployed middlewares of a site publish accounting data)
https://rt.egi.eu/rt/Ticket/Display.html?id=1607
  • AP: ML to test sending a GGUS ticket to UNICORE:
See ticket https://gus.fzk.de/dmsu/dmsu_ticket.php?ticket=68192 Who got it?
Update: https://gus.fzk.de/dmsu/dmsu_ticket.php?ticket=68177 was produced as another test ticket. Check why nobody got the tickets!
Progress: Ticket towards GGUS created: https://gus.fzk.de/ws/ticket_info.php?ticket=68354
Updated: GGUS ticket correctly sent to Jülich managed list emi-support@unicore.eu , this list has currently a small number of subscribed members. Awaiting internal EMI decision to add more people like e.g. KB

Final: Currently keep the small number who will get the GGUS ticket, who will forward them to the UNICORE tracking system in sourceforge. Follow up of this AP is out of the scope for this task force. Will be taken up gain within EMI. UNICORE related GGUS FAQ has been transported to EGI wiki: GGUS:UNICORE-Client FAQ and GGUS:UNICORE-Server FAQ.


  • AP: DM to rename the first 3 UNICORE service types.
unicore6.registry
unicore6.UNICOREX
unicore6.gateway
Final: GOCDB now uses the new Unicore service type names (unicore6.gateway, unicore6.registry, unicore6.UNICOREX). See: GOCDB/Input System User Documentation#Service_types



  • AP: KB sends a summary of CIS, why it is not supported by EMI and why the XML approach is advantageous monitoring wise.
Why this is not supported by EMI is a political decision based on limited manpower and effort, KB wrote a detailed description on the usage of CIS in the monitoring system.


  • AP: ML making a doodlepoll for next meeting in March


  • AP: ML to send EI minutes of GOCDB dedicated phone conference


  • AP: KB contacting Germany through ticket 306
https://rt.egi.eu/rt/Ticket/Display.html?id=306